Пребарување на медицински документи со мултимодални податоци

Abstract

In this thesis we have researched the field of medical documents retrieval. Our work in the field resulted in a complex medical document retrieval system which contains the proposed methods. The implemented system is divided into multiple subsystems: subsystem for modality classification of medical images, subsystem for medical image retrieval and multiple subsystems for medical articles retrieval. The subsystem for modality classification of medical images annotates the images with the appropriate modality based on their visual and/or textual characteristics. This subsystem contains multiple descriptors for feature extraction from the images such as LBP, FCTH, CEDD, SIFT and OSIFT. The textual characteristics are formed based on the articles where the images appear. The classification is done using support vector machines. The evaluation of the subsystem shows that the OSIFT descriptor provides the best performance compared to the other descriptors, but combining all visual descriptors provides even better results. Finally, combining the visual descriptors and the textual characteristics provide the best overall performance, which are state-of-the-art for the datasets we used for evaluation. The subsystem for medical image retrieval works based on the visual (content) and/or textual characteristics of the images. The text-based part of the medical image retrieval subsystem uses the textual data of the articles where the images appear and creates a representation using the vector model. The retrieval of this part is boosted by query expansion using pseudo-relevance feedback. The content-based part of the subsystem uses RGB histograms which are encoded into Fisher vectors to describe the images. The method of product quantization is applied here, so that this part of the subsystem is scalable and allows fast retrieval over large image collections. The medical image retrieval subsystem works in combination with the modality classification subsystem, in such a way that the initially retrieved images are classified and are re-ranked based on their relation with the submitted query. This subsystem is evaluated over several standardized datasets and provides state-ofthe- art results. The subsystems related to medical articles retrieval perform the retrieval based on the textual data in them. The common thing for the subsystems is that they create a textual representation of the medical articles, by first enriching them using external medical or generic knowledge databases. The queries provided by the users are modified, in the retrieval phase, using external medical knowledge tools and/or pseudo-relevance feedback. All subsystems for medical articles retrieval are evaluated and provide good results which have been published in multiple papers.