Interpreting and summarizing the insights gained from medical images such as radiology output is a time-consuming task that involves highly trained experts and often represents a bottleneck in clinical diagnosis pipelines.

Consequently, there is a considerable need for automatic methods that can approximate this mapping from visual information to condensed textual descriptions. The more image characteristics are known, the more structured are the radiology scans and hence, the more efficient are the radiologists regarding interpretation. We work on the basis of a large-scale collection of figures from open access biomedical journal articles (PubMed Central). All images in the training data are accompanied by UMLS concepts extracted from the original image caption.

The first step to automatic image captioning and scene understanding is identifying the presence and location of relevant concepts in a large corpus of medical images. Based on the visual image content, this projects provides the building blocks for the scene understanding step by identifying the individual components from which captions are composed. The concepts can be further applied for context-based image and information retrieval purposes.

ImageCLEFcaption is a task of the ImageCLEF evaluation campaign. ImageCLEF is part of the Cross Language Evaluation Forum (CLEF). Since 2017, this caption task was added. The goal is to create databases to interpret medical images . The organisers of the ImageCLEFcaption challenge distribute the “collection” consisting of images and annotations for image concept extraction. Participants then apply their tools and techniques which are then evaluated in a blind collection.

Follow @imageclefExample of an image in the training set and its corresponding UMLS concepts.

Project members

Acknowledgements






Publications

  1. Garcia Seco De Herrera, A., Ionescu, B., Müller, H., Péteri, R., Ben Abacha, A., Friedrich, CM., Rückert, J., Bloch, L., Brüngel, R., Idrissi-Yaghir, A., Schäfer, H., Kozlovski, S., Dicente Cid, Y., Kovalev, V., Chamberlain, J., Clark, A., Campello, A., Schindler, H., Deshayes, J., Popescu, A., S¸tefan, L-D., Constantin, MG. and Dogariu, M., (2022). ImageCLEF 2022: Multimedia Retrieval in Medical, Nature, Fusion, and Internet Applications
  2. Ionescu, B., Müller, H., Péteri, R., Abacha, AB., Sarrouti, M., Demner-Fushman, D., Hasan, SA., Kozlovski, S., Liauchuk, V., Cid, YD., Kovalev, V., Pelka, O., de Herrera, AGS., Jacutprakart, J., Friedrich, CM., Berari, R., Tauteanu, A., Fichou, D., Brie, P., Dogariu, M., Ştefan, LD., Constantin, MG., Chamberlain, J., Campello, A., Clark, A., Oliver, TA., Moustahfid, H., Popescu, A. and Deshayes-Chossart, J., (2021). Overview of the ImageCLEF 2021: Multimedia Retrieval in Medical, Nature, Internet and Social Media Applications
  3. Pelka, O., Abacha, AB., Seco de Herrera, AG., Jacutprakart, J., Friedrich, CM. and Müller, H., (2021). Overview of the ImageCLEFmed 2021 Concept & Caption Prediction Task
  4. Jacutprakart, J., Andrade, FP., Cuan, R., Compean, AA., Papanastasiou, G. and Seco de Herrera, AG., (2021). NLIP-Essex-ITESM at ImageCLEFcaption 2021 task: Deep learning-based information retrieval and multi-label classification towards improving medical image understanding
  5. Ionescu, B., Müller, H., Péteri, R., Ben Abacha, A., Demner-Fushman, D., Hasan, SA., Sarrouti, M., Pelka, O., Friedrich, C., Garcia Seco De Herrera, A., Jacutprakart, J., Kovalev, V., Kozlovski, S., Liauchuk, V., Dicente Cid, Y., Chamberlain, J., Clark, A., Campello, A., Moustahfid, H., Oliver, T., Schulz, A., Brie, P., Berari, R., Fichou, D., Tautenau, A., Dogariu, M., Stefan, LD., Constantin, MG., Deshayes, J. and Popescu, A., (2021). The 2021 ImageCLEF Benchmark: Multimedia Retrieval in Medical, Nature, Internet and Social Media Applications
  6. Ionescu, B., Müller, H., Péteri, R., Dang-Nguyen, D-T., Zhou, L., Piras, L., Riegler, M., Halvorsen, P., Tran, M-T., Lux, M., Gurrin, C., Chamberlain, J., Clark, A., Campello, A., Garcia Seco De Herrera, A., Ben Abacha, A., Datla, V., A. Hasan, S., Liu, J., Demner-Fushman, D., Obioma, P., Friedrich, CM., Dicente Cid, Y., Kozlovski, S., Liauchuk, V., Kovalev, V., Berari, R., Brie, P., Fichou, D., Dogariu, M., Daniel Stefan, L. and Constantin, MG., (2020). ImageCLEF 2020: Multimedia Retrieval in Lifelogging, Medical, Nature, and Security Applications
  7. Ionescu, B., Müller, H., Péteri, R., Abacha, AB., Datla, V., Hasan, SA., Demner-Fushman, D., Kozlovski, S., Liauchuk, V., Cid, YD., Kovalev, V., Pelka, O., Friedrich, CM., Garcia Seco De Herrera, A., Ninh, V-T., Le, T-K., Zhou, L., Piras, L., Riegler, M., Halvorsen, PL., Tran, M-T., Lux, M., Gurrin, C., Dang-Nguyen, D-T., Chamberlain, J., Clark, A., Campello, A., Fichou, D., Berari, R., Brie, P., Dogariu, M., Stefan, LD. and Constantin, MG., (2020). ImageCLEF 2020: Multimedia Retrieval in Lifelogging, Medical, Nature, and Internet Applications
  8. Pelka, O., Friedrich, CM., García Seco de Herrera, A. and Müller, H., (2020). Overview of the ImageCLEFmed 2020 Concept Prediction Task: Medical Image Understanding
  9. García Seco de Herrera, A., Parrilla Andrade, F., Bentley, L. and Aceves Compean, A., (2020). Essex at ImageCLEFcaption 2020 task
  10. Ionescu, B., Müller, H., Péteri, R., Abacha, AB., Datla, V., Hasan, SA., Demner-Fushman, D., Kozlovski, S., Liauchuk, V., Cid, YD., Kovalev, V., Pelka, O., Friedrich, CM., García Seco de Herrera, A., Ninh, V-T., Le, T-K., Zhou, L., Piras, L., Riegler, M., Halvorsen, P., Tran, M-T., Lux, M., Gurrin, C., Dang-Nguyen, D-T., Chamberlain, J., Clark, A., Campello, A., Fichou, D., Berari, R., Brie, P., Dogariu, M., Ştefan, LD. and Constantin, MG., (2020). Overview of the ImageCLEF 2020: Multimedia Retrieval in Medical, Lifelogging, Nature, and Internet Applications
  11. Pelka, O., Friedrich, CM., Seco De Herrera, AG. and Müller, H., (2019). Overview of the ImageCLEFmed 2019 concept detection task
  12. Ionescu, B., Müller, H., Péteri, R., Cid, YD., Liauchuk, V., Kovalev, V., Klimuk, D., Tarasau, A., Abacha, AB., Hasan, SA., Datla, V., Liu, J., Demner-Fushman, D., Dang-Nguyen, D-T., Piras, L., Riegler, M., Tran, M-T., Lux, M., Gurrin, C., Pelka, O., Friedrich, CM., Garcia Seco De Herrera, A., Garcia, N., Kavallieratou, E., del Blanco, CR., Cuevas, C., Vasillopoulos, N., Karampidis, K., Chamberlain, J., Clark, A. and Campello, A., (2019). ImageCLEF 2019: Multimedia Retrieval in Medicine, Lifelogging, Security and Nature
  13. Ionescu, B., Müller, H., Péteri, R., Dang-Nguyen, D-T., Piras, L., Riegler, M., Tran, M-T., Lux, M., Gurrin, C., Cid, YD., Liauchuk, V., Kovalev, V., Ben Abacha, A., Hasan, SA., Datla, V., Liu, J., Demner-Fushman, D., Pelka, O., Friedrich, CM., Chamberlain, J., Clark, A., Garcia Seco De Herrera, A., Garcia, N., Kavallieratou, E., del Blanco, CR., Rodríguez, CC., Vasillopoulos, N. and Karampidis, K., (2019). ImageCLEF 2019: Multimedia Retrieval in Lifelogging, Medical, Nature, and Security Applications
  14. Garcia Seco De Herrera, A., Eickhof, C., Andrearczyk, V. and Müller, H., (2018). Overview of the ImageCLEF 2018 Caption Prediction Tasks
  15. Ionescu, B., Müller, H., Villegas, M., Garcia Seco De Herrera, A., Eickhoff, C., Andrearczyk, V., Dicente Cid, Y., Liauchuk, V., Kovalev, V., Hasan, SA., Ling, Y., Farri, O., Liu, J., Lungren, M., Dang-Nguyen, D-T., Piras, L., Riegler, M., Zhou, L., Lux, M. and Gurrin, C., (2018). Overview of ImageCLEF 2018: Challenges, Datasets and Evaluation
  16. Eickhoff, C., Schwall, I., Garcia Seco De Herrera, A. and Müller, H., (2017). Overview of ImageCLEFcaption 2017 – Image Caption Prediction and Concept Detection for Biomedical Images
  17. Ionescu, B., Müller, H., Villegas, M., Arenas, H., Boato, G., Dang-Nguyen, D-T., Dicente Cid, Y., Eickhoff, C., Garcia Seco De Herrera, A., Gurrin, C., Islam, B., Kovalev, V., Liauchuk, V., Mothe, J., Piras, L., Riegler, M. and Schwall, I., (2017). Overview of ImageCLEF 2017: Information extraction from images
  18. Abacha, AB., Seco De Herrera, AG., Gayen, S., Demner-Fushman, D. and Antani, S., (2017). NLM at ImageCLEF 2017 caption task
  19. Garcia Seco De Herrera, A., Schaer, R., Bromuri, S. and Müller, H., (2016). Overview of the ImageCLEF 2016 Medical Task
  20. Villegas, M., Müller, H., Garcia Seco De Herrera, A., Schaer, R., Bromuri, S., Gilbert, A., Piras, L., Wang, J., Yan, F., Ramisa, A., Dellandrea, E., Gaizauskas, R., Mikolajczyk, K., Puigcerver, J., Toselli, AH., Sánchez, J-A. and Vidal, E., (2016). General Overview of ImageCLEF at the CLEF 2016 Labs