ImageCLEFmed GANs

Motivation

Welcome to the first edition of the GANs Task!

The GANs task is a completely new challenge in the ImageCLEFmedical track.
The task is focused on examining the existing hypothesis that GANs are generating medical images that contain the "fingerprints" of the real images used for generative network training.
If the hypothesis is correct, artificial biomedical images may be subject to the same sharing and usage limitations as real sensitive medical data. On the other hand, if the hypothesis is wrong, GANs may be potentially used to create rich datasets of biomedical images that are free of ethical and privacy regulations.

• The participants will test the hypothesis by solving one or several tasks related to the detection of relations between real and artificial biomedical image datasets. The final challenge description, data, and metrics will be announced soon.






News

To be added soon.

Task Description

Investigate the hypothesis that GANs are generating medical images that are in some way similar to the ones used for the GAN training. The task is related to the problem of security of personal medical image data in the context of generating and using artificial images in different real-life scenarios.

The objective of the task is to detect “fingerprints” within the synthetic biomedical image data to determine which real images were used in training to produce the generated images. The task supposes performing analysis of test image datasets and assessment of the probability with which certain images of real patients were used for training image generators and which were not.

Note that identification of artificial images or classification image datasets to the real and artificial ones is NOT assumed.

Data

The benchmarking image data are the axial slices of 3D CT images of about 8000 lung tuberculosis patients. This particularly means that some of them may appear pretty “normal” whereas the others may contain certain lung lesions including the severe ones. These images are stored in the form of 8 bit/pixel PNG images with dimensions of 256x256 pixels.

The artificial slice images are 256x256 pixels in size. All of them were generated using the Diffuse Neural Networks.

The published development dataset for task includes 500 artificial images, 80 real images which were not used for training generative neural networks as well as 80 real images taken from the image set which has been used for training corresponding generative model.
Development dataset is available in the official GitHub repo of this task.

The test dataset was created in similar way. The only difference is that the two subsets of real images are mixed and no proportion of non-used and used ones has been disclosed. Thus, a total of 10,000 generated and 200 real images are provided.
Test dataset is available in the official GitHub repo of this task.

Evaluation methodology

For assessing performance the following metrics will be used: F1 score, Accuracy, Recall.
More information to be added soon.

Participant registration

Please refer to the general ImageCLEF registration instructions

Preliminary Schedule

Each of the tasks sets its own schedule, so please check the corresponding task webpage for specific dates. A (tentative) global schedule can be found below:

• 14.11.2022: registration opens for all ImageCLEF tasks
• 17.01.2023: development data release starts
• 14.03.2023: test data release starts
• 14.05.2023: deadline for submitting the participants runs
• 17.05.2023: release of the processed results by the task organizers
• 05.06.2023: deadline for submission of working notes papers by the participants
• 23.06.2023: notification of acceptance of the working notes papers
• 07.07.2023: camera ready working notes papers
• 18-21.09.2023: CLEF 2023, Thessaloniki, Greece

Submission Instructions

For the submission we expect the following format:
<figure_id>,<score>

You need to respect the following constraints:

• The separator between the figure ID and the prediction score (1/0) has to be a comma (,)
• Each figure ID of test set must be included in the submitted file exactly once

Submissions are done by pull request in 2023_ImageCLEFmed_GANs_submissions repository

Results

Group rank	Group name	Submission #	F1-score
#1	VCMI	submission 2	0.802
#2	VCMI	submission 1	0.731
#3	VCMI	submission 3	0.707
#4	PicusLabMed	submission 8	0.666
#5	VCMI	submission 4	0.654
#6	AIMultimediaLab	submission 1	0.626
#7	PicusLabMed	submission 6	0.624
#8	VCMI	submission 5	0.621
#9	Clef-CSE-GAN-Team	submission 1	0.614
#10	VCMI	submission 7	0.613
#11	VCMI	submission 6	0.605
#12	VCMI	submission 10	0.594
#13	AIMultimediaLab	submission 2	0.585
#14	one five one zero	submission 2	0.563
#15	PicusLabMed	submission 9	0.562
#16	PicusLabMed	submission 4	0.552
#17	KDE lab	submission 5	0.548
#18	one five one zero	submission 3	0.522
#19	Clef-CSE-GAN-Team	submission 2	0.521
#20	VCMI	submission 9	0.514
#21	one five one zero	submission 1	0.507
#22	GAN-ISI	submission 5	0.502
#23	GAN-ISI	submission 2	0.489
#24	PicusLabMed	submission 10	0.487
#25	GAN-ISI	submission 3	0.486
#26	GAN-ISI	submission 4	0.483
#27	DMK	submission 1	0.480
#28	PicusLabMed	submission 2	0.470
#29	KDE lab	submission 2	0.469
#30	GAN-ISI	submission 1	0.469
#31	KDE lab	submission 1	0.465
#32	KDE lab	submission 4	0.457
#33	DMK	submission 2	0.449
#34	VCMI	submission 8	0.448
#35	PicusLabMed	submission 1	0.434
#36	Clef-CSE-GAN-Team	submission 3	0.431
#37	PicusLabMed	submission 3	0.419
#38	PicusLabMed	submission 5	0.417
#39	KDE Lab	submission 3	0.407
#40	PicusLabMed	submission 7	0.093

CEUR Working Notes

To be added soon.

Citations

When referring to ImageCLEF 2023, please cite the following publication:

• Bogdan Ionescu, Henning Müller, Ana-Maria Drăgulinescu, Wen-wai Yim, Asma Ben Abacha, Neal Snider, Griffin Adams, Meliha Yetisgen, Johannes Rückert, Alba García Seco de Herrera, Christoph M. Friedrich, Louise Bloch, Raphael Brüngel, Ahmad Idrissi-Yaghir, Henning Schäfer, Steven A. Hicks, Michael A. Riegler, Vajira Thambawita, Andrea Storås, Pål Halvorsen, Nikolaos Papachrysos, Johanna Schöler, Debesh Jha, Alexandra-Georgiana Andrei, Ahmedkhan Radzhabov, Ioan Coman, Vassili Kovalev, Alexandru Stan, George Ioannidis, Hugo Manguinhas, Liviu-Daniel Ștefan, Mihai Gabriel Constantin, Mihai Dogariu, Jérôme Deshayes, Adrian Popescu, Overview of the ImageCLEF 2023: Multimedia Retrieval in Medical, Social Media and Recommender Systems Applications, in Experimental IR Meets Multilinguality, Multimodality, and Interaction.Proceedings of the 14th International Conference of the CLEF Association (CLEF 2023), Springer Lecture Notes in Computer Science LNCS, Thessaloniki, Greece, September 18-21, 2023.
• BibTex:
  @inproceedings{ImageCLEF2023,
  author = {Bogdan Ionescu and Henning M\"uller and Ana{-}Maria Dr\u{a}gulinescu and Wen{-}wai Yim and Asma {Ben Abacha} and Neal Snider and Griffin Adams and Meliha Yetisgen and Johannes R\"uckert and Alba {Garc\’{\i}a Seco de Herrera} and Christoph M. Friedrich and Louise Bloch and Raphael Br\"ungel and Ahmad Idrissi{-}Yaghir and Henning Sch\"afer and Steven A. Hicks and Michael A. Riegler and Vajira Thambawita and Andrea Storås and Pål Halvorsen and Nikolaos Papachrysos, Johanna Schöler, Debesh Jha, Alexandra{-}Georgiana Andrei, Ahmedkhan Radzhabov, Ioan Coman, Vassili Kovalev, Alexandru Stan, George Ioannidis and Hugo Manguinhas and Liviu{-}Daniel \c{S}tefan and Mihai Gabriel Constantin and Mihai Dogariu and J\'er\^ome Deshayes and Adrian Popescu}
  title = {{Overview of ImageCLEF 2023}: Multimedia Retrieval in Medical, SocialMedia and Recommender Systems Applications},
  booktitle = {Experimental IR Meets Multilinguality, Multimodality, and Interaction},
  series = {Proceedings of the 14th International Conference of the CLEF Association (CLEF 2023)},
  year = {2023},
  publisher = {Springer Lecture Notes in Computer Science LNCS},
  pages = {},
  month = {September 18-21},
  address = {Thessaloniki, Greece}

  }

When referring to ImageCLEF2023medical GANs general goals, general results, etc. please cite the following publication which will be published by September 2023:

• Alexandra-Georgiana Andrei, Ahmedkhan Radzhabov, Ioan Coman, Vassili Kovalev, Bogdan Ionescu and Henning Müller. Overview of the ImageCLEFmedical GANs 2023 Task: Identifying Training Data "Fingerprints" in Synthetic Biomedical Images Generated by GANs for Medical Image Security, Experimental IR Meets Multilinguality, Multimodality, and Interaction. CEUR Workshop Proceedings (CEUR-WS.org), Thessaloniki, Greece, September 18-21, 2023.
• BibTex:
  @inproceedings{ImageCLEF2023medicalGANs,
  author = { Alexandra{-}Georgiana Andrei and Ahmedkhan Radzhabov and Ioan Coman and Vassili Kovalev and Bogdan Ionescu and Henning M\"uller},
  title = {Overview of {ImageCLEFmedical GANs} 2023 Task -- {Identifying Training Data "Fingerprints" in Synthetic Biomedical Images Generated by GANs for Medical Image Security}},
  booktitle = {CLEF2023 Working Notes},
  series = {{CEUR} Workshop Proceedings},
  year = {2023},
  volume = {},
  publisher = {CEUR-WS.org },
  pages = {},
  month = {September 18-21},
  address = { Thessaloniki, Greece }
  }

Contact

Organizers:

• Alexandra Andrei <andrei.alexandra96(at)yahoo.com>, Politehnica University of Bucharest, Romania
• Ahmedkhan Radzhabov <filipovichigor(at)yandex.by>, Belarus State University, Minsk, Belarus
• Ioan Coman <coman.ioan95(at)gmail.com>, Politehnica University of Bucharest, Romania
• Vassili Kovalev <vassili.kovalev(at)gmail.com>, Belarusian Academy of Sciences, Minsk, Belarus
• Bogdan Ionescu <bogdan.ionescu(at)upb.ro>, Politehnica University of Bucharest, Romania
• Henning Müller <henning.mueller(at)hevs.ch>, University of Applied Sciences Western Switzerland, Sierre, Switzerland

Acknowledgments

The contribution of Alexandra Andrei, Bogdan Ionescu and Henning Müller to this task is supported under project AI4Media, A European Excellence Centre for Media, Society and Democracy, H2020 ICT-48-2020, grant #951911.