Subtask 1: concept annotation

In the concept annotation task your goal is to detect the presence of the various concepts in the images and provide us with the annotations on an per-image basis, see the figure below for an example.

    
Images annotated with the concept 'reflection'.

Data format

We can distinguish between the 'concept' files, where each file refers to a single concept in which its presence or absence in the images is indicated, and the 'annotation' files, where each file refers to a single image in which the presence or absence of the concepts is indicated.

Concept files: clean version
The clean version of the concept files only lists the image identifiers for which the human annotators decided by majority vote the concept was present. The format of a file referring to a particular concept is as follows:

[image identifier a]
[image identifier b]
[image identifier c]
...
[image identifier z]

Concept files: raw version
The raw version of the concept files lists for every image several judgments given by human annotators on whether or not the concept was present. The format of a file referring to a particular concept is as follows:

[image identifier 0] [judgment] [judgment] ... [judgment]
[image identifier 1] [judgment] [judgment] ... [judgment]
...
[image identifier N-1] [judgment] [judgment] ... [judgment]

where each line contains an image identifier and at least three judgments, each of which can be either 0, indicating a particular annotator did not believe the concept was present, or 1, indicating the annotator believed the concept was present; the elements on a single line are separated from each other by a single space.

Annotation files: clean version
The clean version of the annotation files only lists the concepts for which the human annotators decided by majority vote it was present in the image. The format of a file referring to a particular image is as follows:

[concept a]
[concept b]
[concept c]
...
[concept z]

Annotation files: raw version
The raw version of the annotation files lists for each concept the agreement between the judgments of the human annotators whether or not it was present in the image. The format of a file referring to a particular image is as follows:

[concept 0] [agreement]
[concept 1] [agreement]
[concept 2] [agreement]
...
[concept N-1] [agreement]

where each line contains the name of a concept and the average agreement on the presence of a concept, where both elements are separated by a single space.

Be aware that there is no guarantee that the concepts are 100% correct. Even though the concepts have been annotated by multiple persons, there is a lot of room for personal interpretation and thus subjectivity. Some concepts are relatively easy to detect, e.g. 'sun', 'cat', whereas others are more difficult, e.g. 'partial blur', 'active', so make sure your algorithm is flexible enough to handle different levels of annotation quality.

Submission format

The submission format to indicate which concepts are present in an image is similar to the format above. Each image in the test collection is to be represented by a single line in your submission file, which should look as follows:

[image identifier] [confidence score for concept 0] [binary score for concept 0] ... [confidence score for concept N-1] [binary score for concept N-1]

where:

• The image identifier refers to the filename of the image as above.
• The confidence score refers to a floating-point value between 0 and 1 that indicates how confident your algorithm is in the presence of the concept in the image, where a higher value denotes a higher confidence.
• The binary score refers to a value of either 0 or 1 that indicates the final decision of your algorithm on the presence of the concept in the image, with 0 meaning the absence of the concept and 1 meaning the presence of the concept.
• The elements are separated from each other by a single space.

We call each submission a 'run'. When you submit a file you will be required to indicate which features you used for the run, i.e. visual features only, textual features only or a combination of visual and textual features. You are allowed to submit up to five runs in total. The submission system will perform an automatic check to see if your submission file is correct.

Evaluation

To determine the quality of your annotations we will apply three different measures:

Mean Average Precision (MAP)

This evaluation measure first ranks the images by their confidence scores, from high to low, for each concept separately. The images are inspected one by one and each time a relevant image is encountered the precision and recall values are computed. In case of ties we consider all the images with the same confidence score together at once and produce only a single precision and recall value for them. We then interpolate the values so the recall measurements range from 0.0 to 1.0 with steps of 0.1; the precisions at these recall levels are obtained by taking the maximum precision obtained at any non-interpolated recall level equal or greater to the interpolated recall step level under consideration. To obtain the overall non-interpolated MAP (MnAP) value we average the non-interpolated precisions for each concept and then average these averages, whereas to obtain the overall interpolated MAP (MiAP) we instead average the average interpolated precisions over all concepts. Note that your primary focus should be on the interpolated MAP, which is the value we report below, although for completeness we report both the non-interpolated and interpolated MAP values in the detailed results.

Geometric Mean Average Precision (GMAP)

This evaluation measure is an extension to MAP. When comparing runs with each other the GMAP specifically highlights improvements obtained on relatively difficult concepts, e.g. increasing the average precision of a concept from 0.05 to 0.10 has a larger impact in its contribution to the GMAP than increasing the average precision from 0.25 to 0.30. To compute the non-interpolated GMAP (GMnAP) and the interpolated GMAP (GMiAP), we follow the same procedure as with MnAP and MiAP, but we instead average the logs of the average precision for each concept, after which we exponentiate the resulting average back to obtain the GMAP. To avoid taking the log of an average precision of zero we add a very small epsilon value to each average precision before computing its log, which we remove again after exponentiating the averages of these logs; when the epsilon value is very small its effect on the final GMAP is negligible. Note that your primary focus should be on the interpolated GMAP, which is the value we report below, although for completeness we report both the non-interpolated and interpolated GMAP values in the detailed results.

F1

The F1-measure uses the provided binary scores to determine how well the annotations are. We have computed the instance-averaged, micro-averaged and macro-averaged F1 scores for the photos as well as for the concepts. The instance-F1 for the photos is computed by determining the number of true positives, false positives, true negatives and false negatives in terms of detected concepts and using this to compute the F1-score for each invididual photo, after which the F1-scores are averaged over all photos. The micro-F1 for the photos is computed by averaging the precision and recall scores for each individual photo and then computing the F1-score from these averages. The macro-F1 for the photos is computed by aggregating the number of true positives, false positives, true negatives and false negatives over all photos and then computing the F1-score based on these numbers. The micro-F1 and macro-F1 for the concepts is computed in a similar fashion, swapping the roles of the photos and concepts. Note that your primary focus should be on the photo-based micro-F1, which is the value we report below, although for completeness we report all instance-F1 and macro-F1 values in the detailed results.

Results

In the table below we present the results for the runs, organized alphabetically by team name. We have evaluated the quality of your annotations on all concepts combined, as well as for the different concept categories. You can download the detailed results at the bottom of this page. For future reference you can download the Java-based evaluation code below; note that the code may not look as simple as MATLAB code that hides away many of the details, but it was written with legibility in mind. We will release the ground truth on the first day of the CLEF conference, so afterwards you will be able to apply the code to your own runs.

Note that the MAP and GMAP are very dependent on ties in the confidence scores of images. An example case is URJCyUNED, where all images in each run received a confidence score of 1.0. This resulted in only a single precision-recall value computed for all these images combined, which happened to be the same for each run. In contrast, the F-ex scores are based on the binary annotations, and these scores were thus not affected by any ties.