Subtask 2: concept retrieval

The queries for the concept-based retrieval task are based on an inspection of the query logs of a popular image search engine. We analyzed what people look for on the internet related to the concepts we defined for the photo annotation task in order to form a realistic yet challenging set of queries, see the figure below for an example.

    
Images retrieved for the query 'find images containing light trails of traffic'.

Data format

The concept-based retrieval queries are provided in the XML format, where the first line contains the typical XML version header, the root node is <queries> and each query itself is a child of the root node. An example is as follows:

<query>
  <number>14</number>
  <title>traffic light trails</title>
  <description>The user is looking for photos showing light trails made by traffic at night. The picture effectively is taken with a long
                       exposure time and thus the trails form straight or smoothly curving lines following the flow of traffic.</description>
  <image>4bc87acb57e75469a5cbc7f3de1247b1</image>
  <image>5b14d24c4e8747ade5e7a393e07f3dfe</image>
  <image>6d5fda3c612a8d888bc969cdc7852ff</image>
</query>

where:

• The number uniquely refers to a particular query, where the queries are numbered starting at 0 and onwards.
• The title refers to what the user may possibly type as the query he would issue to an image search engine to find the images he has in mind.
• The description refers to what the user really is looking for and may contain one or more constraints indicating the kinds of images that will be considered correct and/or incorrect matches to the query.
• The query will be accompanied by three example images taken from the training set of the photo annotation task.

Submission format

The submission format follows the TREC format, so we can use their tools to evaluate your runs. For each query you may return up to 1000 relevant images, each representing a single line in your submission file and which should look as follows:

[query number] [placeholder] [image identifier] [retrieval rank] [retrieval score] [run identifier]

where:

• The query number refers to the query you are reporting the results on as above.
• The placeholder is necessary to conform to the TREC format, but is unused. Please always use the value 'IC12' here.
• The image identifier refers to the image you consider to be relevant to the query, where the identifiers are the filenames of images from the retrieval collection, e.g. 'im223647'.
• The retrieval rank refers a value between 0 and N-1, where N is the number of images you are reporting as relevant to the query. The rank indicates the position of the image within all the other relevant images, with 0 meaning the best position and N-1 meaning the worst position.
• The retrieval score refers to a floating-point value larger or equal to zero, where more relevance to the query should be expressed in a higher score.
• The run identifier is a tag that uniquely identifies yourself and the run. All lines within the same submission file should thus use the same tag, but should change between runs. Please change the identifier from year to year, since we may compare results across years and having the same name show up for both years is confusing. Make sure to use a tag that is a single alphanumerical string, so it should not contain any other characters than letters and digits.
• The elements are separated from each other by a single space.

In your submission file, you must process each query in order starting at number 0. Within a query the relevant images must be ordered by ascending retrieval ranks and descending retrieval scores. The retrieval rank and retrieval scores are inversely connected: the most relevant image to the query gets ranked at position 0 and assigned the highest score, whereas the least relevant image gets ranked at position N-1 and assigned the lowest score. Note that while ties in retrieval scores are possible they are not possible in retrieval ranks, so even when images are equally relevant a preference decision has to be made which image gets more highly ranked than the other. In the submission file the most straightforward approach is thus to order the images relevant to a query by their retrieval ranks.

You are allowed to submit up to 10 runs in total. When you submit a file you will be required to indicate which method you used for the run, i.e. whether the query formulation is done completely automatic or whether you required manual intervention to construct the query out of the query descriptions, e.g. by deriving a boolean connection of concepts (of course, after the query formulation step all runs should retrieve the images in an automated manner otherwise it would be cheating). Furthermore you will be required to indicate which features you used for the run, i.e. visual features only, textual features only or a combination of visual and textual features. The submission system will perform an automatic check to see if your submission file is correct. Finally, you will be asked to designate one of your runs as the primary run, where the images retrieved for this run will definitely contribute to the pooled relevance assessment, see below for more details.

Evaluation

We use a subset of the MIRFLICKR collection comprising of 200,000 photos. Due to this sheer quantity of images it is difficult to create a ground truth using human annotation and we will use 'pooling' instead. With pooling, we aggregate the top 100 results for all runs submitted by all teams for each of the topics and manually assess their relevance. We evaluate your runs and report on the results using the following evaluation measures:

Mean Average Precision (MAP)

This evaluation measure first ranks the images by their confidence scores, from high to low, for each query separately. The images are inspected one by one and each time a relevant image is encountered the precision and recall values are computed. In case of ties we consider all the images with the same confidence score together at once and produce only a single precision and recall value for them. We then interpolate the values so the recall measurements range from 0.0 to 1.0 with steps of 0.1; the precisions at these recall levels are obtained by taking the maximum precision obtained at any non-interpolated recall level equal or greater to the interpolated recall step level under consideration. To obtain the overall non-interpolated MAP (MnAP) value we average the non-interpolated precisions for each concept and then average these averages, whereas to obtain the overall interpolated MAP (MiAP) we instead average the average interpolated precisions over all concepts. Note that your primary focus should be on the non-interpolated MAP, which is the value we report below, although for completeness we report both the non-interpolated and interpolated MAP values in the detailed results.

Geometric Mean Average Precision (GMAP)

This evaluation measure is an extension to MAP. When comparing runs with each other the GMAP specifically highlights improvements obtained on relatively difficult queries, e.g. increasing the average precision of a query from 0.05 to 0.10 has a larger impact in its contribution to the GMAP than increasing the average precision from 0.25 to 0.30. To compute the non-interpolated GMAP (GMnAP) and the interpolated GMAP (GMiAP), we follow the same procedure as with MnAP and MiAP, but we instead average the logs of the average precision for each query, after which we exponentiate the resulting average back to obtain the GMAP. To avoid taking the log of an average precision of zero we add a very small epsilon value to each average precision before computing its log, which we remove again after exponentiating the averages of these logs; when the epsilon value is very small its effect on the final GMAP is negligible. Note that your primary focus should not be on any of the GMAP values, we nonetheless report both the non-interpolated and interpolated GMAP values in the detailed results.

AP@X

This evaluation measure reports the average precision obtained once a certain number of images has been encountered. We have computed the scores for values ranging from 10 to 100 in steps of 10, although your primary focus should be on AP@10, AP@20 and AP@100.

Results

In the table below we present the results for the runs, organized alphabetically by team name. You can download the detailed results here. For future reference you can download the Java-based evaluation code below; note that the code may not look as simple as MATLAB code that hides away many of the details, but it was written with legibility in mind. We will release the ground truth on the first day of the CLEF conference, so afterwards you will be able to apply the code to your own runs.