ImageCLEF 2008: Photo Retrieval Task

The photo retrieval task of ImageCLEF2008 will take a different approach to evaluation by studying image clustering. A good image search engine ensures that duplicate or near duplicate documents retrieved in response to a query are hidden from the user. Ideally the top results of a ranked list will contain diverse items representing different sub-topics within the results. Providing this functionality is particularly important when a user types in a query that is either poorly specified or ambiguous; a common query in image search. Given such a query, a search engine that retrieves a diverse, yet relevant set of images at the top of a ranked list is more likely to satisfy its users [1,2].

Another reason why it's a good idea to promote diversity is because often different people type in the same query but wish to see different results. So if a search engine knows nothing about the user entering the query, a good strategy for the engine is to produce results that are both diverse (i.e. representative of all sub-topics) and relevant, effectively the engine is spreading its bets on what the user might want to retrieve.

This is an ImageCLEF task.

Participants will run each provided topic on their image search system and produce a ranking that in the top 20, holds as many relevant images that are representative of the different sub-topics within the results. The definition of what consitutes diversity will vary across the topics, but there will be a clear indication in the topic (using a new topic tag, "cluster") indicating what the clustering criteria the evaluators will use.

For each topic in the ImageCLEFPhoto set, relevant images will be manually clustered into sub-topics and relevance judgements will be augmented to indicate which cluster an image belongs to. Relevance assessors will be instructed to look for simple clusters based on the form of a topic. For example if a topic asks for images of beaches in Brazil, clusters will be formed based on location; if a topic asks for photos of animals, clusters will be formed based on animal type.

Participating groups will return to us, for each topic, a ranked list of images IDs. We will determine which images are relevant and count how many clusters are represented in the ranking. We do not require you to identify or label clusters in the ranked list how you choose to do the clustering is an internal matter for you.

Evaluation will be based on two measures: precision at 20 and instance recall at rank 20 (also called S-recall) [3], which calculates the percentage of different clusters represented in the top 20. It will be important to maximise both measures: simply getting lots of relevant images from one cluster or filling the ranking with diverse, but non-relevant images, will result in a poor overall effectiveness score.

Note, that it is quite possible to submit runs from a "standard" non-clustering image search system, though we would expect clustering systems to out-perform the standard systems in producing a diverse ranked list in the top 20.

A version of the collection will be made available that allows participants to explore cross language aspects of image clustering. In this version, members of the clusters will be captioned in two languages.

Topics download (note password protected).

<top>
<num> Number: 5 </num>
<title> animals swimming </title>
<cluster>animal</cluster>
<narr> </narr>
<image> 3739.jpg </image>
<image> 4968.jpg </image>
<image> 30823.jpg </image>
</top>

<DOC> <DOCNO>annotations/00/60.eng</DOCNO> <TITLE>Palma </TITLE> <DESCRIPTION>two lane street with large shops on the right and smaller shops on the left; people are walking on the sidewalk, some are crossing the street; cars are parked along the left side of the street as well; </DESCRIPTION> <NOTES>The main shopping street in Paraguay; </NOTES> <LOCATION>Asunción, Paraguay </LOCATION> <DATE>March 2002 </DATE> <IMAGE>images/00/60.jpg </IMAGE> <THUMBNAIL>thumbnails/00/60.jpg </THUMBNAIL> </DOC>

The collection will be available in two forms:

• All English, where all titles, descriptions, locations, etc are written in the English language
• A multi-lingual version, where each image is annotated in a different language. The languages used are English and German. (We hope to release this collection by the end of April.)

There is one way to cheat at this year's ImageCLEFPhoto, we're trusting you not to do any of these:

• Use last year's qrels: the topics this year are the same as last year, which means some of you might still have copies of last year's relevance judgements (the judgements we release this year will be clustering information on top of the old qrels). We are trusting you not use last year's relevance judgements for this year's runs.

• EN-EN-AUTO-TXTIMG for the English-English monolingual run using fully automatic text and image clustering methods
• EN-EN-MAN-TXT for the English-English monolingual run using text clustering methods using some manual intervention
• EN-RND-AUTO-IMG for the English-Random language run using fully automated image clustering methods

Primary Contact

Thomas Arni, Department of Information Studies, University of Sheffield, UK