The photo retrieval
task of ImageCLEF2008 will take a different approach to evaluation by
studying image clustering. A good image search engine ensures that duplicate or
near duplicate documents retrieved in response to a query are hidden from the
user. Ideally the top results of a ranked list will
contain diverse items representing different sub-topics within the results.
Providing this functionality is particularly important when a user types
in a query that is either poorly specified or ambiguous; a common query
in image search. Given such a query, a search engine that retrieves a diverse,
yet relevant set of images at the top of a ranked list
is more likely to satisfy its users [1,2].
Another reason why it's a
good idea to promote diversity is because often different people type in the
same query but wish to see different results. So if a search engine knows
nothing about the user entering the query, a good strategy for the engine is to
produce results that are both diverse (i.e.
representative of all sub-topics) and relevant, effectively the engine
is spreading its bets on what the user might want to retrieve.
This is an
The Task - Promote
Participants will run each provided topic on their image search system and produce a
ranking that in the top 20, holds as many relevant images that are
representative of the different sub-topics within the results.
The definition of what consitutes diversity will vary across the topics,
but there will be a clear indication in the topic (using a new topic tag, "cluster")
indicating what the clustering criteria the evaluators will use.
For each topic in the ImageCLEFPhoto set,
relevant images will be manually clustered into sub-topics and relevance
judgements will be augmented to indicate which
cluster an image belongs to. Relevance assessors will be instructed to look for
simple clusters based on the form of a topic. For example if a topic asks for
images of beaches in Brazil, clusters will be formed based on location; if a
topic asks for photos of animals, clusters will be formed based on animal type.
Participating groups will
return to us, for each topic, a ranked list of images IDs. We will determine which
images are relevant and count how many clusters are represented in the ranking.
We do not require you to identify or label clusters in the ranked list
how you choose to do the clustering is an internal matter for you.
Evaluation will be based
on two measures: precision at 20 and instance recall at rank 20 (also called
S-recall) , which calculates the
percentage of different clusters represented in the top 20. It will
be important to maximise both measures: simply getting lots of relevant images
from one cluster or filling the ranking with diverse, but non-relevant images,
will result in a poor overall effectiveness score.
Note, that it is quite
possible to submit runs from a "standard" non-clustering image search system,
though we would expect clustering systems to out-perform the standard
systems in producing a diverse ranked list in the top 20.
A version of the
collection will be made available that allows participants to explore cross
language aspects of image clustering. In this version, members of the clusters
will be captioned in two languages.
Topics download (note password protected).
We will use existing
topic statements of past ImageCLEFPhoto years. The topic format is the same as
previous years but with an additional tag, the cluster tag, which defines how
the clustering of images should take place.
<top> Note: this year we
will only offer topic languages in English.
<num> Number: 5 </num>
<title> animals swimming </title>
<image> 3739.jpg </image>
<image> 30823.jpg </image>
The collection is composed of a set of images (full size and thumbnails) and a set of annotations. Note site password protected. Annotations from the Visual Concept Detection Task of ImageCLEF 2008 are being made available at this site.
image collection of the
TC-12 photographic collection consists of 20,000 still natural images (plus
20,000 corresponding thumbnails) taken from locations around the world and
comprising an assorted cross-section of still natural images. This includes
pictures of different sports
and actions, photographs of people, animals, cities, landscapes and many other
aspects of contemporary life.
Each image is associated with an
alphanumeric caption stored in a semi-structured format. These captions include
the title of the image, its creation date, the location at which the photograph
was taken, the name of the photographer, a semantic description of the contents
of the image (as determined by the photographer) and additional notes.
<DESCRIPTION>two lane street with large shops on the
right and smaller shops on the left; people are walking on the sidewalk, some
are crossing the street; cars are parked along the left side of the street as
<NOTES>The main shopping street in
<DATE>March 2002 </DATE>
The collection will
be available in two forms:
about the image collection and links to related publications can be found
- All English, where all
titles, descriptions, locations, etc are written in the English
- A multi-lingual
version, where each image is annotated in a different language. The languages
used are English and German. (We hope to release this
collection by the end of April.)
How to Cheat (but please don't)
There is one way to cheat at this year's ImageCLEFPhoto, we're trusting you not to do any of these:
- Use last year's qrels: the topics this year are the same as last year, which means some of you might still have copies of last year's relevance judgements (the judgements we release this year will be clustering information on top of the old qrels). We are trusting you not use last year's relevance judgements for this year's runs.
Submission format and
When you submit your runs, please name your runs using the following code
elements. All submissions should be sent by email to Thomas Arni (firstname.lastname@example.org).
EN, RND (DE, EN)
IMG, TXT, TXTIMG
Used to specify the query language used in the run (this year English
Used to specify the
target language (i.e. the annotation set) used for the run: English (EN) or
describes the use of visual (image) or text features in your submission. A
text-only run will have modality text (TXT); a purely visual run will
have modality image (IMG) and a combined submission (e.g. initial text
search followed by a possibly combined visual search) will have modality
We distinguish between manual (MAN) and automatic (AUTO) submissions.
Automatic runs will involve no user interaction; whereby manual runs are those
in which a human has been involved in query construction and the iterative
retrieval process, e.g. manual relevance feedback is performed. We encourage
groups who want to investigate manual intervention further to participate in
the interactive evaluation (iCLEF)
submit a run in any of the permutations detailed in the previous table (above)
It is extremely
important that we can get a description of the techniques used for each
submitted run. This should be as detailed as possible to ease the comparison or
classification of techniques and results.
- EN-EN-AUTO-TXTIMG for the
English-English monolingual run using fully automatic text and image clustering methods
- EN-EN-MAN-TXT for the
English-English monolingual run using text clustering methods using some manual intervention
- EN-RND-AUTO-IMG for the
English-Random language run using fully automated image clustering methods
Participants are required to submit ranked lists
of (up to) the top 1000 images ranked in descending order of similarity (i.e.
the highest nearer the top of the list). The format of submissions for this
ad-hoc task can be found
and the filenames should distinguish different types of submission according to
the table above. Participants can submit (via email) as many system runs as
that there should be at least 1 document entry in your results for each topic
(i.e. if your system returns no results for a query then insert a dummy
entry, e.g. 25 1 16/16019 0 4238 xyzT10af5 ). The reason for this is to make
sure that all systems are compared with the same number of topics and relevant
documents. Submissions not following the required format will not be evaluated.
22 April 2008:
Data and Topic Release
UPDATE : Extended to 22 June 2008:
Submission of retrieval runs due
15 July 2008:
Release of retrieval results
15 August 2008:
Workshop papers due
17-19 September 2008:
CLEF workshop in Aarhus, Denmark
Department of Information Studies, University of Sheffield, UK
Mark Sanderson, Department
of Information Studies, University of Sheffield, UK
Paul Clough, Department of Information
Studies, University of Sheffield, UK
Michael Grubinger, School
of Computer Science and Mathematics, Victoria University, Australia
set up a mailing list: email@example.com for participants. Please
contact Paul Clough to be added to the list.
and Karger, D. R. 2006. Less is more: probabilistic models for retrieving fewer
relevant documents. In Proceedings of the 29th Annual international ACM SIGIR
Conference on Research and Development in information Retrieval (Seattle,
Washington, USA, August 06 - 11, 2006). SIGIR '06. ACM, New York, NY,
K., Tian, Y., Gao, W., and Huang, T. 2006. Diversifying the image retrieval
results. In Proceedings of the 14th Annual ACM international Conference on
Multimedia (Santa Barbara, CA, USA, October 23 - 27, 2006). MULTIMEDIA '06.
ACM, New York, NY, 707-710.
X., Cohen, W. W., and Lafferty, J. 2003. Beyond independent relevance: methods
and evaluation metrics for subtopic retrieval. In Proceedings of the 26th
Annual international ACM SIGIR Conference on Research and Development in
informaion Retrieval (Toronto, Canada, July 28 - August 01, 2003). SIGIR '03.
ACM, New York, NY, 10-17.
||Grubinger, M., Clough, P., Müller, H. and Deselaers, T. (2006),
The IAPR TC-12 Benchmark: A New Evaluation Resource for Visual Information
Systems, In Proceedings of International Workshop OntoImage2006
Language Resources for Content-Based Image Retrieval, held in
conjuction with LREC'06, pages 13-23, Genoa, Italy, 22 May 2006 (pdf).