IAPR TC-12 Benchmark

1 Introduction and Overview

The image collection of the IAPR TC-12 Benchmark consists of 20,000 still natural images taken from locations around the world and comprising an assorted cross-section of still natural images. This includes pictures of different sports and actions, photographs of people, animals, cities, landscapes and many other aspects of contemporary life. Example images can be found in Section 2.

Each image is associated with a text caption in up to three different languages (English, German and Spanish) . These annotations are stored in a database which is managed by a benchmark administration system that allows the specification of parameters according to which different subsets of the image collection can be generated. Section 3 provides more information and an annotation example.

The IAPR TC-12 Benchmark is now available free of charge and without copyright restrictions. Information on how to access (and download) the complete benchmark as well as the resources used at ImageCLEFphoto 2006 - 2008 is given in Sections 4 and 5, while Section 6 provides links to related publications.

2 Collection Content

The 20,000 images are high quality, multi-object, colour photographs that have been chosen according to strict image selection rules (see [2] for more details). Here are a couple of example images of some chosen categories:

Sports

City pictures

Landscape shots

Animal pictures

People shots

Action shots

3 Image Annotations

Image ID: annotations/00/25.eng
Title: Plaza de Armas
Description: a yellow building with white columns in the background; two palm trees in front of the house; cars are parking in front of the house; a woman and a child are walking over the square;
Notes: The Plaza de Armas is one of the most visited places in Cochabamba. The locals are very proud of the colourful buildings.
Location: Cochabamba, Bolivia
Date: 1 February 2002
Originator: Michael Grubinger

4 Access and Download

The following archive contains the complete IAPR TC-12 Benchmark, which is now available free of charge and without any copyright restrictions:

http://www-i6.informatik.rwth-aachen.de/imageclef/resources/iaprtc12.tgz

This is the most updated version of the IAPR TC-12 Benchmark and should be used from researchers from now on. This archive thereby comprises:

• 20000 images
• 1000 additional images previously used in object annotation tasks and/or the MUSCLE live event
• all complete (full-text) annotations (English, German, Random)
• all light annotations (English, German, Spanish, Random), i.e. all annotation tags except for the description tag

Note:

In publications based on the IAPR TC-12 Benchmark and/or the use of its data or a subset thereof, please cite the following publication:

The IAPR Benchmark: A New Evaluation Resource for Visual Information Systems, Grubinger, Michael, Clough Paul D., Müller Henning, and Deselaers Thomas , International Conference on Language Resources and Evaluation, 24/05/2006, Genoa, Italy, (2006)

Additional information on this data is available from the PhD thesis of Michael Grubinger:

Michael Grubinger. Analysis and Evaluation of Visual Information Systems Performance. PhD Thesis. School of Computer Science and Mathematics, Faculty of Health, Engineering and Science, Victoria University, Melbourne, Australia, 2007.

The thesis is available here:

http://nla.gov.au/anbd.bib-an43036734
http://wallaby.vu.edu.au/adt-VVUT/public/adt-VVUT20080408.130459/index.html

5 ImageCLEFphoto resources 2006 - 2008

Subsets of the IAPR TC-12 Benchmark were used at ImageCLEFphoto evaluation campaign from 2006 to 2008. These subsets are now also available, together with the complete evaluation resources:

ImageCLEFphoto 2008

The dataset that was used at ImageCLEFphoto 2008 can be downloaded here:

http://www-i6.informatik.rwth-aachen.de/imageclef/08/photo/ImageCLEFphot...

This archive contains the images and complete annotations (English, German, Random) used for evaluation. All other resources (topics, qrels, results, guidelines, overview paper, etc) will be added to this archive once they are finalised

ImageCLEFphoto 2007

The dataset that was used at ImageCLEFphoto 2007 can be downloaded here:

http://www-i6.informatik.rwth-aachen.de/imageclef/07/photo/ImageCLEFphot...

This archive contains the images, the light annotations (English, German, Spanish, Random) used for evaluation, the topics in 30 languages including three sample images which are not part of the image set used for evaluation, qrels, results, guidelines and the overview paper.

ImageCLEFphoto 2006

The dataset that was used at ImageCLEFphoto 2006 can be downloaded here:

http://www-i6.informatik.rwth-aachen.de/imageclef/06/photo/ImageCLEFphot...

This archive thereby contains the images, the incomplete annotations (English, German) used for evaluation, the complete annotations (English, German) used for relevance assessments, the topics in 30 languages, qrels, results, guidelines and the overview paper.

6 Related Publications

[1] Clement H.C. Leung, Horace Ip: Benchmarking for Content Based Visual Information Search. Proceedings of the Fourth International Conference on Visual Information Systems (VISUAL'2000), number 1929 in Lecture Notes in Computer Science, pages 442 - 456, Lyon, France. Springer Verlag.

[2] Michael Grubinger, Clement H. C. Leung: A Benchmark for Performance Calibration in Visual Information Search. Proceedings of The 2003 International Conference on Visual Information Systems (VIS 2003), pages 414 - 419, Miami, FL, USA, September 2003. Knowledge Systems Institute.

[3] Paul Over, Clement H. C. Leung, Horace Ip, Michael Grubinger: Multimedia Retrieval Benchmarks. Digital Multimedia on Demand, IEEE Multimedia April-June 2004, pages 80 - 84, 2004.

[4] Michael Grubinger, Clement H. C. Leung: Incremental Benchmark Development and Administration. Proceedings of The Seventh International Conference of Visual Information Systems (VIS'2004), pages 328 - 333, San Francisco, CA, USA, September 2004. Knowledge Systems Institute.

[5] Michael Grubinger, Clement H. C. Leung, Paul Clough: The IAPR Benchmark for Assessing Retrieval Performance in Cross Language Evaluation Tasks. Proceedings of the MUSCLE ImageCLEF Workshop on Image and Video Retrieval Evaluation, pages 33 - 50, Vienna, Austria, September 2005.

[6] Michael Grubinger, Paul D. Clough, Clement Leung: The IAPR TC-12 Benchmark for Visual Information Search. IAPR Newsletter April 2006, Volume 28, Number 2, pages 10 - 12, 2006.

[7] Michael Grubinger, Paul D. Clough, Henning Müller, Thomas Deselaers: The IAPR TC-12 Benchmark - A New Evaluation Resource for Visual Information Systems. Proceedings of the International Workshop OntoImage'2006 Language Resources for Content-Based Image Retrieval, held in conjunction with LREC'06, pages 13 - 23, Genoa, Italy, May 2006.

[8] Paul D. Clough, Henning Müller, Thomas Deselaers, Michael Grubinger, Thomas Lehmann, Jeffery Jensen, William Hersh: The CLEF 2005 Cross-Language Image Retrieval Track. Accessing Multilingual Information Repositories, volume 4022 of Lecture Notes in Computer Science (LNCS), pages 535 - 557, Vienna, Austria, September 2006. Springer.

[9] Paul D. Clough, Michael Grubinger, Thomas Deselaers, Allan Hanbury, Henning Müller: Overview of the ImageCLEF 2006 photographic retrieval and object annotation tasks. Evaluation of Multilingual and Multi-modal Information Retrieval, volume 4730 of Lecture Notes in Computer Science (LNCS), pages 579-594, Alicante, Spain, 2007. Springer.

[10] Michael Grubinger, Paul Clough: On the Creation of Query Topics for ImageCLEFphoto. Proceedings of the Third Workshop on Image and Video Retrieval Evaluation, pages 50-63, Budapest, Hungary, 2007.

[11] Hugo Jair Escalante, Manuel Montes, L. Enrique Sucar, Michael Grubinger: Towards a Region-Level Automatic Image Annotation Benchmark. Proceedings of the Third Workshop on Image and Video Retrieval Evaluation, pages 64-73, Budapest, Hungary, 2007.

[12] Michael Grubinger, Paul Clough, Allan Hanbury, Henning Müller: Overview of the ImageCLEFphoto 2007 photographic retrieval task. Advances in Multilingual and Multimodal Information Retrieval: 8th Workshop of the Cross-Language Evaluation Forum, CLEF 2007,
volume 5152 of Lecture Notes in Computer Science (LNCS) Budapest, Hungary, September 19-21, 2007.

[13] Michael Grubinger. "Analysis and Evaluation of Visual Information Systems Performance". PhD Thesis. School of Computer Science and Mathematics Faculty of Health, Engineering and Science Victoria University, Melbourne, Australia, 2007.