Interactive

Short Description

This task will be performed in a showcase event at the ICPR conference. Participants have to bring their systems and laptops and are asked to fulfill several image retrieval tasks. A database of text-annotated photographs and a few example topics will be made available to the participants beforehand. Participants have to register to ensure enough room, space etc. Participation can be limited depending on the amount of space that is made available.
The goal of this interactive session is to present working systems, see their performance and their limitations and give credit and visibility to groups that have a retrieval system rather than only simple scripts or unstable components. And the winning system is honored with a price.

Data Collection

The interactive task will use the Belga News collection . It contains ~500.000 images from Belga News Agency, which is an image search engine for news photographs. Each photograph will be up to a maximum of 512 pixels in either width or height, accompanied by a caption composed of English text up to a few sentences in length. The caption might contain date and place where image was taken.

An example image and its annotations is shown below:

 
 
BELGIUM DENDERMONDE COMMEMORATION KNIFE ATTACK

20090126 - DENDERMONDE, BELGIUM: Lots of people pictured during a commemoration for the victims of the knife attack in Sint-Gilles, Dendermonde, Belgium, on Monday 26 January 2009. Last friday 20-Year old Kim De Gelder killed three people, one adult and two childs, in a knife attack at the chidlren's day care center "Fabeltjesland" in Dendermonde. BELGA PHOTO BENOIT DOPPAGNE

The Setup

In the live event at the ICPR conference, the task is to perform interactive retrieval on a multimedia collection with visual and semantic topics that are not released before the event within a time limit. The collection will be released next month, so participants can start indexing and then testing their system. Later we will also provide some example topics.
For the event itself, we will have the systems, an (hopefully huge) audience from whom we can draw testers and a jury of experts. The volunteering testers - one for each system - should be unfamiliar with the systems and don't need to be experts in search systems and/or retrieval. To avoid bias due to the level of expertise, the testers switch to another system for the next topic. If the system owners do want to study the user behavior, they are allowed to sit next to the testers - but please do not use this for cheating :o) The jury, for now, consists of the organizers of the track, but we hope to employ others during the conference.
The evaluation will be twofolds:

• (1) usability: the user experience according to the testers, they're asked to rank all the systems that they worked with
• (2) precision: the quality of the result images shown on the final screen judged by the jury

Data Release and User Agreement

Before we can release the data to you next month, all participants have to sign an user agreement (attached at the end of the page) and send it back to us by mail, e-mail or fax till 15.6.2010. In case of e-mail and fax, we would like to ask you to give us the original form at the conference. Additionally, we would like you to send a short description of your retrieval system, especially which resources it is using (visual-only, text-only, multi media, others).

A common journal submission will assure visibility to the participants.

Important dates

• 15.6.2010 - Data for the interactive task is made available
• 15.7.2010 - Test topics for the interactive task are made available
• 23.8.-26.8.2010 - Conference – Interactive Retrieval Session

Contact person

• Jana Kludas, Computer Vision and Multimedia Laboratory, University of Geneva, Switzerland, Jana.Kludas@unige.ch
• Stefanie Nowak, Fraunhofer IDMT, Germany
• Henning Müller, University of Applied Sciences Western Switzerland, Sierre, and University of Geneva, Switzerland

How to register for the task

The online registration system is ready here.