Personal Photo Retrieval 2013

Schedule

• 15/12/2012: Registration opens
• 31/01/2013: Document collection available
• 06/02/2013: Release of base line system
• 15/03/2013: Topics available
• 15/04/2013 -- 01/05/2013: Submission system for runs is open
• 25/05/2013: Submissions results released
• 15/06/2013: Deadline for submission of working notes papers
• 23/09/2013 -- 26/09/2013: CLEF 2013 Conference, Valencia, Spain

Motivation

Although obtaining large amounts of images from the web has become very easy, the actual contents of personal photo collections remain a blind spot of research. Personal photo collections contain a lot of duplicate images, and vary in quality and image content. To adress this issue, this year's retrieval task will concentrate on the retrieval from such a collection that has been sampled from real photographers with a demographic span that models a lifetime's photo collection. Besides the noisy content in a personal photo collection, an objective of this task is to find out whether the participating retrieval systems can exploit data from different search strategies, i.e., query-by-example and browsing data, in order to find both visual concepts (see Fig. 3) and photos depicting events (see Fig. 4). This discrimination is motivated by a conducted user study that was accompanying the collection of the personal photos.

Figure 3. Samples of the Visual Concept "Asian Temple Interior".

Figure 4. Samples of the Event Class "Rock Concert".

Overview

This year's subtask will extend the pilot task of 2012 with a focus on different usage scenarios and user groups. That is, the subtask will reveal if the tested algorithms are stable in terms of retrieval quality for different user groups. This becomes possible because each image's relevance has been judged by multiple assessors on a gradual scale. As of January 2013 the ground truths for the following user groups are available (the list might be extended):

1. Average (equivalent to the relevance assessments used in Cranfield-based experiments)
2. CBIR/MIR experts
3. CBIR/MIR laypersons
4. People with an IT background
5. People without an IT background
6. Female users
7. Male users

The subtask will be ad-hoc, i.e., no additional training data is released. The participants will get multiple QBE documents and/or browsing data and will have to find the best matching documents illustrating an event or depicting a visual concept.

Dataset

The data set is described in detail in a paper which is available below. It consists of 5,555 images plus rich metadata as they have found on the hard disks of the 19 contributors ranging from year of birth 1944 to 1985. Thus, one can interpret the content of the collection as a mirror of a photographer’s life span with typical changing usage behaviors, cameras, topics, and places.

 

Figure 4: Contribution by each Photographer (left); Global Distribution of Photographs (right)

Visual Features and Baseline System

We provide the following low-level visual features for the data set in order to improve the comparability of each submitted run. Participants are free to combine them freely or to use their own implementations.
Additionally, we will provide a baseline system for the participants that do not wish to concentrate on the implementation of feature extraction and similarity calculation. The system is implemented in C++ and can be compiled on Linux, Mac OS X, and Windows. The release of the system will be announced soon on this website.

1. Auto Color Correlogram
2. BIC
3. BRIEF (5 variants)
4. CEDD
5. Color Histogram
6. Color Layout (MPEG-7)
7. Color Structure (MPEG-7)
8. Contour Shape (MPEG-7)
9. Dominant Color (MPEG-7)
10. Edge Histogram (MPEG-7)
11. FCTH
12. Gabor
13. Region Shape (MPEG-7)
14. Scalable Color (MPEG-7)
15. SURF (5 variants)
16. SURF Hash (5 variants
17. Tamura

Using face detection features is not encouraged in this task because some faces had to be anonymized to preserve individual privacy rights. To compensate, a manually created Excel sheet with the number of depicted persons for each image is provided and can be used as face detection input. The intervals of depicted people and their frequency in the data set can be found in Fig. 7.

Figure 7: Number of People Depicted on Images

Building the Baseline System

The baseline system is licensed under the Apache License, Version 2.0 and subject to copyright 2010-2013 BTU Cottbus, Chair of Database and Information Systems. The full license text is included within the source code's archive.
It is known to compile with the following compilers:

• Mac OS X Snow Leopard and Lion: latest shipped gcc suite from Apple (we have no experience with the compilers from MacPorts etc.)
• gcc v4.4.5 (32 bit) and gcc v4.4.3 (64 bit)
• gcc v4.5 (mingw32) and Visual Studio 9 2008 x64

In order to build the system from scratch make sure that you have the following tools and libraries installed:

• Qt 4 (we have tested 4.7.0)
• CMake
• The shipped 3rd party libraries for your architecture that can be found in the lib/ subdirectory in the source code archive

Please note that we cannot provide assistance in building the system. If you follow the instructions in the readme files that come with the source, CMake will find all required libraries and tools. Also note that OpenCV is optional but will reduce the amount of available features. SQLlite is also not needed.

Recommended Build Steps

1. Ensure that your systems meets the aforementioned requirements.
2. Unzip the archive.
3. Navigate to the path containing the extracted files using a terminal.
4. Create a subdirectory called "build" at the same level as the src/ folder, e.g., using the command  .
5. Change to this directory:  
6. Run cmake and make it use the correct source, i.e.,  
7. Run make to start the build:  
8. Deploy the application using  

If you follow these steps you will find all binaries below the build/ directory in a folder called "applications". In order to run the binaries, make sure that your systems includes the build/libs/ subdirectory in the path for dynamic library/shared object look-up. You will also have to create an environment variable called "PYTHIA_PLUGIN_PATH" pointing to build/plugins/.

To obtain the baseline system, please contact the david.zellhoefer[at]tu-cottbus.de for a password.

Operating the System

There is no GUI for the system, it runs in a terminal. All tools have an online help. A short manual can be found at the bottom of this page.

Registering for the Task and Accessing the Data

To obtain the data set, please contact the david.zellhoefer[at]tu-cottbus.de for a password.

Topics

This year's subtask on personal photo retrieval consists of 74 topics. In contrast to the setup of the pilot task in 2012, the topics are no longer separated into visual concepts or events. Hence, you will have to find out which retrieval strategy will be appropriate based on the provided query data.

In order to retrieve the documents a user is looking for, the participants can rely on one QBE document and/or up to three browsed documents. For some topics, there are no QBE documents in order to model the following usage behavior: a user browsed a personal photo collection and toggled an action to show more similar images without stating an explicit preference. In contrast, whenever a QBE document is present, the user has deliberately chosen to search for more documents that resemble the QBE document. In this scenario, the browsed documents might give deeper insights into the user's current information need. The QBE documents have been assessed relevant by the assessors while some browsed document might be irrelevant or only relevant to a low degree.

A file containing all query data for each topic can be obtained via the link at the bottom of this page.

The topic file is structured as follows:

Topic ID;QBE;Browse -3;Browse -2;Browse -1
1;SL373313.jpg;SL373312.jpg;;
2;PICT5079.jpg;;;
3;090719_Schloss_Babelsberg_12.jpg;090719_Schloss_Babelsberg_15.jpg;;
4;;PICT0029.jpg;P1020728.jpg;P1020729.jpg

All fields are separated by a semicolon. The semantics of the fields from left to right are:

1. The topic ID
2. A QBE document (if available)
3. The 3rd last browsed document
4. The 2nd last browsed document
5. The last browsed document

Each document field refers to a file from the collection.

Participants are free to rely only on the provided QBE documents or to use data derived from the browsed documents as well. Please state on submission if you have used the browsing data. Participants are free to experiment with whatever methods they wish for image retrieval, e.g., relevance feedback or the integration of other modalities such as spatial or temporal information. We ask participants to indicate which of the following applies to each of their runs. Please note that you must not use the IPTC metadata fields that are present in the image documents.

Submission Details

The participants are requested to hand in their results in trec_eval format. After the publication of the results, the ground truths will be made available to the participants in trec_eval's QREL format for later usage.
The submissions will be received through the ImageCLEF 2013 system using subtask "ImageCLEFphoto-retrieval".
The trec_eval format is as follows:

42	Q0	542	1	0.427824	YourRunID
42	Q0	560	2	0.379514	YourRunID
42	Q0	506	3	0.373361	YourRunID
42	Q0	555	4	0.367622	YourRunID
42	Q0	538	5	0.356699	YourRunID
42	Q0	527	6	0.337737	YourRunID
42	Q0	577	7	0.337599	YourRunID
42	Q0	524	8	0.33359		YourRunID
42	Q0	587	9	0.325482	YourRunID
42	Q0	174	10	0.320099	YourRunID
...

All fields are seperated by a tab (\t) and every line has the following semantics (from left to right):

1. A topic ID which will be specified by the topics that have been released in March.
2. This field will be ignored and we recommend to leave it as shown above.
3. The file ID of the retrieved document out of an interval of [1,5555].
4. The rank of the retrieved document. Please make sure that the documents are ordered ascendingly.
5. The calculated similarity score of the retrieved document.
6. An arbitrary identifier that will help you to recognize the used retrieval technique.

The participants will be permitted to submit up to 5 runs.

The full submission details are available at this page.

Performance Measures

In order to reflect the gradual relevance scale for the topics, the retrieval performance evaluation will be primarily based on NDCG. More information will follow shortly.

Results

License

This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 Germany License.

Please cite this report for proper attribution:

@inproceedings{Zel12,
 author = {Zellh\"ofer, David},
 title = {An Extensible Personal Photograph Collection for Graded Relevance Assessments and User Simulation},
 pages = {to appear},
 publisher = {ACM},
 isbn = {978-1-4503-1329-2},
 series = {ICMR '12},
 booktitle = {{P}roceedings of the 2nd {A}{C}{M} {I}nternational {C}onference on {M}ultimedia {R}etrieval},
 year = {2012}
}

A technical report with similar content is available.

"© ACM, 2012. This is the author's version of the work. It is posted here by permission of ACM for
your personal use. Not for redistribution.
The definitive version was published in Proceedings of the 2nd ACM International Conference on Multimedia Retrieval,
{VOL#, ISS#, (DATE)} http://doi.acm.org/10.1145/nnnnnn.nnnnnn"

Organization

• David Zellhöfer, Brandenburg University of Technology, Germany, david.zellhoefer[at]tu-cottbus.de
• Workgroup of Database and Information Systems, Brandenburg University of Technology, Website

This research was supported by a grant of the Federal Ministry of Education and Research (Grant Number 03FO3072).