Plant Identification 2013

Context

If agricultural development is to be successful and biodiversity is to be conserved, then accurate knowledge of the identity, geographic distribution and uses of plants is essential. Unfortunately, such basic information is often only partially available for professional stakeholders, teachers, scientists and citizens, and often incomplete for ecosystems that possess the highest plant diversity. So that simply identifying plant species is usually a very difficult task, even for professionals (such as farmers or wood exploiters) or for the botanists themselves. Using image retrieval technologies is nowadays considered by botanists as a promising direction in reducing this taxonomic gap. Evaluating recent advances of the IR community on this challenging task might therefore have a strong impact. The organization of this task is funded by the French project Pl@ntNet (INRIA, CIRAD, Telabotanica) and supported by the European Coordination Action CHORUS+.

Task Overview

Following the success of ImageCLEF 2011 and ImageCLEF 2012 plant identification tasks, we are glad to organize this year a new challenge dedicated to botanical data. This year, the task will be focused on tree and herb species identification, based on different types of images. Main novelties compared to last year are the following:
- more species: the number of species will be this year about 250, which is an important step towards covering the entire flora of a given region.
- multi-view plant retrieval vs. leaf-based retrieval: query and test pictures will now cover different organs (or views) of the individual plants and not only their leaves.

The leaf is actually far from being the only useful organ for accurate identification. As an example, the 6 species depicted in the figure below share the same French common name of laurel ("laurier") in spite of even though they belong to different taxonomic groups (5 genera, 5 families). In this case, while it is difficult to identify the species with more or less the same-sized elliptic shaped leaves, it is indisputably easier with the flowers.

6 different species sharing the same common name

The training and test data will be composed of images collected through a citizen sciences initiative that was initiated 2 years ago in collaboration with Telabotanica (social network of amateur and expert botanists). This makes the task closer to the conditions of a real-world application: (i) images of the same species are coming from distinct trees living in distinct areas (ii) pictures are taken by different users that might not used the same protocol to acquire the images (iii) pictures are taken at different periods in the year.

Additional information will include contextual meta-data (author, date, locality name) and some EXIF data.

Dataset

The task will be based on the Pl@ntView dataset which focuses on 250 herb and tree species from France area. It contains 26077 pictures belonging each to one of the 2 following categories:
- SheetAsBackground (or uniform background) (42%): exclusively pictures of leaves in front of a white or colored uniform background produced with a scanner or a camera with a sheet.
- NaturalBackground (for most of the time cluttered natural background") (58%): free natural photographs of different views on different subparts of a plant into the wild.

These 2 main categories can be subdivided into 2 and 5 sub-categories:
- SheetAsBackground Scan (22%): scan of a single leaf.
- SheetAsBackground Scan-like (22%): photograph of a single leaf in front of an uniform artificial background.
- NaturalBackground Leaf (16%): photograph of one leaf or more directly on the plant or near the plant on the floor or other non uniform background.
- NaturalBackground Flower (18%): photograph of one flower or a group of flowers (inflorescence) directly on the plant.
- NaturalBackground Fruit (8%): photograph of one fruit or a group of fruits (infructescence) directly on the plant.
- NaturalBackground Stem (8%): photograph of the stalk, or a stipe, or the bark of the trunk or a main branch of the plant.
- NaturalBackground Entire (8%): photograph of the entire plant, from the floor to the top.

The following figure provides examples of the 2 categories and 7 sub-categories:

These information of categories are reported into the meta-data, in a xml file (one per image) with explicit tags:
- Acquisition Type: "SheetAsBackground" for scan or scan-like pictures of a leaf, "NaturalBackground" for photograph with a cluttered background
- View Content: Leaf, Flower, Fruit, Stem, Entire

Another important additional information in the meta-data is IndividualPlantId, an unique identifier for one single individual plant observed by one same person the same day with the same device with the same lightening conditions. Each individual plant observed may thus be associated to several pictures. It means that these pictures are strongly connected, and may be very similar, even being near-duplicate pictures. As a consequence, pictures with the same IndividualPlantId can not be split across subsets during the identification task like training or test subsets, because they belong to one same indivisible event.

One individual-plant observed the same day by a same author involving several pictures with the same IndividualPlantId.

To sum up each image is associated with the following meta-data:
- Acquisition Type: Uniform ("SheetAsBackground") or Cluttered ("NaturalBackground"),
- View Content: Leaf, Flower, Fruit, Stem, Entire
- IndividualPlantId: the plant id from which several pictures may be associated (see just above-written)
- Taxon: the full taxon names (species, genus, family…) following as possible the most recent and complet information
- ClassId: the class label that must be used as ground-truth. It is a non-official short name of the taxon used for easily designating a taxon (most of the time the species and genus names without the name of the author of the taxon)
- VernacularName: the english common name(s) ,
- Date
- Locality: locality name (a district or a country division or a regions),
- GPSLocality: GPS coordinates of the locality where the plant was observed,
- Author: name of the author of the picture,
- Organization: name of the organization of the author.
And if the image was included in previous plant task:
- Year: ImageCLEF2011 or ImageCLEF2012,
- IndividualPlantId2012: the plant id used in 2012,
- ImageID2012: the image id.jpg used in 2012.

We provide here a set of 2 images and associated xml files (click on a picture):

Partial meta-data information can be found in the image's EXIF, and might include:
- the camera or the scanner model,
- the image resolutions and the dimensions,
- for photos, the optical parameters, the white balance, the light measures…

All data are published under a creative commons license.

Localities in the ImageCLEF 2013 Plant Task dataset

Task description

The task will be evaluated as a plant species retrieval task. Test images will be divided into two subtasks following the two main category SheetAsBackground and NaturalBackground involving distinct scores like as previous years.

Goal

The goal of the task is to retrieve the correct plant species among the top k species of a ranked list of returned species for each test image. Each participant is allowed to submit up to 4 runs built from different methods. Semi-supervised and interactive approaches, particularly for segmenting leaves from the background, are allowed but will be compared independently from fully automatic methods. Any human assistance in the processing of the test queries has therefore to be signaled in the submitted runs (see next section on how to do that).

training and test data

A part of Pl@ntView dataset will be provided as training data whereas the remaining part will be used later as test data. Training AND test pictures of leaves used during ImageCLEF 2012 art part of the Pl@ntView dataset and they will be systematically included in the training set. New scans and scan-like images of leaves will be introduced and used as test images. rFor the other new views of flower, fruit, stem and entire plant, test pictures will be chosen by randomly sampling 1/3 of the individual plants of each species and removing the pictures with less than 2 training images of the same view type and species in the training data.

- The training data finally results in 20985 images (9781 scans and scan-like pictures of leaf with a "SheetAsBackground", 11204 photographs of "natural background" more precisely 3522 "flower", 2080 "leaf", 1455 "entire", 1387 "fruit", 1337 "stem") with complete xml files associated to them. A ground-truth file listing all images of each species will be provided complementary. Download link of training data will be sent to participants on 29/01/2013.

- The test data results in 5092 images (1250 scans and scan-like pictures of leaf with a "SheetAsBackground", 3842 photographs of "natural background" more precisely 1233 "flower", 790 "leaf", 694 "entire", 520 "fruit", 605 "stem") with purged xml files (i.e without the taxon information that has to be predicted).

run format

The run file must be named as "teamname_runX.run" where X is the identifier of the run (i.e. 1,2 or 3). The run file has to contain as much lines as the total number of predictions, with at least one prediction per test image and a maximum of 250 predictions per test image (250 being the total number of species). Each prediction item (i.e. each line of the file) has to respect the following format :

<test_image_name.jpg;ClassId;rank;score>

The ClassId is the pair <Genus_name_without_author_name Species_name_without_author_name> and forms a unique identifier of the species. These strings have to respect the format provided in the ground-truth file provided with training set (i.e. the same format as the fields <ClassId> in the xml metadata files, see examples in previous section). <rank> is the ranking of a given species for a given test image. <Score> is a confidence score of a prediction item (the lower the score the lower the confidence). Here is a fake run example respecting this format:
myteam_run2.txt

The order of the prediction items (i.e. the lines of the run file) has no influence on the evaluation metric, so that contrary to our example prediction items might be sorted in any way. On the other side, the <rank> field is the most important one since it will be used as the main key to sort species and compute the final metric.

For each submitted run, please give in the submission system a description of the run. A combobox will specify wether the run was performed fully automatically or with a human assistance in the processing of the queries. Then, a textarea should contain a short description of the used method, particularly for helping differentiating the different runs submitted by the same group, for instance:
matching-based method using SIFT features, RANSAC algorithm and K-NN classifier with K=10
Optionally, you can add one or several bibtex reference(s) to publication(s) describing the method more in details.

metric

The primary metric used to evaluate the submitted runs will be a score related to the rank of the correct species in the list of retrieved species. Each test image will be attributed with a score between 0 and 1 : of 1 if the 1st returned species is correct and will decrease quickly while the rank of the correct species increases. An average score will then be computed on all test images. A simple mean on all test images would however introduce some bias. Indeed, we remind that the Pl@ntViews dataset was built in a collaborative manner. So that few contributors might have provided much more pictures than many other contributors who provided few. Since we want to evaluate the ability of a system to provide correct answers to all users, we rather measure the mean of the average classification rate per author. Furthermore, some authors sometimes provided many pictures of the same individual plant (to enrich training data with less efforts). Since we want to evaluate the ability of a system to provide the correct answer based on a single plant observation, we also have to average the classification rate on each individual plant. Finally, our primary metric is defined as the following average classification score S:

U : number of users (who have at least one image in the test data)
Pu : number of individual plants observed by the u-th user
Nu,p : number of pictures taken from the p-th plant observed by the u-th user
Su,p,n : score between 1 and 0 equals to the inverse of the rank of the correct species (for the n-th picture taken from the p-th plant observed by the u-th user)

Participants are allowed to train distinct classifiers, use different training subsets or use distinct methods for each data type.

How to register for the task

ImageCLEF has its own registration interface. Here you can choose a user name and a password. This registration interface is for example used for the submission of runs. If you already have a login from the former ImageCLEF benchmarks you can migrate it to ImageCLEF 2012 here

Schedule

• 15.12.2012: registration opens for all CLEF tasks
• 29.01.2013: training data release
• 18.03.2013: test data release
• 08.05.2013 deadline for submission of runs
• 15.05.2013: release of results
• 15.06.2013: deadline for submission of working notes
• 23-26.09.2013: CLEF 2013 Conference (Valencia)

Frequently asked questions

In the "test" dataset there are associated xml files where "type" and "content" attributes are indicated. Are we allowed to use this information during the prediction task or would it be considered as a manual intervention on the process.

Yes, you are allowed to use this information during the prediction (like in the two previous years). We consider that species identification is a very challenging task and we don't want to add more difficulties with an organ/view prediction step.

Results

A total of 12 groups submitted 33 runs. Thanks to all participants for their efforts and their constructive feedbacks regarding the organization.
Since several participants used distinct methods for the two image categories, we give here the results in two separate tables.

SheetAsBackground (scans and scan-like photos of single leaves)

The following table and graphic below present the scores obtained for the category "SheetAsBackground"
Click on the graphics to enlarge them.

NaturalBackground (photos of leaves, flowers, fruits, bark and entire plants)

The following table and graphic below present the scores obtained for the category "NaturalBackground".
Click on the graphics to enlarge them.

Detailed scores for each sub-categories of NaturalBackground: entire, flower, fruit, leaf, stem

The following table and graphic below present the detailed scores obtained for the sub-categories of "NaturalBackground" images. Remember that we use a specific metric weighted by authors and plants, and not by sub-categories, explaining why the NaturalBackground score is not the mean of the five scores of the sub-categories.

Click on the graphics to enlarge them.

Additional statistics and comments on these results will be provided in the overview working note of the task published within CLEF 2013.

_____________________________________________________________________________________________________________________

Contacts

Hervé Goeau (INRIA-ZENITH, INRIA-IMEDIA): herve(replace-that-by-a-dot)goeau(replace-that-by-an-arrobe)inria.fr
Alexis Joly (INRIA-ZENITH): alexis(replace-that-by-a-dot)joly(replace-that-by-an-arrobe)inria.fr
Pierre Bonnet (AMAP): pierre(replace-that-by-a-dot)bonnet(replace-that-by-an-arrobe)cirad.fr