WikipediaMM 2009 results

Introduction

The results for the wikipediaMM task have been computed with the trec_eval tool (version 8.1). Please use this tool in combination with the sources provided below.

The submitted runs have been corrected (where necessary) so as to correpond to valid runs in the correct TREC format. The following corrections have been made:

• The runs comply with the TREC fomat as specified in the submission guidelines for the task
• When a topic contains an image example that is part of the wikipediaMM collection, this image is removed from the retrieval results, i.e., we are seeking relevant images that the users are not familiar with (as they are with the images they provided as examples).
• When an image is retrieved more than once for a given topic, only its highest ranking for that topic is kept and the rest are removed (and the ranks in the retrieval results are appropriately fixed).
• Ensure that each of the submitted runs has a unique name.

The relevance assessments (qrels) for the wikipediaMM 2009 topics can be found here.

Results

The evaluation results for all submitted runs in the wikipediaMM taks are listed below ranked by MAP. For the evaluation results for the runs submitted by each participant please select the appropriate group from the following list: cea     dcu     deuceng     iiit_hyd     lahc     sztaki     sinai     ualicante

	Participant	Run	Modality	Topic field(s)	FB/QE	MAP	P@10	P@20	R-prec.	Bpref	Number of retrieved documents	Number of relevant retrieved documents
1	deuceng	deuwiki2009_205	TXT	TITLE	QE	0.2397	0.4000	0.3133	0.2683	0.2191	43052	1351
2	deuceng	deuwiki2009_204	TXT	TITLE	QE	0.2375	0.4000	0.3111	0.2692	0.2170	39257	1351
3	deuceng	deuwiki2009_203	TXT	TITLE	QE	0.2375	0.4000	0.3111	0.2692	0.2170	43052	1351
4	deuceng	deuwiki2009_202	TXT	TITLE	QE	0.2358	0.3933	0.3189	0.2708	0.2217	43052	1352
5	lahc	run_TXTIMG_100_3_1_5_meanstd	TXTIMG	TITLEIMG_Q	NOFB	0.2178	0.3378	0.2811	0.2538	0.2006	44993	1213
6	lahc	run_TXTIMG_50_3_1_5_meanstd	TXTIMG	TITLEIMG_Q	NOFB	0.2148	0.3356	0.2867	0.2536	0.2023	44993	1218
7	cea	cealateblock	TXTIMG	TITLEIMG_Q	QE	0.2051	0.3622	0.2744	0.2388	0.1938	35453	1207
8	cea	ceaearlyblock	TXTIMG	TITLEIMG_Q	QE	0.2046	0.3556	0.2833	0.2439	0.2014	35112	1127
9	cea	ceaearlyblockres	TXTIMG	TITLEIMG_Q	QE	0.2046	0.3556	0.2833	0.2439	0.2014	35147	1127
10	cea	ceabofblock	TXTIMG	TITLEIMG_Q	QE	0.1975	0.3689	0.2789	0.2342	0.1886	30413	1183
11	cea	ceatlepblock	TXTIMG	TITLEIMG_Q	QE	0.1959	0.3467	0.2733	0.2236	0.1847	30413	1182
12	cea	ceabofblockres	TXTIMG	TITLEIMG_Q	QE	0.1949	0.3689	0.2789	0.2357	0.1890	35165	1131
13	cea	ceatlepblockres	TXTIMG	TITLEIMG_Q	QE	0.1934	0.3467	0.2733	0.2236	0.1847	35165	1131
14	lahc	run_TXTIMG_Siftdense_0.084	TXTIMG	TITLEIMG_Q	NOFB	0.1903	0.3111	0.2700	0.2324	0.1828	44993	1212
15	lahc	run_TXT_100_3_1_5	TXT	TITLE	NOFB	0.1890	0.2956	0.2544	0.2179	0.1687	38004	1205
16	lahc	run_TXT_50_3_1_5	TXT	TITLE	NOFB	0.1880	0.3000	0.2489	0.2145	0.1715	37041	1198
17	ualicante	Alicante-MMLCA	TXTIMG	TITLEIMG_Q	FB	0.1878	0.2733	0.2478	0.2138	0.1734	42009	1183
18	deuceng	deuwiki2009_201	TXT	TITLE	QE	0.1865	0.2978	0.2389	0.2146	0.1691	41242	1283
19	deuceng	deuwiki2009_200	TXT	TITLE	NOFB	0.1861	0.2956	0.2333	0.2133	0.1679	41242	1283
20	lahc	run_TXTIMG_Meanstd_0.025	TXTIMG	TITLEIMG_Q	NOFB	0.1845	0.3178	0.2422	0.2175	0.1679	44993	1208
21	lahc	run_TXTIMG_Sift_0.012	TXTIMG	TITLEIMG_Q	NOFB	0.1807	0.2978	0.2489	0.2047	0.1637	44995	1200
22	ualicante	Alicante-MMLCA_MIXED	TXTIMG	TITLEIMG_Q	FB	0.1806	0.2533	0.2356	0.2015	0.1582	44994	1260
23	ualicante	Alicante-MMPRF_MIXED	TXTIMG	TITLEIMG_Q	FB	0.1801	0.2644	0.2267	0.2103	0.1633	44994	1233
24	lahc	run_TXTIMG_Meanstd_0.015	TXTIMG	TITLEIMG_Q	NOFB	0.1792	0.2867	0.2411	0.2133	0.1618	44993	1213
25	ualicante	Alicante-Baseline	TXT	TITLE	NOFB	0.1784	0.2556	0.2289	0.2049	0.1575	29062	1218
26	dcu	DCUTFIDF-DBpediaMetadata-QE	TXT	TITLE	NOFB	0.1752	0.2578	0.2189	0.2018	0.1580	44597	1237
27	ualicante	Alicante-PRF	TXT	TITLE	FB	0.1745	0.2511	0.2133	0.1985	0.1506	44994	1244
28	ualicante	Alicante-LCA	TXT	TITLE	FB	0.1745	0.2511	0.2133	0.1985	0.1506	44994	1244
29	sztaki	bp_acad_txt4_min_txtimg	TXTIMG	TITLEIMG_Q	NOFB	0.1699	0.2867	0.2389	0.2161	0.1629	40788	1169
30	ualicante	Alicante-MMPRF	TXTIMG	TITLEIMG_Q	FB	0.1697	0.2422	0.2178	0.2000	0.1512	42736	1211
31	dcu	DCUTFIDF-DBpediaQE	TXT	TITLE	QE	0.1685	0.2600	0.2144	0.1963	0.1555	44319	1166
32	sztaki	bp_acad_txt4	TXT	TITLE	NOFB	0.1676	0.2911	0.2411	0.2108	0.1606	40788	1169
33	lahc	LaHC_TXT_tfidf	TXT	TITLE	NOFB	0.1667	0.2733	0.2200	0.1989	0.1496	35611	1192
34	dcu	DCUTFIDF-DBpediaMetadata-DBpediaQE	TXT	TITLE	QE	0.1641	0.2378	0.2078	0.1897	0.1459	44994	1235
35	sztaki	bp_acad_txt6_min_txtimg	TXTIMG	TITLEIMG_Q	NOFB	0.1604	0.2778	0.2200	0.2010	0.1517	31614	1138
36	cea	ceatxt	TXT	TITLE	QE	0.1604	0.2333	0.2022	0.1915	0.1417	35453	1130
37	ualicante	Alicante-RRTFIDF	TXTIMG	TITLEIMG_Q	NOFB	0.1592	0.2400	0.2200	0.1994	0.1475	29086	1219
38	sztaki	bp_acad_txt6	TXT	TITLE	NOFB	0.1583	0.2667	0.2122	0.1927	0.1476	31614	1138
39	sztaki	bp_acad_txt5_min_txtimg	TXTIMG	TITLEIMG_Q	NOFB	0.1581	0.2756	0.2356	0.2076	0.1536	44025	1135
40	dcu	DCUTFIDF-baseline	TXT	TITLE	NOFB	0.1576	0.2600	0.2089	0.1883	0.1485	28654	1092
41	sinai	sinai_NTWn_T	TXT	TITLE	NOFB	0.1566	0.2556	0.2167	0.1951	0.1540	44995	1017
42	sztaki	bp_acad_txt5	TXT	TITLE	NOFB	0.1561	0.2622	0.2356	0.2048	0.1503	44025	1135
43	sinai	sinai_NT_T	TXT	TITLE	NOFB	0.1538	0.2333	0.2044	0.1809	0.1472	44995	1092
44	lahc	LaHC_TXT_okapi	TXT	TITLE	NOFB	0.1432	0.2444	0.1967	0.1845	0.1344	35611	1164
45	sinai	sinai_NT_TWn	TXT	TITLE	QE	0.1275	0.2000	0.1567	0.1466	0.1142	44995	1048
46	ualicante	Alicante-MMLCA+RRTFIDF	TXTIMG	TITLEIMG_Q	FB	0.1222	0.1533	0.1422	0.1363	0.0989	42057	1200
47	cea	cealatewindow	TXTIMG	TITLEIMG_Q	QE	0.1182	0.2267	0.1889	0.1518	0.1129	35453	1130
48	cea	ceaearlywindow	TXTIMG	TITLEIMG_Q	QE	0.1178	0.2267	0.1900	0.1506	0.1119	35453	1130
49	sinai	sinai_NTWn_TWn	TXT	TITLE	QE	0.0998	0.1800	0.1433	0.1326	0.0971	44996	867
50	cea	ceabofwindow	TXTIMG	TITLEIMG_Q	QE	0.0944	0.1889	0.1544	0.1260	0.0883	35453	1130
51	cea	ceatlepwindow	TXTIMG	TITLEIMG_Q	QE	0.0921	0.1800	0.1633	0.1265	0.0920	35453	1130
52	lahc	run_inter_TXT_IMG_Siftdense	TXTIMG	TITLEIMG_Q	NOFB	0.0365	0.1267	0.0922	0.0544	0.0503	619	142
53	lahc	run_inter_TXT_IMG_Meanstd	TXTIMG	TITLEIMG_Q	NOFB	0.0338	0.1356	0.0800	0.0478	0.0443	574	76
54	lahc	run_inter_TXT_IMG_Sift	TXTIMG	TITLEIMG_Q	NOFB	0.0321	0.1044	0.0644	0.0423	0.0393	637	120
55	iiit_hyd	iiithr1	TXT	TITLE	NOFB	0.0186	0.0689	0.0389	0.0246	0.0220	618	98
56	dcu	DCUIMG	IMG	IMG_Q	NOFB	0.0079	0.0244	0.0222	0.0229	0.0158	44995	144
57	sztaki	bp_acad_pure_min_IMG	IMG	IMG_Q_	NOFB	0.0068	0.0244	0.0144	0.0130	0.0101	44993	80

cea

The corrected runs submitted by cea for the wikipediaMM task are: submissions_cea.tar.gz. These are the evaluation results for these runs as produced by trec_eval: summary results over all topics results for each of the topics Interpolated Recall Precision Averages at eleven standard recall levels cea Summary statistics for the runs submitted by cea sorted by MAP Participant Run Modality Topic field(s) FB/QE MAP P@10 P@20 R-prec. Bpref Number of retrieved documents Number of relevant retrieved documents 7 cea cealateblock TXTIMG TITLEIMG_Q QE 0.2051 0.3622 0.2744 0.2388 0.1938 35453 1207 8 cea ceaearlyblock TXTIMG TITLEIMG_Q QE 0.2046 0.3556 0.2833 0.2439 0.2014 35112 1127 9 cea ceaearlyblockres TXTIMG TITLEIMG_Q QE 0.2046 0.3556 0.2833 0.2439 0.2014 35147 1127 10 cea ceabofblock TXTIMG TITLEIMG_Q QE 0.1975 0.3689 0.2789 0.2342 0.1886 30413 1183 11 cea ceatlepblock TXTIMG TITLEIMG_Q QE 0.1959 0.3467 0.2733 0.2236 0.1847 30413 1182 12 cea ceabofblockres TXTIMG TITLEIMG_Q QE 0.1949 0.3689 0.2789 0.2357 0.1890 35165 1131 13 cea ceatlepblockres TXTIMG TITLEIMG_Q QE 0.1934 0.3467 0.2733 0.2236 0.1847 35165 1131 36 cea ceatxt TXT TITLE QE 0.1604 0.2333 0.2022 0.1915 0.1417 35453 1130 47 cea cealatewindow TXTIMG TITLEIMG_Q QE 0.1182 0.2267 0.1889 0.1518 0.1129 35453 1130 48 cea ceaearlywindow TXTIMG TITLEIMG_Q QE 0.1178 0.2267 0.1900 0.1506 0.1119 35453 1130 50 cea ceabofwindow TXTIMG TITLEIMG_Q QE 0.0944 0.1889 0.1544 0.1260 0.0883 35453 1130 51 cea ceatlepwindow TXTIMG TITLEIMG_Q QE 0.0921 0.1800 0.1633 0.1265 0.0920 35453 1130

dcu

The corrected runs submitted by dcu for the wikipediaMM task are: submissions_dcu.tar.gz. These are the evaluation results for these runs as produced by trec_eval: summary results over all topics results for each of the topics Interpolated Recall Precision Averages at eleven standard recall levels dcu Summary statistics for the runs submitted by dcu sorted by MAP Participant Run Modality Topic field(s) FB/QE MAP P@10 P@20 R-prec. Bpref Number of retrieved documents Number of relevant retrieved documents 26 dcu DCUTFIDF-DBpediaMetadata-QE TXT TITLE NOFB 0.1752 0.2578 0.2189 0.2018 0.1580 44597 1237 31 dcu DCUTFIDF-DBpediaQE TXT TITLE QE 0.1685 0.2600 0.2144 0.1963 0.1555 44319 1166 34 dcu DCUTFIDF-DBpediaMetadata-DBpediaQE TXT TITLE QE 0.1641 0.2378 0.2078 0.1897 0.1459 44994 1235 40 dcu DCUTFIDF-baseline TXT TITLE NOFB 0.1576 0.2600 0.2089 0.1883 0.1485 28654 1092 56 dcu DCUIMG IMG IMG_Q NOFB 0.0079 0.0244 0.0222 0.0229 0.0158 44995 144

deuceng

The corrected runs submitted by deuceng for the wikipediaMM task are: submissions_deuceng.tar.gz. These are the evaluation results for these runs as produced by trec_eval: summary results over all topics results for each of the topics Interpolated Recall Precision Averages at eleven standard recall levels deuceng Summary statistics for the runs submitted by deuceng sorted by MAP Participant Run Modality Topic field(s) FB/QE MAP P@10 P@20 R-prec. Bpref Number of retrieved documents Number of relevant retrieved documents 1 deuceng deuwiki2009_205 TXT TITLE QE 0.2397 0.4000 0.3133 0.2683 0.2191 43052 1351 2 deuceng deuwiki2009_204 TXT TITLE QE 0.2375 0.4000 0.3111 0.2692 0.2170 39257 1351 3 deuceng deuwiki2009_203 TXT TITLE QE 0.2375 0.4000 0.3111 0.2692 0.2170 43052 1351 4 deuceng deuwiki2009_202 TXT TITLE QE 0.2358 0.3933 0.3189 0.2708 0.2217 43052 1352 18 deuceng deuwiki2009_201 TXT TITLE QE 0.1865 0.2978 0.2389 0.2146 0.1691 41242 1283 19 deuceng deuwiki2009_200 TXT TITLE NOFB 0.1861 0.2956 0.2333 0.2133 0.1679 41242 1283

iiit_hyd

The corrected runs submitted by iiit_hyd for the wikipediaMM task are: submissions_iiit_hyd.tar.gz. These are the evaluation results for these runs as produced by trec_eval: summary results over all topics results for each of the topics Interpolated Recall Precision Averages at eleven standard recall levels iiit_hyd Summary statistics for the runs submitted by iiit_hyd sorted by MAP Participant Run Modality Topic field(s) FB/QE MAP P@10 P@20 R-prec. Bpref Number of retrieved documents Number of relevant retrieved documents 55 iiit_hyd iiithr1 TXT TITLE NOFB 0.0186 0.0689 0.0389 0.0246 0.0220 618 98

lahc

The corrected runs submitted by lahc for the wikipediaMM task are: submissions_lahc.tar.gz. These are the evaluation results for these runs as produced by trec_eval: summary results over all topics results for each of the topics Interpolated Recall Precision Averages at eleven standard recall levels lahc Summary statistics for the runs submitted by lahc sorted by MAP Participant Run Modality Topic field(s) FB/QE MAP P@10 P@20 R-prec. Bpref Number of retrieved documents Number of relevant retrieved documents 5 lahc run_TXTIMG_100_3_1_5_meanstd TXTIMG TITLEIMG_Q NOFB 0.2178 0.3378 0.2811 0.2538 0.2006 44993 1213 6 lahc run_TXTIMG_50_3_1_5_meanstd TXTIMG TITLEIMG_Q NOFB 0.2148 0.3356 0.2867 0.2536 0.2023 44993 1218 14 lahc run_TXTIMG_Siftdense_0.084 TXTIMG TITLEIMG_Q NOFB 0.1903 0.3111 0.2700 0.2324 0.1828 44993 1212 15 lahc run_TXT_100_3_1_5 TXT TITLE NOFB 0.1890 0.2956 0.2544 0.2179 0.1687 38004 1205 16 lahc run_TXT_50_3_1_5 TXT TITLE NOFB 0.1880 0.3000 0.2489 0.2145 0.1715 37041 1198 20 lahc run_TXTIMG_Meanstd_0.025 TXTIMG TITLEIMG_Q NOFB 0.1845 0.3178 0.2422 0.2175 0.1679 44993 1208 21 lahc run_TXTIMG_Sift_0.012 TXTIMG TITLEIMG_Q NOFB 0.1807 0.2978 0.2489 0.2047 0.1637 44995 1200 24 lahc run_TXTIMG_Meanstd_0.015 TXTIMG TITLEIMG_Q NOFB 0.1792 0.2867 0.2411 0.2133 0.1618 44993 1213 33 lahc LaHC_TXT_tfidf TXT TITLE NOFB 0.1667 0.2733 0.2200 0.1989 0.1496 35611 1192 44 lahc LaHC_TXT_okapi TXT TITLE NOFB 0.1432 0.2444 0.1967 0.1845 0.1344 35611 1164 52 lahc run_inter_TXT_IMG_Siftdense TXTIMG TITLEIMG_Q NOFB 0.0365 0.1267 0.0922 0.0544 0.0503 619 142 53 lahc run_inter_TXT_IMG_Meanstd TXTIMG TITLEIMG_Q NOFB 0.0338 0.1356 0.0800 0.0478 0.0443 574 76 54 lahc run_inter_TXT_IMG_Sift TXTIMG TITLEIMG_Q NOFB 0.0321 0.1044 0.0644 0.0423 0.0393 637 120

sztaki

The corrected runs submitted by sztaki for the wikipediaMM task are: submissions_sztaki.tar.gz. These are the evaluation results for these runs as produced by trec_eval: summary results over all topics results for each of the topics Interpolated Recall Precision Averages at eleven standard recall levels sztaki Summary statistics for the runs submitted by sztaki sorted by MAP Participant Run Modality Topic field(s) FB/QE MAP P@10 P@20 R-prec. Bpref Number of retrieved documents Number of relevant retrieved documents 29 sztaki bp_acad_txt4_min_txtimg TXTIMG TITLEIMG_Q NOFB 0.1699 0.2867 0.2389 0.2161 0.1629 40788 1169 32 sztaki bp_acad_txt4 TXT TITLE NOFB 0.1676 0.2911 0.2411 0.2108 0.1606 40788 1169 35 sztaki bp_acad_txt6_min_txtimg TXTIMG TITLEIMG_Q NOFB 0.1604 0.2778 0.2200 0.2010 0.1517 31614 1138 38 sztaki bp_acad_txt6 TXT TITLE NOFB 0.1583 0.2667 0.2122 0.1927 0.1476 31614 1138 39 sztaki bp_acad_txt5_min_txtimg TXTIMG TITLEIMG_Q NOFB 0.1581 0.2756 0.2356 0.2076 0.1536 44025 1135 42 sztaki bp_acad_txt5 TXT TITLE NOFB 0.1561 0.2622 0.2356 0.2048 0.1503 44025 1135 57 sztaki bp_acad_pure_min_IMG IMG IMG_Q_ NOFB 0.0068 0.0244 0.0144 0.0130 0.0101 44993 80

sinai

The corrected runs submitted by sinai for the wikipediaMM task are: submissions_sinai.tar.gz. These are the evaluation results for these runs as produced by trec_eval: summary results over all topics results for each of the topics Interpolated Recall Precision Averages at eleven standard recall levels sinai Summary statistics for the runs submitted by sinai sorted by MAP Participant Run Modality Topic field(s) FB/QE MAP P@10 P@20 R-prec. Bpref Number of retrieved documents Number of relevant retrieved documents 41 sinai sinai_NTWn_T TXT TITLE NOFB 0.1566 0.2556 0.2167 0.1951 0.1540 44995 1017 43 sinai sinai_NT_T TXT TITLE NOFB 0.1538 0.2333 0.2044 0.1809 0.1472 44995 1092 45 sinai sinai_NT_TWn TXT TITLE QE 0.1275 0.2000 0.1567 0.1466 0.1142 44995 1048 49 sinai sinai_NTWn_TWn TXT TITLE QE 0.0998 0.1800 0.1433 0.1326 0.0971 44996 867

ualicante

The corrected runs submitted by ualicante for the wikipediaMM task are: submissions_ualicante.tar.gz. These are the evaluation results for these runs as produced by trec_eval: summary results over all topics results for each of the topics Interpolated Recall Precision Averages at eleven standard recall levels ualicante Summary statistics for the runs submitted by ualicante sorted by MAP Participant Run Modality Topic field(s) FB/QE MAP P@10 P@20 R-prec. Bpref Number of retrieved documents Number of relevant retrieved documents 17 ualicante Alicante-MMLCA TXTIMG TITLEIMG_Q FB 0.1878 0.2733 0.2478 0.2138 0.1734 42009 1183 22 ualicante Alicante-MMLCA_MIXED TXTIMG TITLEIMG_Q FB 0.1806 0.2533 0.2356 0.2015 0.1582 44994 1260 23 ualicante Alicante-MMPRF_MIXED TXTIMG TITLEIMG_Q FB 0.1801 0.2644 0.2267 0.2103 0.1633 44994 1233 25 ualicante Alicante-Baseline TXT TITLE NOFB 0.1784 0.2556 0.2289 0.2049 0.1575 29062 1218 27 ualicante Alicante-PRF TXT TITLE FB 0.1745 0.2511 0.2133 0.1985 0.1506 44994 1244 28 ualicante Alicante-LCA TXT TITLE FB 0.1745 0.2511 0.2133 0.1985 0.1506 44994 1244 30 ualicante Alicante-MMPRF TXTIMG TITLEIMG_Q FB 0.1697 0.2422 0.2178 0.2000 0.1512 42736 1211 37 ualicante Alicante-RRTFIDF TXTIMG TITLEIMG_Q NOFB 0.1592 0.2400 0.2200 0.1994 0.1475 29086 1219 46 ualicante Alicante-MMLCA+RRTFIDF TXTIMG TITLEIMG_Q FB 0.1222 0.1533 0.1422 0.1363 0.0989 42057 1200