In the first table the average EER and AUC are presented for each run in alphabetical order. The results of the best run for each group are listed below with EER and AUC for each concept.
Additionally we added a run called "random" which contains random numbers for all concepts and photos. You can find a pdf with detailed results and diagrams for all runs per group at the ftp. We decided to split the ROC curves into three diagrams per run to ensure readability.
For evaluation we used the matlab script provided by Thomas last year.
As a test we did two evaluations: One with all 53 concepts and a second one without the concepts "aesthetic impression", "fancy" and, "high grade of overall quality". As you can see from the result list of the groups, it did not change much in the order of the best runs. Here is the result list reporting the best run for each group with all concepts, and here the one without the 3 concepts mentioned before.
________________________________________________________________________________________________________
Run |
Average EER |
Average AUC |
apexlab_61_2_1245568188287.txt |
0.482693 |
0.070400 |
apexlab_61_2_1245569401060.txt |
0.488715 |
0.082151 |
apexlab_61_2_1245571723120.txt |
0.488715 |
0.082151 |
AVEIR_81_2_1245664594607.txt |
0.485345 |
0.520723 |
AVEIR_81_2_1245664654451.txt |
0.440589 |
0.550866 |
AVEIR_81_2_1245664755158.txt |
0.464311 |
0.552315 |
AVEIR_81_2_1245664820829.txt |
0.454191 |
0.566118 |
bpacad_18_2_1245621915717.txt |
0.346106 |
0.707860 |
bpacad_18_2_1245622065724.txt |
0.322632 |
0.733264 |
bpacad_18_2_1245622367725.txt |
0.296315 |
0.771324 |
bpacad_18_2_1245622473940.txt |
0.291718 |
0.773133 |
bpacad_18_2_1245623805085.txt |
0.304113 |
0.746300 |
CEA LIST_24_2_1245339305674.txt |
0.501514 |
0.460804 |
CEA LIST_24_2_1245339411928.txt |
0.500495 |
0.469035 |
CEA LIST_24_2_1245339505739.txt |
0.500535 |
0.456957 |
CEA LIST_24_2_1245339660021.txt |
0.504021 |
0.463241 |
CVIUI2R_22_2_1244628714641.txt |
0.253296 |
0.813893 |
CVIUI2R_22_2_1244629050173.txt |
0.255945 |
0.811421 |
FIR2_92_2_1245417583652.txt |
0.253572 |
0.817153 |
FIR2_92_2_1245516363637.txt |
0.253566 |
0.817159 |
FIR2_92_2_1245516525876.txt |
0.253685 |
0.816811 |
FIR2_92_2_1245516822169.txt |
0.271969 |
0.762276 |
IAM Southampton_30_2_1245438072355.txt |
0.330401 |
0.714825 |
IAM Southampton_30_2_1245519187248.txt |
0.333452 |
0.711566 |
IAM Southampton_30_2_1245519327555.txt |
0.342123 |
0.699769 |
INAOE TIA_66_2_1245620227762.txt |
0.486201 |
0.100839 |
INAOE TIA_66_2_1245630111443.txt |
0.484685 |
0.099306 |
INAOE TIA_66_2_1245631132743.txt |
0.492958 |
0.044276 |
INAOE TIA_66_2_1245631427651.txt |
0.492388 |
0.062220 |
INAOE TIA_66_2_1245631711558.txt |
0.487231 |
0.094741 |
ISIS University of Amsterdam_76_2_1245075326057.txt |
0.234476 |
0.838699 |
ISIS University of Amsterdam_76_2_1245075554764.txt |
0.234476 |
0.838699 |
ISIS University of Amsterdam_76_2_1245075919629.txt |
0.235479 |
0.837531 |
ISIS University of Amsterdam_76_2_1245076320558.txt |
0.252997 |
0.821731 |
ISIS University of Amsterdam_76_2_1245076767051.txt |
0.243547 |
0.830005 |
KameyamaLab_21_2_1245594455534.txt |
0.452374 |
0.164048 |
KameyamaLab_21_2_1245616956073.txt |
0.466948 |
0.127788 |
KameyamaLab_21_2_1245617358988.txt |
0.497610 |
0.066594 |
KameyamaLab_21_2_1245617609041.txt |
0.457482 |
0.152011 |
KameyamaLab_21_2_1245617818119.txt |
0.455052 |
0.156040 |
LEAR_44_2_1245581505805.txt |
0.258642 |
0.813300 |
LEAR_44_2_1245581860906.txt |
0.304383 |
0.756649 |
LEAR_44_2_1245581967963.txt |
0.273385 |
0.793920 |
LEAR_44_2_1245582143586.txt |
0.256169 |
0.804713 |
LEAR_44_2_1245582451309.txt |
0.249469 |
0.823105 |
LIP6_17_2_1245498965150.txt |
0.372169 |
0.673089 |
LIP6_17_2_1245579903668.txt |
0.383789 |
0.651276 |
LIP6_17_2_1245610812916.txt |
0.406509 |
0.629883 |
LIP6_17_2_1245617011478.txt |
0.498451 |
0.192313 |
LIP6_17_2_1245618383113.txt |
0.410032 |
0.622688 |
LSIS_14_2_1245595798047.txt |
0.481796 |
0.505016 |
LSIS_14_2_1245595947674.txt |
0.330819 |
0.720931 |
LSIS_14_2_1245596027969.txt |
0.488454 |
0.498070 |
LSIS_14_2_1245664294478.txt |
0.360759 |
0.685542 |
LSIS_14_2_1245664386697.txt |
0.428975 |
0.600487 |
MMIS_33_2_1245434554581.txt |
0.312366 |
0.744231 |
MMIS_33_2_1245586552541.txt |
0.352478 |
0.689410 |
MMIS_33_2_1245601239738.txt |
0.356945 |
0.684821 |
MMIS_33_2_1245611281967.txt |
0.352485 |
0.689407 |
MMIS_33_2_1245674693001.txt |
0.352612 |
0.689342 |
MRIM_50_2_1245573948330.txt |
0.442568 |
0.528129 |
MRIM_50_2_1245574356224.txt |
0.392492 |
0.592542 |
MRIM_50_2_1245574874836.txt |
0.439504 |
0.573579 |
MRIM_50_2_1245578233616.txt |
0.383630 |
0.643450 |
Random |
0.500280 |
0.499307 |
TELECOM ParisTech_39_2_1245415171956.txt |
0.527132 |
0.458622 |
TELECOM ParisTech_39_2_1245415387504.txt |
0.526302 |
0.459922 |
UAIC_34_2_1244812428616_changed.txt |
0.479700 |
0.105589 |
Wroclaw University of Technology_49_2_1245621871943.txt |
0.454221 |
0.170540 |
Wroclaw University of Technology_49_2_1245622279557.txt |
0.451702 |
0.180370 |
Wroclaw University of Technology_49_2_1245622900052.txt |
0.447512 |
0.214875 |
Wroclaw University of Technology_49_2_1245622992589.txt |
0.447512 |
0.214875 |
Wroclaw University of Technology_49_2_1245624864334.txt |
0.446024 |
0.220957 |
XRCE_36_2_1245429345115.txt |
0.267301 |
0.802704 |
____________________________________________________________________________________________________________
_________________________________________________________________________________________________________
_________________________________________________________________________________________________________
_________________________________________________________________________________________________________
________________________________________________________________________________________________________
_________________________________________________________________________________________________________
________________________________________________________________________________________________________
________________________________________________________________________________________________________
________________________________________________________________________________________________________
________________________________________________________________________________________________________
_________________________________________________________________________________________________________
_________________________________________________________________________________________________________
________________________________________________________________________________________________________
_________________________________________________________________________________________________________
________________________________________________________________________________________________________
________________________________________________________________________________________________________
________________________________________________________________________________________________________
________________________________________________________________________________________________________
_________________________________________________________________________________________________________
_________________________________________________________________________________________________________