This workshop assesses current evaluation procedures for object detection, highlights their shortcomings and opens discussion for possible improvements.
Through a focus on evaluation using challenges, the object detection community has been able to quickly identify which methods are effective by examining performance metrics. However, as this technological boom progresses, it is important to assess whether our evaluation metrics and procedures adequately align with how object detection will be used in practical applications. Quantitative results should be easily reconciled with a detector’s performance in applied tasks. This workshop provides a forum to discuss these ideas and evaluate whether current standards meet the needs of the object detection community.
In addition, this workshop hosts the latest iteration of the Probabilistic Object Detection (PrOD) Challenge which requires competitors to estimate semantic and spatial uncertainty.
Online Workshop Details: 28th August 2020
Due to the current COVID-19 crisis, ECCV was held online with pre-recorded video presentations and two interactive sessions. To access all recordings and papers, login to the ECCV online platform here.
We have approximately 45 min presentations by invited speakers and shorter (max 10 min) videos from contributed papers and PrOD Challenge competitors. Videos are currently available through the ECCV virtual platform and on YouTube. Full links to the papers are currently only available through ECCV.
Please note that ECCV links given below require you to be logged-in to the ECCV conference.
A full YouTube playlist of all presentations and interactive sessions can be found here
Invited Speaker/Organizer Presentations:
- Assistant Prof. Emre Akbas (Middle East Technical University): Unified Evaluation and Training of Object Detectors using Localization-Recall-Precision (LRP) - (ECCV, YouTube)
- Assistant Prof. Walter Scheirer (University of Notre Dame): Visual Psychophysics as an Evaluation Regime for Object Recognition - (ECCV, YouTube)
- Dr Larry Zitnick (Facebook AI Research): The Evaluation Streetlight - (ECCV, YouTube)
- Dr David Hall (Queensland University of Technology): Are our detection evaluation norms “enough” - (ECCV, YouTube)
- Dr David Hall (Queensland University of Technology): PrOD Challenge Overview and Discussion of Results - (ECCV, YouTube)
- Assessing different box merging strategies and uncertainty estimation methods in multimodel object detection - Felippe Schmoeller Roza, Maximilian Henne, Karsten Roscher and Stephan Günnemann - (ECCV, YouTube)
- Implementing Planning KL-Divergenge - Jonah Philion, Amlan Kar and Sanja Fidler - (ECCV, YouTube)
- ODIN: an Object Detection and Instance Segmentation Diagnosis Framework - Rocio Torres, Piero Fraternali and Jesus Romero - (ECCV, YouTube)
- Shift Equivariance in Object Detection - Marco Manfredi and Yu Wang - (ECCV, YouTube)
Probabilistic Object Detection (PrOD) Challenge Papers
- 1st: Probabilistic Object Detection with an Ensemble of Experts - Dimity Miller - (ECCV, YouTube)
- 2nd: EPrOD: Evolved Probabilistic Object Detector with Diverse Samples - Jaewoong Choi, Sungwook Lee, Seunghyun Lee and Byung Cheol Song - (ECCV, YouTube)
- 3rd: Probabilistic Object Detection via Deep Ensembles - Zongyao Lyu, Nolan Guitierrez, Aditya Rajguru and William Beksi - (ECCV, YouTube)
We had 2 seperate interactive sessions where ECCV delegates could talk with our invited speakers, accepted paper authors, and some workshop organizers. Each session was an interactive panel with questions submitted by members of the community either within the session or in advance.
Each interactive session was recorded and you can find YouTube links to the sessions below.
Session 1 28th August 00:00-02:00 UTC+1
- Walter Scheirer
- Larry Zitnick
- Sungwook Lee
- Zongyao Lyu
- Dimity Miller
- Felippe Schmoeller Roza
- Rocio Torres
- David Hall
- Niko Sünderhauf
Session 2 28th August 08:00-10:00 UTC+1
- Emre Akbas
- Marco Manfredi
- Jonah Philion
- David Hall
- Niko Sünderhauf
Call for Papers
We invited authors to contribute papers to the workshop. Topics of interest comprise, but are not limited to:
- New evaluation measures/metrics for object detection
- New evaluation/visualization tools to analyze object detection systems
- New evaluation procedures for better understanding object detection performance
- Examinations of current evaluation procedures
- New datasets designed to examine specific challenges in object detection
- New detection methods that provide contributions/insights unrewarded by current evaluation procedures (e.g. improved detector calibration, probabilistic object detection, etc.)
Participate in the Competition
To participate in the competition, and for more information around the data and submission format, please go to our Codalab Page.
Our challenge requires participants to detect objects in video data (from high-fidelity simulation). As a novelty, our evaluation metric rewards accurate estimates of spatial and semantic uncertainty using probabilistic bounding boxes. We developed a new probability-based detection quality (PDQ) evaluation measure for this challenge, please see the arxiv paper for more details.
Submissions must be accompanied by a paper explaining the method and external data used. Top performing submissions from the challenge will be invited to present their methods for the workshop.
This version of the competition is now over but you can test your methods on our continuous evaluation server in preparation for the next iteration of the challenge
- 05 August 2020 Final Submissions to the Evaluation Server via Codalab
- 12 August 2020 Paper Submission (Workshop and Challenge) via CMT
- 15 August 2020 Winner Announcements and Workshop Invitations
- 21 August 2020 Videos and photo-ready papers for Workshop due
- 28 August 2020 ECCV Workshop Interactive sessions (00:00 - 02:00 and 08:00-10:00 UTC+1)
The workshop organisers are with the Australian Centre for Robotic Vision