This workshop assesses current evaluation procedures for object detection, highlights their shortcomings and opens discussion for possible improvements.
Through a focus on evaluation using challenges, the object detection community has been able to quickly identify which methods are effective by examining performance metrics. However, as this technological boom progresses, it is important to assess whether our evaluation metrics and procedures adequately align with how object detection will be used in practical applications. Quantitative results should be easily reconciled with a detector’s performance in applied tasks. This workshop provides a forum to discuss these ideas and evaluate whether current standards meet the needs of the object detection community.
In addition, this workshop is hosting the latest iteration of the Probabilistic Object Detection (PrOD) Challenge which requires competitors to estimate semantic and spatial uncertainty.
Online Workshop Details: 28th August 2020
Due to the current COVID-19 crisis, ECCV is going to be held online with pre-recorded video presentations and two interactive sessions. You can also check the latet details from the ECCV COVID updates page here.
Please note that to access the videos and interactive sessions you will need to be registered for ECCV 2020. An early registration fee rate can be applied to authors of accepted papers.
We plan to have 45 min presentations by invited speakers and shorter (max 10 min) videos from contributed papers and PrOD Challenge competitors. All videos should be submitted by August 16th and will be made available to view through ECCV on August 21st.
Please note that video presentations will not be made publicly available until after the ECCV 2020 conference.
- Invited Talk: Assistant Prof. Emre Akbas (Middle East Technical University)
- Invited Talk: Assistant Prof. Walter Scheirer (University of Notre Dame): “Visual Psychophysics as an Evaluation Regime for Object Recognition”
- Invited Talk: Dr Larry Zitnick (Facebook AI Research)
- Organizer Talk: Dr David Hall (Queensland University of Technology)
- PrOD Challenge Overview and Discussion of Results: Dr David Hall (Queensland University of Technology)
- Contributed paper talks: TBA
- PrOD competitor talks: TBA
We have 2 seperate interactive sessions where ECCV delegates can talk with our invited speakers, accepted paper authors, and workshop organizers. Each session will be an interactive panel with questions submitted by members of the community either within the session or in advance through Slido (see link below). Note you require ECCV registration to access the interactive sessions.
Submit questions for the panel to discuss
Please make sure if you have a question for a specific panel member that you address them in your question.
Session 1 28th August 00:00-02:00 UTC+1 - Expected Participants
- Assistant Prof. Walter Scheirer (University of Notre Dame)
- Dr Larry Zitnick (Facebook AI Research)
- Dr David Hall (Queensland University of Technology)
Session 2 28th August 08:00-10:00 UTC+1 - Expected Participants
- Assistant Prof. Emre Akbas (Middle East Technical University)
- Dr David Hall (Queensland University of Technology)
Call for Papers
We invite authors to contribute papers to the workshop. Topics of interest comprise, but are not limited to:
- New evaluation measures/metrics for object detection
- New evaluation/visualization tools to analyze object detection systems
- New evaluation procedures for better understanding object detection performance
- Examinations of current evaluation procedures
- New datasets designed to examine specific challenges in object detection
- New detection methods that provide contributions/insights unrewarded by current evaluation procedures (e.g. improved detector calibration, probabilistic object detection, etc.)
- Submissions must follow the full submission guidelines outlined here
- Before ECCV decisions:
- papers submitted to workshop cannot be shorter than 4 pages (max 14 pages)
- if a paper submitted to a workshop has a substantial overlap with a paper submitted to a ECCV it must be exactly 4 pages long
- After ECCV decisions:
- 4-page long papers that overlap with a rejected paper can be published in the workshop proceedings (can be longer than 4 pages)
- 4-page long papers that overlap with a paper accepted to ECCV can present at the workshop but the paper will not be published in the workshop proceedings.
- Submit your paper through CMT
- Accepted papers will get to present a short (10 min max) video for the workshop and authors should be available for one of the workshop’s interactive sessions.
- This video cannot be made publicly available until after the ECCV 2020 conference
- If you have not already registered for ECCV 2020, accepted papers will have access to the early registration discount after acceptance and only need to register as regular delegates.
Participate in the Competition
To participate in the competition, and for more information around the data and submission format, please go to our Codalab Page.
Our challenge requires participants to detect objects in video data (from high-fidelity simulation). As a novelty, our evaluation metric rewards accurate estimates of spatial and semantic uncertainty using probabilistic bounding boxes. We developed a new probability-based detection quality (PDQ) evaluation measure for this challenge, please see the arxiv paper for more details.
Submissions must be accompanied by a paper following the guidelines above explaining the method and external data used. Top performing submissions from the challenge will be invited to present their methods for the workshop.
- 05 August 2020 Final Submissions to the Evaluation Server via Codalab
- 07 August 2020 Paper Submission (Workshop and Challenge) via CMT
- 12 August 2020 Winner Announcements and Workshop Invitations
- 21 August 2020 Videos and photo-ready papers for Workshop due
- 28 August 2020 ECCV Workshop Interactive sessions (00:00 - 02:00 and 08:00-10:00 UTC+1)
The workshop organisers are with the Australian Centre for Robotic Vision