Introduction

Big benchmark challenges like ILSVRC or COCO supported much of the remarkable progress in computer vision and deep learning over the past years.

We aim to recreate this success for robotic vision.

We develop a set of new benchmark challenges specifically for robotic vision, and evaluate:

We combine the variety and complexity of real-world data with the flexibility of synthetic graphics and physics engines.

First Challenge - Probabilistic Object Detection

Our probabilistic object detection challenge requires participants to detect objects in video data from high-fidelity simulation. As a novelty, our evaluation metric rewards accurate estimates of spatial and semantic uncertainty using probabilistic bounding boxes.

We made a validation and test-dev dataset with a public leaderboard available for ongoing evaluation of probabilistic object detection approaches. In addition, a test-challenge dataset and evaluation server will become available when we are organising a public competition. The first of those competitions was organised for a workshop at CVPR 2019, and the next will be available around Setember 2019, leading up to IROS.

We organise a workshop at IROS 2019 on the topic of The Importance of Uncertainty in Deep Learning for Robotics. For that workshop, we will run a second round of the probabilistic object detection challenge. Stay tuned for further details.

To participate and for more information around the dataset read more here ….

News

September 2019 Two papers got accepted for publication: Probabilistic Object Detection: Definition and Evaluation will appear in the IEEE Winter Conference on Applications of Computer Vision (WACV) in March 2020. The Nature Machine Intelligence journal published our article A probabilistic challenge for object detection in its September issue.

August 2019 We opened our evaluation servers again for the 2nd Probabilistic Object Detection Challenge. Submissions are welcome until 8 October, and the results will be presented at our workshop at IROS in November.

May 2019: We will organise a workshop at IROS 2019 on the topic of The Importance of Uncertainty in Deep Learning for Robotics. Stay tuned for further details.

January 2019: We are happy to announce that CVPR 2019 is hosting our workshop. Participants of our Robotic Vision object detection challenge will present their approaches and results, and we will announce the competition winners at the workshop.

December 2018: We released our first Robotic Vision object detection challenge, requiring object detection on video data and rewarding accurate estimates of spatial and semantic uncertainty.

June 2018: We presented our initial ideas for new benchmarks and metrics at two workshops during CVPR and RSS. Thanks to all who engaged in discussions and shared their thoughts during the workshops on Real-World Challenges and New Benchmarks for Deep Learning in Robotic Vision at CVPR, and New Benchmarks, Metrics, and Competitions for Robotic Learning at RSS.

Stay in touch and follow us on Twitter for news and announcements: @robVisChallenge.

Coming Soon

Stay tuned for more challenges, focussing on active vision, and active and continuous learning in 2019.

Motivation

Big computer vision challenges and competitions like ILSVRC or COCO had a significant influence on the advancements in object recognition, object detection, semantic segmentation, image captioning, and visual question answering in recent years. These challenges posed motivating problems to the research community and proposed datasets and evaluation metrics that allowed to compare different approaches in a standardized way.

However, visual perception for robotics faces challenges that are not well covered or evaluated by the existing benchmarks. These challenges comprise calibrated uncertainty estimation, continuous learning for domain adaptation and incorporation of novel classes, active learning, and active vision.

There is currently a lack of meaningful standardised evaluation protocols and benchmarks for these research challenges. This is a significant roadblock for the evolution of robotic vision, and impedes reproducible and comparable research.

We believe that by posing a new robotic vision challenge to the research community, we can motivate computer vision and robotic vision researchers around the world to develop solutions that lead to more capable, more robust, and more widely applicable robotic vision systems.

Organisers, Support, and Acknowledgements

Stay in touch and follow us on Twitter for news and announcements: @robVisChallenge.

Niko Sünderhauf
Queensland University of Technology
Feras Dayoub
Queensland University of Technology
David Hall
Queensland University of Technology
John Skinner
Queensland University of Technology
Haoyang Zhang
Queensland University of Technology
Ben Talbot
Queensland University of Technology
Suman Bista
Queensland University of Technology



The Robotic Vision Challenges organisers are with the Australian Centre for Robotic Vision at Queensland University of Technology (QUT) in Brisbane, Australia.

This project was supported by a Google Faculty Research Award to Niko Sünderhauf in 2018.

Supporters

We thank the following supporters for their valuable input and engaging discussions.

Gustavo Carneiro
University of Adelaide
Anelia Angelova
Google Brain
Anton van den Hengel
University of Adelaide