CVPR 2021 WS: Embodied AI - Robotic Vision Scene Understanding Challenge

News

• 20 June 2021 CVPR 2021 Workshop
• 17 May 2021 Challenge extended to May 31st.
• 29 March 2021 Nvidia update main GPU prize from Titan RTXs to Quadro RTX6000s
• 17 February 2021 Challenge released as part of CVPR 2021 Embodied AI workshop

Overview

The CVPR2021 Embodied AI Workshop of our Robotic Vision Scene Understanding Challenge evaluates how well a robotic vision system can understand the semantic and geometric aspects of its environment. The challenge consists of two distinct tasks: Object-based Semantic SLAM, and Scene Change Detection.

Key features of this challenge include:

• Prizes including $2500 USD, Quadro RTX6000 GPUs, and Jetson Nano GPUs
• BenchBot, a complete software stack for running robotic systems
• Running algorithms in realistic 3D simulation, and on real robots, with only a few lines of Python code
• The BenchBot API, which allows simple interfacing with robots while supporting both OpenAI Gym-style and simple object-oriented Agent approaches
• Easy-to-use scripts for running simulated environments, executing code on a simulated robot, evaluating semantic scene understanding results, and automating code execution across multiple environments
• Opportunities for the best teams to execute their code on a real robot in our lab, which uses the same API as the simulated robot
• Use of the Nvidia Isaac SDK for interfacing with, and simulation of, high fidelity 3D environments

Submit your results to EvalAI

Watch the video below to learn more!

Important Dates

• February 17th - Challenge launch
• May 31st - Results and paper submissions due
• June 19th-25th - CVPR 2021

Challenge Tasks

The semantic scene understanding challenge is split into 3 modes, across 2 different semantic scene understanding tasks, for a total of 6 different challenge variations. The different variations open up the challenge to a wider range of participants, from those who want to focus solely on visual scene understanding to those wish to integrate robot localisation with visual detection algorithms.

The two different semantic scene understanding tasks are:

1. Semantic SLAM: Participants use a robot to traverse around the environment, building up an object-based semantic map from the robot’s RGBD sensor observations and odomtry measurements.
2. Scene change detection (SCD): Participants use a robot to traverse through an environment scene, building up a semantic understanding of the scene. Then the robot is moved to a new start position in the same environment, but with different conditions. Along with a possible change from day to night, the new scene has a number objects added and / or removed. Participants must produce an object-based semantic map describing the changes between the two scenes.

The object-based semantic maps generated by submissions are evaluated against the corresponding ground-truth object-based semantic map (for task 2 this is the map of changes in the second scene, with respect to the first). Please see the BenchBot Evaluation documentation for details on object-based semantic maps submission formats, and further explanation of the two different tasks.

Each task has 3 variations, corresponding to the following modes:

1. Passive control, with ground-truth localisation (PGT): The robot follows a fixed-trajectory, and participants are given a single method to control the robot: moving to the next pose. The task cannot be continued once the entire trajectory has been traversed. Participants receive ground-truth poses for all robot components after each action.
2. Active control, with ground-truth localisation (AGT): The robot can be controlled by either moving forward a requested distance, or rotating on the spot a requested number of degrees. The task cannot be continued if the robot collides with the environment. Participants receive ground-truth poses for all robot components after each action.
3. Active control, using dead reckoning (i.e. no localisation) (ADR): The robot can be controlled by either moving forward a requested distance, or rotating on the spot a requested number of degrees. Participants receive poses derived from robot odometry after each action, with localisation error that accumulates over time.

Please see the BenchBot API documentation for full details about what actions and observations are available in each mode, and how to use them with your semantic scene understanding algorithms.

How to Participate - [Challenge CLOSED]

The CVPR2021 challenge has been open since February 2021, and available here on the EvalAI website. Please create an account, sign in, & click the “Participate” tab to enter our challenge. Full details on how to participate, the available software framework, and submission requirements are provided on the site.

The challenge is open until the May 31st, and a total of $2,500 USD will be awarded to high-performing competitors. The two best performing teams also receive a Titan RTX GPU and up to 5 Jetson Nanos (1 per team member) provided by our sponsors Nvidia. A full breakdown of the prizes available is outlined in Challenge Prizes.

Participating in the Semantic Scene Understanding Challenge is as simple as the 4 steps below. The BenchBot software stack is designed from the ground up to eliminate as many obstacles as possible, so you can focus on what matters: solving semantic scene understanding problems. A collection of resources, documentation, and examples are also available within the BenchBot ecosystem to support your experience while participating in the challenge.

To participate in our challenge:

1. Download & install the BenchBot software stack. Use the examples to dive straight in & start playing (benchbot_run –list-examples)
2. Choose a task to start working on a solution for, using benchbot_run --list-tasks to list supported tasks
3. Start with the development environments “miniroom” and “house” which include ground-truth maps to aid in your algorithm development
4. Develop a solution using the benchbot_run, benchbot_submit, & benchbot_eval scripts
5. Create some results for your solution in the challenge environments using: benchbot_batch -r carter -t <your_task> -E <batch_name> -z -n <your_submission_cmd>
6. Use the Submit tab at the top of our EvalAI page to submit your results for evaluation

Pariticipants need to provide a short summary of the method at submission time. Each participant is also encouraged to write a short paper (approx 4 pages) outlining the technique which was used in the challenge. These papers are no longer required at the same time as the initial submission but will be made available to the general public through our challenge website. Please submit summaries and papers by e-mail to contact@roboticvisionchallenge.org.

Challenge Prizes

As the challenge is complex, with multiple components, we provide a tiered prize list. The highest scoring on any given leaderboard will be awarded the corresponding prize. Teams are allowed to participate across all challenges and win multiple prizes.

1. Scene Change Detection (ADR) - $900 USD, 1 Quadro RTX6000 GPU, up to 5 Jetson Nano GPUs
2. Semantic SLAM (ADR) - $800 USD, 1 Quadro RTX6000 GPU, up to 5 Jetson Nano GPUs
3. Semantic SLAM (AGT) - $500 USD
4. Semantic SLAM (PGT) - $300 USD

Workshop Details

Our Robotic Vision Scene Understanding Challenge is currently part of the CVPR 2021 Embodied AI workshop. Please check out the workshop website for more details and for information about 9 other robotic vision/embodied AI challenges being run.

Questions?

Talk to us on Slack or contact us via email at contact@roboticvisionchallenge.org.

Organisers, Support, and Acknowledgements

Stay in touch and follow us on Twitter for news and announcements: @robVisChallenge.

The Robotic Vision Challenges organisers are with the Australian Centre for Robotic Vision and the Queensland University of Technology (QUT) Centre for Robotics in Brisbane, Australia.