In order to fully integrate deep learning into robotics, it is important that deep learning systems can reliably estimate the uncertainty in their predictions. This would allow robots to treat a deep neural network like any other sensor, and use the established Bayesian techniques to fuse the network’s predictions with prior knowledge or other sensor measurements, or to accumulate information over time.

Deep learning systems, e.g. for classification or detection, typically return scores from their softmax layers that are proportional to the system’s confidence, but are not calibrated probabilities, and therefore not useable in a Bayesian sensor fusion framework.

Current approaches towards uncertainty estimation for deep learning are calibration techniques, or Bayesian deep learning with approximations such as Monte Carlo Dropout or ensemble methods.

Our work focusses on Bayesian Deep Learning approaches for the specific use case of object detection on a robot in open-set conditions.

Publications

  1. Evaluating Merging Strategies for Sampling-based Uncertainty Techniques in Object Detection Dimity Miller, Feras Dayoub, Michael Milford, Niko Sünderhauf. arXiv preprint, 2018. There has been a recent emergence of sampling-based techniques for estimating epistemic uncertainty in deep neural networks. While these methods can be applied to classification or semantic segmentation tasks by simply averaging samples, this is not the case for object detection, where detection sample bounding boxes must be accurately associated and merged. A weak merging strategy can significantly degrade the performance of the detector and yield an unreliable uncertainty measure. This paper provides the first in-depth investigation of the effect of different association and merging strategies. We compare different combinations of three spatial and two semantic affinity measures with four clustering methods for MC Dropout with a Single Shot Multi-Box Detector. Our results show that the correct choice of affinity-clustering combinations can greatly improve the effectiveness of the classification and spatial uncertainty estimation and the resulting object detection performance. We base our evaluation on a new mix of datasets that emulate near open-set conditions (semantically similar unknown classes), distant open-set conditions (semantically dissimilar unknown classes) and the common closed-set conditions (only known classes). [arXiv]
  1. Dropout Sampling for Robust Object Detection in Open-Set Conditions Dimity Miller, Lachlan Nicholson, Feras Dayoub, Niko Sünderhauf. In Proc. of IEEE International Conference on Robotics and Automation (ICRA), 2018. Dropout Variational Inference, or Dropout Sampling, has been recently proposed as an approximation technique for Bayesian Deep Learning and evaluated for image classification and regression tasks. This paper investigates the utility of Dropout Sampling for object detection for the first time. We demonstrate how label uncertainty can be extracted from a state-of-the-art object detection system via Dropout Sampling. We show that this uncertainty can be utilized to increase object detection performance under the open-set conditions that are typically encountered in robotic vision. We evaluate this approach on a large synthetic dataset with 30,000 images, and a real-world dataset captured by a mobile robot in a versatile campus environment.
  1. Dropout Variational Inference Improves Object Detection in Open-Set Conditions Dimity Miller, Lachlan Nicholson, Feras Dayoub, Niko Sünderhauf. In Proc. of NIPS Workshop on Bayesian Deep Learning, 2017. One of the biggest current challenges of visual object detection is reliable operation in open-set conditions. One way to handle the open-set problem is to utilize the uncertainty of the model to reject predictions with low probability. Bayesian Neural Networks (BNNs), with variational inference commonly used as an approximation, is an established approach to estimate model uncertainty. Here we extend the concept of Dropout sampling to object detection for the first time. We evaluate Bayesian object detection on a large synthetic and a real-world dataset and show how the estimated label uncertainty can be utilized to increase object detection performance under open-set conditions.
  1. Episode-Based Active Learning with Bayesian Neural Networks Feras Dayoub, Niko Sünderhauf, Peter Corke. In Workshop on Deep Learning for Robotic Vision, Conference on Computer Vision and Pattern Recognition (CVPR), 2017. We investigate different strategies for active learning with Bayesian deep neural networks. We focus our analysis on scenarios where new, unlabeled data is obtained episodically, such as commonly encountered in mobile robotics applications. An evaluation of different strategies for acquisition, updating, and final training on the CIFAR-10 dataset shows that incremental network updates with final training on the accumulated acquisition set are essential for best performance, while limiting the amount of required human labeling labor.