Robust computer vision methods for autonomous water surface vehicles

Collaborating partners: University of Ljubljana, Faculty of Computer and Information Science, Faculty of Electrical Engineering; Robotina d.o.o.

Type of the research project: basic research project, financed by Slovenian Research Agency (project code: J2-8175)

Project duration: 1st May 2017 - 30th April 2020

Acronym: ViAMaRo (Vision for autonomous marine robots)

Researchers

izr. prof. dr. Matej Kristan (PI)
doc. dr. Janez Perš (PI on FE side)
izr. prof. dr. Danijel Skočaj
prof. dr. Stanislav Kovačič
dr. Luka Čehovin Zajc
dr. Rok Mandeljc
mag. Alan Lukežič
mag. Borja Bovcon
mag. Jon Natanael Muhovič
mag. Mozetič Dean
Duško Vranac

Link to SICRIS

Project overview

Over the last decade the research in “field robotics” has resulted in development of small-sized (~2m long) unmanned surface vehicles (USVs) that can be manually guided or used to follow a pre-programmed path. Due to their portability and ability to navigate relatively shallow waters and narrow marinas, their potential use is indeed large, ranging from coastal water and environmental surveillance, to inspection of man-made structures above and below water surface.

A lot of research in USV has been dedicated to development of hardware, low-level guidance, control, self-organization and communication systems, but the level of autonomy in small-sized USVs is still relatively low. The reason is that research in advanced environment perception capabilities required for a long-term autonomous performance in uncontrolled environments lags behind the control and hardware research. Cameras as light-weight, low-power, information-rich sensors are becoming a viable alternative or addition to other sensorial modalities.

The project overarching goal is to develop functionalities required for robust autonomous navigation of USVs in uncontrolled environments, primarily relying on the captured visual information. The objectives are to develop efficient and robust computer vision approaches for obstacle detection, long-term tracking and fusion with other sensors and camera modalities. A critical requirement of the approaches will be real-time performance, environment adaptation and long-term robustness to temporary failures of sensory information and visual uncertainties. We will propose a framework that will combine such approaches into a model of robot environment, thus enabling robust long-term fully autonomous operation. The developed framework will be verified and validated on an existing integrated system, a USV, performing in real environment.

Workpackages: The work is divided into six work packages. The first four address the project scientific goals:

Development of robust obstacle detection approaches able to detect and extract 3D position of large as well as small obstacles (WP1).
Development of robust tracking approaches tailored to USV dynamics that enable target re-detection and re-identification (WP2).
Development of agile USV environment model that builds a map of USV surrounding, fuses results of multiple sensors, detection and tracking results into a common representation (WP3).
Construction of challenging datasets for objective offline validation of the developed methods and tests of selected methods onboard USV (WP4).

Work packages WP5 and WP6 contain support activities such as results dissemination and project management. In the following we detail the work packages and tasks.

Project phases:

Year 1: Activities on work packages WP1, WP2, WP4, WP5, WP6
Year 2: Activities on work packages WP2, WP3, WP4, WP5, WP6
Year 3: Activities on work packages WP1, WP3, WP4, WP5, WP6

Online resources

Multimodal marine obstacle detection dataset (MODD v2) - This dataset contains marine videos, captured by unmanned surface vehicle (USV). The challenge, posed by this dataset, is to segment each image into three natural regions: the sky, the shore and the sea, and furthermore, detect obstacles in the sea area.

The MaSTr1325 dataset for training deep USV obstacle detection models (MaSTr1325) is a new large-scale marine semantic segmentation training dataset tailored for development of obstacle detection methods in small-sized coastal USVs. The dataset contains 1325 diverse images captured over a two-year span with a real USV, covering a range of realistic conditions encountered in a coastal surveillance task. All images are per-pixel semantically labelled and synchronized with inertial measurements of the on-board sensors. In addition, a dataset augmentation protocol is proposed to address slight appearance differences of the images in the training set and those in deployment.

The USV stereo decalibration dataset contains image pairs and reference calibrations that can be used for developing autocalibration methods for stereo camera systems.