Self-supervised learning of visual appearance solves fundamental problems of optical flow

Flying insects as the inspiration for AI for small drones

How do honeybees land on flowers or avoid obstacles? Most biologists will be interested in such questions. However, the rise of small electronics and robotic systems has also made them relevant to robotics and artificial intelligence (AI). For example, small flying robots are extremely restricted in terms of sensors and processing that they can carry onboard. If these robots are to be as autonomous as very large self-driving cars, they have to use a highly efficient type of artificial intelligence similar to the highly developed intelligence possessed by flying pests.

optical flow

One of the main tricks in the insect’s sleeve is the widespread use of ‘optical flow’: the way objects are moved in their approach. They use it to land on flowers and avoid obstacles or predators. Insects use surprisingly simple and elegant optical flow strategies to deal with complex tasks. For example, for landings, optical flow deviations (how quickly things become large to see) are constant when the honeybee goes down. By following this simple rule, they automatically make smooth, soft landings.

I began my work on optical flow control with excitement about such elegant, simple strategies. However, developing control methods to actually implement these strategies in flying robots became far from trivial. For example, when I first worked on optical flow my flying robots would not actually land, but they started moving continuously, up and down just above the landing surface.

Honeybees are a fertile source of inspiration for AI of small drones. They are capable of an impressive demonstration of complex behaviors with very limited processing (~ 960,000 neurons). Drones are very interesting “models” for biology. Testing hypotheses from biology on drones can bring novel insights into solving problems faced by flying insects such as Habibe.

Fundamental problems

Optical flow has two fundamental problems that have been widely described in the growing literature on bio-inspired robotics. The first problem is that optical flow only provides mixed information on distance and velocity – and not separately on distance or velocity. To describe, if there are two landing drones and one of them flies twice and is twice faster than the other drone, they experience exactly the same optical current. However, for good control both of these drones must actually react differently to deviations in optical flow deviations. If a drone does not adapt its responses to the altitude at the time of landing, it will never come and start oscillating above the landing surface.

The second problem is that the optical flow in the direction in which a robot is moving is very small and slightly informative. This is very unfortunate to avoid an obstacle, as it means that obstacles directly ahead of the robot are the most difficult to detect! The problems are depicted in the figures below.

Illustration of two problems
Left: Problem 1: The white drone is double and twice as fast as the red drone. However, they both see the same optical flux deviation, as the object becomes twice as large in both cases. It can be seen by colored triangles – at the highest position, it captures the full angle of the landing platform, at the lowest position it only covers half of the landing platform in the field of view.
right: Problem 2: The drone moves directly in the direction of its forward camera. Therefore, attention to detail is straight forward. Objects close to this direction, like the red barrier, have very little current. This is illustrated by the red lines in the figure: the angles of these lines are very similar with respect to the camera. Objects from this direction, such as the green barrier, flow considerably. Indeed, the green lines indicate that the angle quickly becomes larger when the drone moves forward.

Visual presence learning as a solution

In an article published today in Nature Machine Intelligence [1], We propose solutions to both problems. The main idea was that if optical flows were not only able to explain, but also the visual appearance of objects in an optical environment, then both problems of optical flow would disappear. This solution is evident from the above figures. Rectangular insets show images captured by the drone. For the first problem, it is clear that the image perfectly captures the height difference between the white and red drones: the landing platform is simply larger in the image of the red drone. The red obstacle for the second problem is larger than the green in the drone image. Given their similar size, the odds are similar to those of drones.

Uncovering the visual appearance captured by an image will allow the robot to see distances to objects in the scene in the same way that we humans can estimate distances in a still picture. This will allow drones to get the correct control advantage immediately for optical flow control and this will allow them to see obstacles in the flight direction. The only question was: how can a flying robot learn such distances?

The key to this question lies in a theory that I formulated a few years ago. [2], Which showed that flying robots can actively induce optical flow oscillations to visualize the distances of objects in the scene. In the approach proposed in the article Nature Machine Intelligence, robots use such oscillations to learn what objects in their environments look like at different distances. In this way, the robot can learn for example how fine the texture of grass is when viewing from different heights during landing, or how far apart the tree bark is when navigating a forest.

Relevance for Robotics and Applications

Implementing this learning process on flying robots promoted optical flow landing, much faster than before. Furthermore, to avoid obstacles, the robots were now able to see the obstacles in the direction of flight very clearly. This not only improved obstacle detection performance but also allowed our robot to speed up. We believe that the proposed methods will be very relevant for resource-constrained flying robots, especially when they fly in a greenhouse to monitor harvest or keep track of stock in warehouses, but rather operate in a confined environment.

It is interesting to compare our method of distance learning with recent methods in the computer vision domain for single-camera (unicellular) distance perception. In the field of computer vision, the self-supervision of unicellular distance perception is done with the help of projective geometry and reconstruction of images. This results in impressively accurate, dense distance maps. However, these maps are still “without points” – they can show that an object is twice the other but cannot express distances in an absolute sense.

In contrast, our proposed method provides “approximate” distance estimates. Interestingly, the scaling is not in terms of meters, but in terms of the control gain that will drive the drone towards oscillation. This makes it very relevant for control. It is very much like the way we humans experience distances. For us also it may be more natural for logic to refer to action (“Is there an object within reach?”, “How many steps do I need to get a place in comparison to a meter?”). It is therefore very reminiscent of the notion of “tolerate”, a concept forwarded by Gibson, who introduced the concept of optical flow [3].

Flying a drone
Picture of our drone during obstacle prevention experiments. First, the drone moves forward, trying to keep the altitude constant by keeping the vertical optical current equal to zero. If it gets close to an obstacle, it will oscillate. Moving oscillations near an obstacle are used to visualize the distance through visual presence. After learning, the drone can fly fast and be safe. For experiments we used a Parrot Bebop 2 drone, replacing the onboard software with it Paparazzi Open Source Autopilot – Performing all the computing onboard of the drone’s core processor.

Relevance to biology

The findings are not only relevant to robotics, but also provide a new hypothesis for insect intelligence. Typical honeybee experiments begin with a learning phase, in which honeybees exhibit different oscillatory behaviors when they are familiar with a new environment and related novel signals such as artificial flowers. After this learning phase is over, the final measurements presented in the articles typically take place and mainly focus on the role of optical flow. The learning process presented constitutes a novel hypothesis of how flying insects improve their naval skills over their lifetimes. This suggests that we should establish more studies to investigate and report on the phase of this study.

Guido de Kroon, Christophe de Wagter, and Tobias Seid.

Stay on top - Get the daily news in your inbox

DMCA / Correction Notice

Recent Articles

Related Stories