Computer vision companion module for stabilized UAV hover

The module integrates seamlessly with existing FPV systems by using these components:

• Single board computer (SBC) running custom Linux image
• Downward-facing camera
• Betaflight and ArduPilot compatible software interface

How the solution works:

• At the target location, the pilot can activate the autonomous position hold 
• The module will maintain the drone’s exact position and altitude by leveraging:
  – IMU readings from the flight controller
  – CV algorithm to track captured features
• The computed control commands are sent to the flight controller to maintain stable hover
• The pilot can adjust the yaw while hovering by using the switch on the remote controller, for example, to change the direction of the attached transmitter 

Key features

Firmware integration

• Betaflight via MSP protocol
• ArduPilot via MAVLink protocol 

Stabilized hover

• Position and altitude hold even without GNSS coverage
• Yaw adjustment while maintaining position hold

Real-time onboard data processing

• ARM-based SBC with custom Linux OS
• Computer vision and control logic execution

Sensor fusion

• Camera-based odometry combined with IMU data
• Image stabilization to compensate for tilt and rotation

Modular and scalable architecture

• Plug-and-play compatibility (7 inches and larger)
• Easy adaptability to different firmware and hardware configurations

Advanced control

More robust PID controllers were developed, better optimized for each driving axis (pitch, roll, yaw, throttle). With extensive field testing, this enabled platform-specific calibration that compensates for characteristics including configurations and drone component variations.

Multi-firmware compatibility

We addressed the complexities of working with different flight controllers and mastered according protocols. Our solution was designed to combine Betaflight’s and ArduPilot capabilities to enable seamless operation across platforms.

1. Betaflight integration
Betaflight firmware was chosen as the target platform, as it’s the leading flight controller to run various crafts. Being designed primarily for manual-only operation, its autonomous flight capabilities are limited, in particular to accommodate our objectives.

The challenge was handled by developing a better-fitting, custom algorithm for autonomous flight control.

2. ArduPilot integration
ArduPilot firmware was selected as the target platform to extend compatibility beyond manual-only operation. It provides multiple autonomous flight modes in standard GNSS environments, thereby providing a foundation for advanced flight control.

To leverage ArduPilot’s capabilities:

• by replacing MSP with MAVLink communication
• And applied a feature to emulate the sensor by using visual data instead of GPS data

Angle stabilization

A software-based gimbal system that stabilizes visual streams during dynamic flight conditions was developed. As drones are tilting and shifting to retain their position, the system automatically compensates for distortion, thus ensuring consistent tracking no matter the weather. 

Altitude management

An altitude estimation algorithm was implemented by integrating visual odometry and drone barometer data. The algorithm can interpret camera movement and prevent visual shifts from causing potential dispositioning, thus ensuring hover stability.

Real-time performance

With the computing power of the single board computer (SBC), real-time performance is a massive challenge. But nonetheless, it’s critical to have a fast (and low-latency) control loop when operating high-speed drones.

This challenge was addressed through careful software optimization and multithreading.

Limited sensors

We aimed to keep the module as affordable as possible, so the only available options were:

• The built-in inertial measurement unit (IMU), which isn’t very accurate, and barometer
• And a downward-facing camera

Computer vision to extract additional information from the integrated camera was leveraged to achieve robust control under these limiting conditions.

Robustness despite weather changes

Wind and luminance adaptability
Important corrections within hardware and algorithms were implemented to mitigate wind speed & direction. Image preprocessing has helped to negate changing luminance, which complicated target aiming and tracking.

This way, the solution works reliably beyond controlled, ideal conditions.

Oscillation leveling
Even when the drone is holding its position, it can start wobbling when affected by sudden weather changes. This drives energy consumption and may also interfere with the other equipment.

The problem was solved through careful PID tuning and proper weight distribution. 

Feature matching

The algorithm is leveraging visual odometry that uses feature matching and estimates up-to-date movements. The location is calculated by integrating odometry displacement over periods (between pairs of frames).

We resolve the challenge by addressing several aspects:

• to match the frame with the last updated key frame 
• The camera is tuned to obtain video footage of the best-possible quality
• What’s more, there were some experiments with combinations to select:
  – keypoint detectors
  – feature descriptors

During trials, the drone has maintained stable hover for over 20 minutes under the following conditions:

• 17 °C (air temperature)
• 5 m/s (wind speed)

The top-left section displays real-time telemetry from the flight controller and the visual odometry (VO) system and highlights how these values change during hover.

The top-right section presents a map that tracks the drone’s exact position when hovering:

• In an ideal environment, the path would appear as a single point and indicate a perfectly stable hold
• In practice, it might slightly drift when affected by changing wind and lighting conditions (the system continuously compensates for deviations to keep the drone as close as possible to its intended position)