Computer vision companion module for autonomous FPV drone last-mile navigation

The module is a companion computer with a forward-facing camera that can be attached to most FPV drones.  The modules are provided with encrypted USB drives that contain the software.

How the solution works:

• The drone with the companion module is activated 
• The encrypted USB drive is inserted into the USB port
• The software is decrypted by using unique keys (the software isn’t stored on the module itself – it is loaded directly into RAM, so when the power is off, it is permanently deleted)
• Once loaded, the drone is ready for mission
• The drone is controlled like any other craft, without requiring additional configuration 
• The parameters are adjusted by the built-in algorithm that considers the current payload weight

When seeing the target coming into visual range, the pilot can activate targeting mode by using the controller. A viewfinder (an enlarged frame center) will appear to simplify target acquisition.

After centering the target, the operator can switch to autonomous attack mode, again, using remote control. The module will take over ensuring the drone does strike the target, even under jammed control or signal.

This works for both static and moving targets.

The autonomous attack mode is “hit-and-forget” – it doesn’t require further actions after target acquisition, but for higher precision, the pilot can adjust target lock:

• To adjust the setting, the pilot needs only the stick normally assigned to pitch and roll
• This allows to reposition the scope during autonomous attack mode and increase the chance of success

Built-in “trackers”

Object tracking

A custom object tracker that attempts to capture object boundaries by using deep learning extracted features. The model works great for clearly visible targets (for example, vehicles on the road) and might also correct operator errors made during the target acquisition stage.

Pixel tracking

An accurate pixel tracker that follows selected points without clearly visible targets by leveraging optical flow. The method can be especially efficient in scenarios when targets are camouflaged. 

Key features

• Integration with Betaflight 4.5.0 (top firmware on the battlefield today) 
• Support for static and moving targets 
• Tracker-based autonomous UAV control 
• Target tracking:
• object tracking based on deep learning, accelerated by Neural Processing Unit (NPU) 
• pixel tracking based on optical flow
• Real-time processing on the single-board computer 
• Custom embedded Linux image for the single-board computer
• CRSF-based manual target adjustment 
• Support for different cameras to mitigate luminance changes
• Automatic calibration for different payload weights
• Code encryption
• Verbose logging and flight analysis ecosystem for future R&D efforts
• GStreamer video processing pipeline

• Betaflight integration

Betaflight firmware was chosen as the target platform, as it’s the leading flight controller to run various crafts. Being designed primarily for manual-only operation, its autonomous flight capabilities are limited, in particular to accommodate our objectives.

This challenge was handled by developing a better-fitting, custom algorithm for autonomous flight control.

•  Betaflight configuration

Betaflight provides many options that allow the pilots to tune drone behavior to their individual preferences. Some parameters were essential to operate our module.

A script was built to override default settings before launching, while keeping the pilot’s personal preferences.

•  Robustness despite weather changes

After achieving reliable operation in controlled, ideal conditions, there were other challenges to resolve:

• We helped to implement a solution that combines algorithm changes, PID tuning, and corrections to hardware to mitigate wind speed and direction, which affected drone behavior
• Changing luminance was negated by applying image preprocessing and supporting low-light cameras, which complicated target aiming and tracking
  
• The potential precipitation damage was minimized by improving the module’s physical enclosure – thereby, though not being completely waterproof, the module water resistance was enhanced

• PID tuning

PID control (Proportional-Integral-Derivative control feedback mechanism) is at the core of our custom model. PID parameters are commonly tuned empirically.

We used the values we obtained from simulations as our reference point, but performed additional fine-tuning for optimal flight control in the real world.

•  Reliable tracking in an extreme environment

While popular off-the-shelf trackers deliver acceptable results under ideal conditions, our case is not that easy. Military targets may either be camouflaged or obscured by smoke.

To mitigate this issue, a custom “pixel tracker” that uses optical flow to track a single pixel only was developed – it works no matter if targets can be locked at this point.

•  Intuitive experience

One of our objectives is designing a solution that’s easy to use and can be operated by less experienced pilots. We’ve already added many convenience features (for example, targeting mode and manual target adjustment) and continue collecting feedback from end-users.

We provide ongoing support and maintenance, and implement on-demand adjustments if requested.

•  Real-time performance

With the computing power of the single board computer (SBC), real-time performance is a massive challenge. But nonetheless, it’s critical to have a fast (and low-latency) control loop when operating high-speed drones.

This challenge was addressed through careful software optimization, the use of multithreading, and leveraging NPU acceleration.

•  Limited sensors

We aimed to keep the module both affordable and not GNSS dependent, so the only available options were:

• The built-in inertial measurement unit (IMU), which isn’t very accurate, and barometer
• And a forward-facing camera

To achieve robust control under these limiting conditions:

• We went for leveraging computer vision to extract additional information from the integrated camera
• And developed an algorithm to split the tasks into phases
  – At the first stage, it accelerates the drone toward target while maintaining constant altitude
  – At the second stage, when reaching desired angle, it dives toward target

• Increased precision – an automated last-mile navigation for topmost success rates
• Reduced investment – easy integration with already procured drones 
• Field-proven technology –already tested and deployed in combat
• EW-withstanding algorithm – no dependency on real-time control signals
• No extra training needed – the interface is easy-to-navigate, no matter skill level
• Future-proof stack – designed adaptable for future software development and improvements

Targeting mode

• The scope is yellow 
• The operator can see the target in the magnified view

Autonomous mode

• The scope turns red and follows the object
• The algorithm is controlling the drone to hold the target in the frame center

The drone will attack the target even after losing connection to the ground control:

Target adjustment

• The scope turns orange and follows the object
• The operator can move the stick to adjust the target and reinitialize the tracker