Computer vision add-on module for autonomous UAV navigation
Project summary
The solution is a companion computer directly attached to existing aerial systems for autonomous UAV flights. It allows extended flights, entry into GNSS-denied zones, reconnaissance operations, search-and-rescue flights – all without a stable GNSS coverage.
Field trials have shown success rates of 80-90% even during wind bursts.
Services:
Project overview
A module to allow unmanned aerial vehicles (UAVs) autonomously navigate even within challenging settings – a significant military advantage in scenarios with compromised GNSS coverage.
This module is developed to provide location information quite similar to standard GNSS modules you’ve seen. The difference – it relies on captured visual information, not usual satellite signals, to detect drone positions even without consistent connection.
And that’s what provides an advantage in scenarios where failure is not an option.
Most firmware already incorporates autonomous capabilities: position hold, waypoint navigation, and others. The problem – they’re dependent on stable GNSS coverage, easily negated by widespread electronic warfare (EW systems).
Abto Software has helped a client to unlock new opportunities in operating unmanned aerial vehicles (UAVs). With expertise in implementing CV and ML algorithms, our engineers have eliminated GNSS dependence in estimating drone positioning.
Abto Software was involved from conceptualization to scaling the solution for production.
A unique, warfare-connected ecosystem of services
Each feature is validated with soldiers in combat, not sanitized lab conditions, to deliver battle-ready systems that minimize the risk and defeat the enemy.
Equip, deploy, and win – choose technology repeatedly validated in real military scenarios.
Main goals
- To enable UAV navigation in challenging, GNSS-denied environments
- And provide a low-cost, scalable solution for integration into on-hand aerial vehicles
Operational reach going beyond GPS coverage
Stay on-course without signal
How the solution works
The module is a companion computer with a bottom-facing camera being attached as a fallback component – the back-up for those critical situations with absent GNSS coverage.
The module is designed to work in parallel with the GNSS module:
- While there’s stable connection, the calibration is performed by using GNSS data
- As soon as there’s no connection, our algorithm steps in to provide visual data
Key components
Visual odometry (VO)
Visual odometry estimates up-to-date UAV motion by detecting and matching visual features between frames. The location is calculated by integrating odometry displacement over periods (between pairs of frames).
- This provides reliable results for short drone trajectories
- For longer drone trajectories, the estimate gradually drifts due to the accumulation of errors that make it unsuitable for extended independent missions
Machine learning (ML)
Machine learning is introduced to adjust the location from time to time and discard accumulated deviations. The frames are compared with preloaded map tiles to find the most similar tiles to the ones the drone “sees” and adjust the location.
To get this task done efficiently, we experimented with several popular models, including the ViT model.
Aggregator submodule
Key challenge: the model can assign equally high confidence scores to areas that have similar-looking objects.
That’s why there’s the aggregator submodule – the element that combines:
- VO results
- With the ML results
The aggregator is leveraging VO trajectory to estimate the map sector where the drone is most likely located. The longer VO reliance since the ML adjustment, the wider the sector to account for uncertainty already caused by accumulated VO errors.
In this predicted sector, we treat all the map tiles as candidates for the drone’s location at the current moment. To find the best-fitting, we consider previous observations from the ML model since the last synchronization.
Key features
- Easy connection (just like GNSS modules)
- Real-time processing on the NVIDIA Jetson single board computer (SBC)
- VO computation by using feature matching
- ML model to match camera images with preloaded map tiles
- A custom CV & ML algorithm to use ML capabilities to compensate for accumulated VO errors
- Code encryption
- Verbose logging and flight analysis ecosystem for future R&D efforts
- ML training and experiment tracking pipeline
Our contribution
R&D (research & development) was performed in parallel throughout stages and included:
- POC development
- Parameter tuning
- Hardware investigation
- Competitor analysis
- CV and ML research to investigate different approaches
- ML experiments, in particular regarding training
Software development
We helped to develop a configurable, scalable solution for real-time UAV localization that implements an algorithm also developed from scratch.
Industrial design
- The selection of optimal price-quality hardware (companion SBC, camera, connectors)
- The design of a physical enclosure to protect the module
Data security
We went with the CI/CD pipeline to create and encrypt the builds for the single board computer (SBC).
Data analysis
An ecosystem for flight data analysis is collecting data across multiple sources (videos, logs, GPS trajectory) and presenting it interactively on dashboards.
Main challenges
Real-time performance
With the computing power of the single board computer (SBC), real-time performance is a massive challenge. Despite this, it’s essential to maintain a fast (and low-latency) control loop when operating high-speed drones.
We addressed this challenge through careful software optimization, the use of multithreading, and leveraging hardware acceleration with the onboard GPU.
Limited sensors
We aimed to keep the module as affordable as possible, so the only available options were:
- The built-in inertial measurement unit (IMU), which isn’t very accurate, and barometer
- And a downward-facing camera
To achieve robust control under these limiting conditions, we went for leveraging computer vision to extract additional information from the integrated camera.
Camera angle data invariance
To provide optimal accuracy, the camera that’s attached must always remain perpendicular to the land plane. The problem: during flights, when the drone tilts, the camera isn’t pinpointing directly downward.
It had been decided to engineer a sophisticated gimbal system to stabilize the camera’s visual feed during dynamic flight conditions, which ensures data accuracy even during drone tilts.
VO drift
What if we know the drone’s initial location before the GNSS failure and want to track its trajectory as before? We can simply calculate the displacements between subsequent video frames to estimate current location.
However, this will introduce small inaccuracies, which will then accumulate and cause serious drifts.
This problem was solved with a ML approach that compares the current video frame with available map tiles:
- We determine the location and adjust its position
- And reset the error that’s accumulated
ML ambiguity
The model is splitting the map into tiles and comparing the current video frame with each to identify a match. As we mentioned before, it sometimes can produce equally high confidence scores.
This problem was solved by developing an algorithm that uses approximate locations to narrow the search, which reduces the ambiguity and increases the accuracy.
Fail-safe flights even within contested airspace
No more single-point failure Tools and technology stack
Tech stack:
- C++
- Python
- OpenCV
- PyTorch
- Bash
- Docker
Device integration:
- Any UAV that supports GNSS modules
Timeline:
- 6 months – R&D phase
- 2 months – POC development
- 4 months – software development
- Total duration: 12 months
Team:
- 1 project manager
- 2 C/C++ developers
- 1 Python developer
- 1 computer vision R&D engineer
- 1 machine learning R&D engineer
- 2 hardware engineers
- 1 DevOps specialist
Value delivered to business
By integrating advanced algorithms, our solution unlocks opportunities for successful autonomous operations – no failed UAV missions in unstable GNSS areas.
Military-focused applications:
Fully autonomous attack drone
While the existing solution provides guidance only during the final flight segment, our module, if integrated with precise object detection, also enables the drone to independently fly toward the approximate target area. And once the target is identified and nearby, the drone automatically switches to autonomous attack mode – all without operator intervention.
A stabilized FPV hover
The solution had been primarily designed to be pilot-controlled and provide an autonomous position hold – but with our module, it can also perform autonomous missions, even without operator-controlled navigation. For example, it can fly into regions with EW activity and activate position hold when necessary.
Reconnaissance missions
The module can extend reconnaissance drones to fly to waypoints, record footage, and return back without GNSS coverage.
Reliable long-range strike drones
GNSS-signal jamming has proven to be a highly effective countermeasure against long-range strike drones – without continuous trajectory corrections from satellites, the drone’s navigation precision rapidly degrades. Within such challenging conditions, our module can provide target localization and maintain flight accuracy.