How AI Is Doing Magic in CV Pipelines

One evening, about a dozen prompts, and an AI assistant took a shaky 5-second phone clip all the way to a calibrated behaviour model. That used to be several weeks of work for a student research team. Here is how it went, step by step.

We all know the usual menu of a CV/ML shop: object counting, object tracking, defect detection on a production line, people-flow analytics in retail, document and license-plate OCR, pose estimation, video stabilization, anomaly detection. At ABTO we have delivered most of these many times. The menu is not what is changing. What is changing is how fast we get from “interesting video” to “measured, explained and simulated behavior”.

This small story shows it better than a slide deck.

Google Photos recently reminded me of a 6-year-old video, shot from the window of my apartment while the building was still under construction. On the glass sat a strange swarm of tiny, Drosophila-like flies. Most likely they are Thaumatomyia notata, a 2–3 mm grass fly that gathers on tall buildings every autumn while looking for a place to spend the winter. They are harmless, but they come back every year and annoy every neighbor who keeps a window open.

The way they moved was magical. They do not swarm like a flock. They avoid each other, stop, wait and go again, and the whole picture looks oddly polite. So I thought: why not play with them using Claude?

The setup was simple: one session in Claude Code, and one model, Claude Sonnet 5.5 at default effort, for every step. Sonnet is the brain. Claude Code is the harness: the hands that run code, read files, search papers and test the result in a browser.

The original clip: 5 seconds, 146 frames, about 240 flies on the glass at any moment.

Step 1. Pull the flies out of the picture

OK, let’s first build a tool that uses simple CV tricks to extract the flies from the video. My prompt was short: flies are the darkest small blobs, so give me a tool with size and darkness thresholds, a fly counter and a player to step through the frames. Split the video into frames first.

A few minutes later I had 146 lossless frames, a reusable detector and an interactive tuner. On my machine OpenCV had been built without its window module, so the first version crashed. I pasted the error, and Claude rewrote the GUI on tkinter within a minute.

Then came the funny part. Claude overthought the dark background from the first minute. It added a local-contrast filter (a pixel must be darker than its own neighbourhood) to reject the dark building windows, and cut the frame down to the sky by default. I asked for something simpler: a quadrilateral ROI editor, config export, and more knobs so I could tune it myself. Two thresholds (a loose one to grow a blob and a strict one it must contain), colour channel, blur, morphology, length limits and a live mask preview.

Manual tuning worked. The final recipe fits in one line: grey level 74 or darker, at least 50 levels darker than the local background, blob area 7–44 px. Claude’s contrast check survived, but as one slider among many rather than the whole strategy.

What Claude added on its own were the blob size filters. Every connected dark blob is classified by its area: green is a fly, red is too big (two flies merged), yellow is too small (noise or dirt on the glass). When I asked for fly orientation, it fitted an ellipse to every blob, so each fly got a body axis: a median fly is 8.5 × 2.8 px.

Left: raw pixels. Right: what the detector sees. Green circle = one fly, yellow = too small to be a fly, magenta line = body axis.

First surprise: I guessed fewer than 100 flies. The counter said about 240 per frame, or 34,866 detections across the clip.

Step 2. Stop the camera from shaking

Done, the flies are extracted as blobs. But the video shakes: it is a hand-held phone clip. For counting that doesn’t matter. For movement it is fatal: a fly walks about 1.7 px per frame, while the camera drifts about 25 px sideways and 29 px down in 5 seconds.

Attempt 1: the window frame. 

I asked Claude to stabilise on features in the upper part of the frame, where the window frame is. It masked the flies out, picked up to 600 corner features in a 100 px strip, and followed them from frame to frame with pyramidal Lucas–Kanade optical flow, checked forward and backward. A RANSAC fit gave rotation plus shift. Then every frame was refined against frame 0 with ECC, a brightness-insensitive alignment, so small errors don’t pile up. That meant 300–480 good features per frame and sub-pixel residuals. It looked done.

Still shaking.

Attempt 2: the building facade. 

So I marked the dark glass facade of the building opposite with the ROI editor and exported it for Claude as an extra reference. Claude really used it: it tracked the facade with 570–600 features per frame and near-perfect correlation. Then it refused to use it for the final result, for a good reason. The facade is far away, so it only shows the camera’s rotation. The glass with the flies is right in front of the lens, so it also moves with the camera’s translation (parallax). The two disagreed by up to 12 px horizontally and 29 px vertically. Claude kept the window frame as the reference, kept the facade as a diagnostic, and checked the result against the flies themselves: a cloud of flies sitting on the glass should not slide as a whole.

Three candidate anchors. The window frame sits at glass depth. The facade is far away and moves differently. The flies sit on the glass itself.

Attempt 3: let the flies stabilise the video. 

Then I had an idea. Flies can’t move like crabs. They crawl along their body axis. So any sideways motion of a fly, measured against its own body, is the camera, not the fly. With 200+ flies at every possible angle in every frame, their sideways components pin down the camera motion.

Flies can’t walk sideways, so their sideways motion measures the camera

The fly-based anti-jitter trick in three steps.

Claude turned this into maths in one go. It fits shift, rotation and scale from the sideways motion of about 200 flies per frame with a robust least-squares solver, to about 0.04 px per step. To know which fly is which between frames it also had to build a tracker (Step 3). Finally it fused both signals the way a drone fuses its sensors: slow drift from the window frame, fast jitter from the flies.

Flies-only scores best on this test because it was fitted to it, but it cannot see a slow drift shared by all flies. So the fused version won. Done, looks perfect. When I looped the video I still saw a jump at the end. Claude traced it to two frames where the window-frame fit slipped, plus a real parallax snap of the far building when the loop restarts.

Step 3. Track identical shadows

Now let’s analyse how they move. The flies are identical dark shadows. That is exactly why they look magic, and exactly why they are hard to track: no colour, no texture, nothing that tells one fly from another.

Claude, track them. Boom, done, and visualised with colours. In fact the tracker already existed, because Claude had built it for the fly-based stabilisation. It matches every fly to its position in the next frame with the Hungarian algorithm, using the same “no crab walking” rule: moving along the body is cheap, moving sideways is expensive, and the body axis may not turn much. Linking and camera estimation take turns: link, fit the camera, link again.

Every fly gets its own colour and a fading trail. Slowed down 2×.

The tracker found 282 tracks that last 30 frames or more, and 149 that last 100+ frames, which is almost the whole clip. Second surprise: 88% of the long tracks head up the window. Claude recognised this as negative geotaxis, the well-known habit of small flies to climb against gravity.

Step 4. Not cars, ships

Nice. Watching the tracks, I got the idea that they move like cars at an intersection. They stop and wait. When they wait too long, they turn. If it takes even longer, they fly away to find a better spot. Let’s invent some parameters for this behaviour. But first let’s check academic papers, something should be there. (Maybe it was better to check before even starting, but we are an innovation office: we allow ourselves to reinvent the bicycle from time to time.)

Bingo, they are not cars, they are ships! A 2016 study by Tehrani-Saleh and Adami is literally titled “Flies as Ship Captains?”. Ships on crossing courses follow a give-way rule, and walking flies have one too. A fly that sees another fly moving back-to-front across its view stops, and the other one keeps going. Zabala et al. (2012) discovered this with a fly-sized robot, and Tehrani-Saleh and Adami evolved digital flies to understand why such a rule pays off.

Then Claude went on, on its own, to calculate the parameters of our swarm. For every pair of flies on crossing courses it predicted who would reach the crossing point first, then measured what each fly actually did next.

Our flies follow the ship rule: the one that would arrive second gives way. Claude also found that 47% of stops happen with another fly less than 25 px ahead (26% would be chance), and that flies stop about 2 body lengths behind the fly in front. The baseline: a fly is stopped 18% of the time, walks in bouts of about half a second, and pauses for about 0.13 s.

My “turn after waiting too long” idea got only weak support: 36% of flies turn by more than 45° after a blocked stop, against 21% after a free stop. Flying away does show up in the clip, but we never extracted that signal from the video (more on that in Step 5). Claude reported the weak result plainly instead of dressing it up.

Step 5. Simulate the swarm

So why don’t we simulate the flies? Claude built an agent-based simulator. Every fly is a small state machine: walk, pause, give way, turn on the spot, take off. Flies crawl only along their body axis, with a pull upward. A fly gives way to a crossing fly that will reach the crossing point first. After waiting longer than its patience, it turns towards the freest direction. After repeated blocks, it takes off.

Then Claude did what a good researcher does: it ran exactly the same analysis on the simulated tracks and tuned the model until the statistics matched the real ones.

Left: the real clip, stabilised. Right: the simulator with our parameters. Spot the difference.

The only real difference is gravity: the real flies climb and pile up near the top of the window, while the simulated ones wrap around the edges of the arena. We left this out on purpose, because density was not one of our parameters.

The rows that were not tuned matter most: the model reproduces them on its own. Claude also ran ablations, switching rules off one at a time. Surprise: without the explicit give-way rule the asymmetry stays, because simple braking for anything ahead produces the same pattern. A 5-second clip cannot tell predictive give-way from reactive braking. Claude flagged this itself, which is exactly the kind of honesty you want from a research assistant.

Gosh, they are perfect. But they are not flying, and that is the missing cherry on the cake. In the first version a fly that took off simply vanished and reappeared elsewhere.

Gosh, they are perfect. But they are not flying, and that is the missing cherry on the cake. In the first version a fly that took off simply vanished and reappeared elsewhere.

Let them fly, I said, but take the flight dynamics from the literature. I didn’t want Claude to extract flight fragments from the video (though I knew how). It searched, read and scaled everything to fly size, with one body length of 2.5 mm:

• Walking at 15 mm/s, inside the 7–45 mm/s range measured by Mendes et al. 2013. Our own tracks gave the same, which supports the 2.5 mm assumption.
• Cruising flight at 0.26 m/s, from Mronz & Lehmann 2008.
• Touchdown at about 7 cm/s and braking within about 1 mm, from van Breugel & Dickinson 2012.

Nice, they fly, but it looks more like jumps. Real flies don’t hop: they fly a few round loops or arcs and then land. Make them loop. Voilà. Each flight now plans 10 random routes of 2–3 arcs, 5–12 body lengths in radius, and picks the one that lands on the freest spot of the glass. A typical flight lasts about a second.

Eight simulated flights: launch (black dot), a few loops, braking, touchdown (red triangle).

Colour coding on: grey walks, blue pauses, red gives way, orange turns on the spot, magenta is about to take off, teal is in the air. Slowed down 3×.

Try the Fly Swarm Simulator Yourself

What this means for CV/ML delivery

One evening, seven work sessions. The laptop was closed for almost three hours in the middle.

How would a classic team do the same? 

Here is a hypothetical junior team we could staff such a research spike with:

• two junior engineers at 0.8 FTE each, one of them strong;
• a team lead at 0.1 FTE who reviews the work and unblocks it.

Hypothetical: the same scope for two juniors (0.8 FTE each) and a team lead (0.1 FTE).

The estimate assumes about 40 working days of junior time and 3–4 days of the team lead’s, with a kick-off and a review every two weeks. Even split between two people, the classic CV part (Steps 1–3) takes about 4 weeks, and everything about 7. With Claude, it took about two hours of Claude’s time and one evening of mine.

To be fair, that evening also included an experienced lead’s ideas. The juniors in the estimate would have to find them on their own.

The outcomes

• Speed. More than a month of work for our usual student research team was done in a few hours of one evening, with about a dozen prompts. The classic CV part, detection plus stabilisation plus tracking, took about an hour.
• The human still matters. The key ideas came from human intuition: a two-threshold detector, the facade, “flies can’t walk like crabs”, traffic at an intersection, loops instead of hops. Claude turned each idea into working, validated code within minutes. When an idea was wrong (the facade), it pushed back with data.
• Research is built in. Literature search, statistics with confidence intervals, simulation, validation and ablations used to be separate phases done by separate people. Here they were part of one conversation.
• Autonomy is next. If we let Claude build sub-agents and give it a good plan, this can run fully autonomously, and even the paper can be written. There is some novelty to report: video stabilisation by the flies themselves, a “turn after waiting too long” behaviour we could not find in the literature, and a simulator that reproduces statistics it was never tuned on. All of it can be checked with a longer clip.

For our clients this changes the economics of custom CV/ML work. Discovery, proof of concept and the “is this even possible?” spike shrink from weeks to days. Our engineers spend their time where it matters most: understanding the domain, asking the right questions and judging the answers.