Drop stale frames, don't queue them
The camera runs at 30 fps but YOLO manages ~10 on CPU. A one-slot mailbox always hands the detector the newest frame, so the preview stays smooth and latency stays flat instead of growing.
How we built it
See, then listen, then decide
Three small programs on three devices — a vision app on the laptop, firmware on an ESP32, and a BirdNET server — each doing one job and testable on its own.
The pipeline
From a flicker of wings to a saved photo
- 1 · LaptopWatchThe webcam feeds YOLOv8, filtered to COCO's single “bird” class.
- 2 · Laptop → ESP32TriggerA sighting calls the ESP32's /trigger endpoint over the local network.
- 3 · ESP32 + INMP441Record5 s of 48 kHz audio over I2S, DC-filtered and turned into 16-bit WAV on the fly.
- 4 · ESP32 → serverUploadThe clip streams straight into a multipart POST to /analyze — no SD card needed.
- 5 · FastAPIIdentifyBirdNET scores 3 s windows and the server filters them down to our ten targets.
- 6 · FastAPI → ESP32DecideA target scoring ≥ 0.70 is “confirmed”; the ESP32 answers CAPTURE, otherwise IGNORE.
- 7 · LaptopCaptureOn CAPTURE the laptop saves the frame. Everything else is discarded.
Measured on our dev laptop
A few real numbers
9–14fps
YOLOv8n detections on a Ryzen 5 5500U CPU at 480 px, no GPU
~25×real-time
BirdNET on the same laptop once warm: ~2–3 s per minute of audio
5s @ 48 kHz
per clip — longer than BirdNET's 3 s window so calls aren't cut in half
0.70threshold
score at which a target bird counts as confirmed
Throughput varies with thermal throttling; on battery the laptop sits at the low end. Accuracy will be measured in the field before we quote it.
Design decisions
What we chose, and why
Overlap the listening windows
BirdNET judges 3-second windows. The ESP32 asks for 2 s of overlap, so a window starts every second and a call that straddles a boundary is still heard whole.
A small classifier on BirdNET's embeddings (next)
Stock BirdNET picks among 6,522 species, and our test recordings of Black Kite and Black Drongo peaked at just 0.43 and 0.51 — below the 0.70 bar. The training pipeline for a classifier that only chooses between our targets and “background” is written; training it should lift clear calls well above that, and can teach it species BirdNET lacks.
Pick hard negatives on purpose
Background training audio includes look-alikes such as Ashy Drongo, Shikra, Tree Sparrow and Rose-ringed Parakeet, plus wind and noise. Positives are copied with added noise to mimic a cheap mic outdoors.
Keep the range filter from hiding our birds
BirdNET's location filter is built from wild sightings and would drop domestic geese and even peafowl in Assam. Our targets always stay eligible; every other species is still filtered.
Say “can't detect”, not nothing
Greater Adjutant and White-winged Duck aren't in BirdNET. The API still lists them with available_in_model: false, so an app shows the truth instead of a silent “not heard”.
Stack
What it runs on
- Vision
- Python, OpenCV, Ultralytics YOLOv8n (CPU-only PyTorch)
- Firmware
- ESP32 (Arduino core, ESP_I2S + WebServer), INMP441 I2S MEMS mic
- Server
- FastAPI, birdnetlib (BirdNET v2.4), TensorFlow, librosa
- Classifier
- Trained on BirdNET embeddings from xeno-canto recordings
- Website
- Next.js, Tailwind CSS, Framer Motion, S3 for the image gallery
Status
Where we are today
- YOLOv8 bird detection running live on the laptop webcam
- ESP32 records from the INMP441 and streams the clip to the server
- FastAPI + BirdNET server with target filtering and the range-filter fix
- Custom target classifier — the training pipeline is written, the model isn't trained yet
- ESP32 replies CAPTURE / IGNORE from the server's decision
- Detector calling /trigger automatically — today we trigger it by hand
- Automatic photo on CAPTURE — today frames are saved with the s key