All work

Computer Vision

Teaching Milestone to See Animals

A YOLO detector on the video server's own GPU turns everything that crosses the home cameras into labeled, searchable bookmarks in Milestone XProtect. The deer was the hard part.

Integration paths tried
3
Labels for one deer
3
Cats called cats
100%
CAM 01 · DRIVEWAY CAM 02 · FRONT SIDEWALK STREET HOME CAR 0.80 CAR 0.52 CAT 0.73 CAT 0.87 DOG 0.79 HORSE 0.67 SHEEP 0.45 DOG 0.72 HORSE 0.59 → ANIMAL DETECTOR · VMS-CM · GPU YOLO11L · 1280 · FP16 BOT-SORT + REID · 2 CAMERAS RTX 2080 SUPER · WSL2 DOCKER GPU 38% RAM 100% GENERIC EVENTS · :9091 TOKEN YOLO_DRIVEWAY_ANIMAL YOLO_FRONT_SIDEWALK_ANIMAL NEW TRACK FIRES ONCE, THEN ONLY WHILE MOVING · 5 S CAR 1 · PARKED · SUPPRESSED BOOKMARKS · XPROTECT 12:31 ANIMAL · DRIVEWAY 12:32 ANIMAL · DRIVEWAY 12:32 ANIMAL · DRIVEWAY 12:33 ANIMAL · DRIVEWAY 12:33 ANIMAL · FRONT SIDEWALK 12:34 ANIMAL · DRIVEWAY DAHUA NVR · RTSP YOLO11 · RTX 2080 SUPER FFMPEG NVENC MEDIAMTX :8554 XPROTECT RECORDS IT NEW TRACK · MOVING YOLO_DRIVEWAY_ANIMAL EVENT SERVER :9091 GENERIC EVENT DEF RULE → BOOKMARK OUTPUT A · ANNOTATED STREAM, BOXES BAKED IN OUTPUT B · ONE STRING PER DETECTION, ONE BOOKMARK PER RULE YOLO11 → XPROTECT · DRIVEWAY + FRONT SIDEWALK LIVE · NO DEER IN COCO, SO ANIMAL IT IS

Problem

Milestone XProtect is the platform I provision for enterprise clients, and it is also what records the six cameras at my house. Out of the box, its search understands two kinds of object, people and vehicles, and only when the cameras report them. Consumer cameras do not. Whatever crossed the front yard at half past midnight was in the recording somewhere, and finding it meant scrubbing through hours of video.

The goal: every person, vehicle and animal on the front yard becomes a labeled, click-to-play bookmark in Milestone, with the detection boxes visible on playback, on hardware already on hand. Animals were the point. The deer, the neighborhood cats, the dogs.

FIG 01 · THE FRONT YARD, DAYLIGHT
The front yard in daylight from the Front Sidewalk camera with detector boxes drawn in: a person walking on the grass labeled person 0.86, a side-by-side utility vehicle labeled truck 0.72, two parked cars labeled car 0.80 and car 0.81, and a light-colored dog on the path labeled sheep 0.44, with a black cat beside it

The capture above is the detector’s own output on a June afternoon. Every box is right except one, and that one is the story of this page.

Approach

FIG 02 · SCHEMATIC · DETECTOR TO BOOKMARK
DETECTOR TO BOOKMARK HOME CAMERAS · SIX FEEDS DRIVEWAY AND FRONT SIDEWALK ANALYZED LIVE YOLO11 OBJECT DETECTOR RUNS ON THE VIDEO SERVER'S OWN GPU OUTPUT A PLAYBACK OUTPUT B SEARCH BOXES AND TRAILS A SECOND COPY OF THE FEED ANIMAL · DRIVEWAY ONE EVENT PER DETECTION MILESTONE RECORDS IT PLAYBACK SHOWS THE BOXES MILESTONE EVENT ONE PER CAMERA AND TYPE RULE → BOOKMARK NAMED BY THE EVENT TWO OUTPUTS PER CAMERA, ONE GPU PARKED OBJECTS STOP FIRING SEARCHABLE, CLICK-TO-PLAYBACK BOOKMARKS

The detector is YOLO11, a real-time object detection model, running on the Milestone server’s own GPU, which until then had never done anything but record video. Each camera produces two outputs. The first is a copy of the feed with the boxes and motion trails drawn on, which Milestone records like any other camera, so playback shows exactly what the detector saw. The second is a one-line event for every detection, which a Milestone rule turns into a bookmark on the right camera. Search, open the bookmark, watch the moment.

An early version bookmarked a parked car all night. The final one only fires when an object is new or actually moving.

Milestone offers two official ways to feed it events from outside, and both rejected the integration for different reasons. The third, a plain-text event port, accepted the first message sent to it. Three paths tried, one that worked, and the dead ends documented so they never need to be walked again.

FIG 03 · THE FRONT YARD, INFRARED
The front yard at night on infrared from the Driveway camera, the solar path lights blooming into starbursts, with a black cat on the lawn boxed as cat 0.79 and two parked cars carrying steady car boxes
FIG 04 · DOG 0.54
A deer walking across the infrared lawn, boxed and labeled dog 0.54 as track 22
FIG 05 · HORSE 0.67
The same deer standing mid-lawn, boxed and labeled horse 0.67 as track 21
FIG 06 · SHEEP 0.40
The same deer two seconds later, still track 21, now labeled sheep 0.40

There is no deer in COCO, the dataset the model learned from. It knows cats, dogs, horses, sheep, cows, even zebras and giraffes, but a whitetail is none of those. Faced with a shape it half recognizes, the model picks the nearest thing, and the nearest thing changes with the pose. One deer, one continuous track: a dog, then a horse at 67 percent, then a sheep two seconds later, then a horse again. The cats, which are in COCO, were cats every single time.

None of that indecision reaches Milestone. Every animal class collapses to a single label, ANIMAL, before the event leaves the detector, so the bookmark reads “Animal · Driveway” whether the model saw a horse or a sheep. The clip below is the whole lesson in two and a half minutes, with the cats as the control group.

FIG 07 · THE CLIP · CAT FOUND A FRIEND
Infrared frame of the front yard with a deer standing mid-lawn boxed and labeled horse 59 percent
FIG 08 · WHAT MILESTONE RECORDS
A frame of the annotated stream the detector publishes: a person walking a dog on the lawn, boxed as person 0.87 and dog 0.76, with the parked cars and the utility vehicle boxed behind them
FIG 09 · DOG 0.88
Another frame of the annotated stream with the dog boxed as dog 0.88 beside the utility vehicle boxed as truck 0.74

Result

Proven end to end on the two cameras that matter most: detections in, bookmarks on the right camera, annotated playback recording alongside the originals. The limit turned out to be memory, not compute. The server has 16 GB of RAM, Milestone already uses half of it, and the two detector processes took the rest while the GPU sat at 38 percent with room to spare. The GPU was sized carefully. The host around it was not.

A memory upgrade brings it back. The integration path is settled, and Milestone can already do the one thing it could not: find the animals.