DL based spatial skeleton tracking

I’ve open sourced a spatial skeleton tracing utility macOS app that I have been using for a few exhibitions.

This project aims at providing a low-cost, yet robust, skeleton tracking system for art installations in a bright environment with relatively stable light conditions. While infrared cameras support only short distances (usually less than 4.5m), and stereo cameras like Zed/Intel costs huge computation power (presumably the CNN models for depth estimation are not that GPU-friendly); and they are both quite pricey - this project relies solely on deep learning based approaches with cheap hardware and provide acceptable tracking results.

Skeleton uses macOS vision framework while depth relies DepthAnythingV2. Although DepthAnythingV2 provides only relative values, but in a stable light condition the values are proven to be stable, so with a simple linear regression process it can be mapped to absolute depth even on a 12m scale. Alternatively, you can replace the current model with a fine tuned metric DepthAnything model.

A download link to a prebuilt macOS app is also included for quick test.

1 Like

btw, zlib seems to fail when compiling with xcode16.3, not sure if there is an easy fix.