Stereo Camera Egocentric Data Collection: Lessons From Real Programs (2026)

Stereo camera egocentric data collection means recording a task from the wearer's point of view with two synchronized, calibrated cameras a known distance apart, usually mounted on the head. The pair turns every frame into a depth measurement in metres rather than a guess. If you need 3D hand trajectories and object poses from human demonstrations, grasp geometry included, that decides whether the footage can supervise a policy or only illustrate one.

At DataX Power we have run stereo camera egocentric collection and multi-camera video collection for robotics training out of Vietnam. That egocentric data collection work has taught us where stereo earns its extra cost and which rig decisions are hard to undo. Very little of it is exotic. Programs go well when the dull checks run every session and catch problems before a batch ships.

We wrote it for robotics and data leads scoping a head-mounted stereo camera program for imitation learning or a vision-language-action (VLA) model, and sometimes a 3D hand-object perception stack.

The structural weakness of monocular egocentric video is scale. A single camera cannot tell a small cup near the lens from a large cup further away. A model has to infer depth and 3D hand pose, and its errors tend to be worst where robot learning needs precision: the last centimetres before a grasp, and the frames where the hand hides the object.

A calibrated stereo pair removes most of that ambiguity inside its working range. Disparity between the left and right images converts directly to metric depth, so wrist position and object location can be labelled in 3D instead of projected from 2D, and the approach direction comes with them. Meta's HOT3D benchmark, recorded on head-mounted devices with multiple cameras, reported that multi-view methods significantly outperform single-view counterparts on 3D hand tracking and object pose.

Stereo is not always the right call. Active RGB-D handles textureless indoor surfaces better. Monocular rigs are cheaper and lighter, and for activity recognition they are good enough.

Four rig families cover most programs. The figures below come from vendor documentation; check the current revision before buying.

Stereolabs ZED Mini: 63 mm baseline, about 60 g, USB 3.0 Type-C and an 800 Hz IMU, recording up to 720p at 60 fps or 1080p at 30 fps on a synchronized rolling shutter. It is light enough for a head strap and easy to host on a laptop.

Stereolabs ZED X Mini: 50 mm baseline, global-shutter sensors at up to 1920x1200 and 60 fps, a 200 Hz IMU and multi-camera hardware sync over GMSL2. It weighs 150 g and needs a GMSL2 capture host, so it belongs on studio rigs and robots more than on someone's head all day.

Meta Project Aria (Gen 1): 75 g glasses with two global-shutter monochrome SLAM cameras (150 degree horizontal field of view, 640x480) angled outward with partial stereo overlap, a rolling-shutter RGB camera, eye tracking and two IMUs on one device clock. Hand-zone depth is weaker than from a forward-facing pair. Wearability and on-device sync are where Aria is hard to beat. Meta says Aria Gen 2 widens stereo overlap to 80 degrees.

RealSense and custom pairs: RealSense D435i and D455 add an IR projector to a global-shutter stereo pair, which helps on plain indoor surfaces. Custom rigs put two machine-vision cameras on a rigid bracket with a shared trigger. You choose baseline, lens and shutter exactly, and validating the mount becomes your job.

Stereo depth error grows with the square of distance and shrinks with baseline and focal length: depth error = distance squared x disparity error / (focal length in pixels x baseline). Plug in illustrative numbers (a 63 mm baseline, a focal length near 640 px, a quarter-pixel matching error) and you get roughly 1.5 mm of uncertainty at 0.5 m, about 2.5 cm at 2 m and around 15 cm at 5 m. Real error is higher once weak texture and motion blur come in, and drift makes it worse, but the curve holds.

Egocentric manipulation happens close in. Hands and task objects mostly sit roughly 0.3-0.8 m from the eyes, where a 50-65 mm baseline performs well. Wider baselines help at room scale, but they push the minimum usable distance out and shrink overlap near the face. That is where the fingers and small objects are.

Write the working distance into the specification before choosing hardware, then verify it in the pilot. Put a calibration board and known objects at three depths across the task zone (near edge, middle, far edge) and measure depth error at each.

A head moves faster than most robot arms. Rolling-shutter sensors read out line by line, so a quick head turn skews geometry within a frame, and if the two eyes do not read out in lockstep, disparity is corrupted. For scale, Aria's documentation gives its RGB camera a readout of about 5 ms at 1408x1408 and 15 ms at 2880x2880. Global shutter avoids the problem. If weight or cost points you to a rolling-shutter pair, keep exposure short and confirm synchronous readout, then model the shutter in calibration (Kalibr supports this).

Sync has two layers. Inside the pair, both eyes must expose at the same instant; integrated cameras handle that on the board, and custom pairs use a shared trigger. Outside the pair, the IMU, any wrist or third-person cameras and force/torque need timestamps on one clock. Use hardware triggering or PTP where supported, and record an LED flash or similar sync event at the start and end of each session so the offset is measured instead of assumed. A common threshold is under half a frame period, about 16 ms at 30 fps, tighter for contact-rich work.

The IMU is worth having. Fused with stereo through visual-inertial odometry, it gives a 6-DoF head trajectory, so hands and objects can be placed in a world frame instead of a bobbing camera frame. Calibrate the camera-IMU extrinsics and time offset (Kalibr handles this too) and deliver both with the data.

Treat factory calibration as a starting point. Stereo depth depends on the exact rotation and translation between the two cameras. A knock can shift that, as can a warm-up cycle or a strap flexing the mount, and neither image will look wrong.

The routine that holds up starts with a full intrinsic and stereo extrinsic calibration for each camera unit at program start, using a ChArUco board with OpenCV or an AprilGrid with Kalibr. After that, every session gets a short board capture at its start and end. Healthy rigs typically show reprojection error under about 0.5 px RMS, and vertical offset between matched features in the rectified pair under about one pixel. Comparing start and end values shows whether the session drifted.

Ship the calibration log with the dataset, or nobody downstream can check the depth. A session that drifted beyond tolerance can still be used as monocular footage, but it should not ship as metric 3D data. Let cameras warm up for several minutes before the first capture. All of this is why stereo belongs in onsite or studio collection, while crowdsourced collection suits simpler wearable kits.

Stereo matching assumes a point looks the same in both images. Independent auto-exposure or auto white balance breaks that assumption: each eye adjusts on its own, and the matcher reads the brightness difference as texture or noise. Integrated cameras share settings. Custom pairs need one camera driving the other, or both locked manually.

Lock exposure, gain and white balance per session and record them in metadata. Choose shutter speed for the hands, not the room. Then deal with flicker. Vietnam, Thailand, Singapore, Malaysia and Australia run 50 Hz mains, so many indoor lights flicker at 100 Hz, and exposures shorter than 10 ms, or not a multiple of it, can band or pulse. Use flicker-free studio lighting, or a 10 ms exposure with enough light to keep gain low.

A head-mounted stereo camera is worn for hours, and comfort shows up in the data. Participants who feel front-heavy weight or warmth on the forehead start to move stiffly. They hold the head still and turn their eyes instead, which shifts the viewpoint distribution the model learns from.

Stereo egocentric video doubles the stream before anything else is added. Uncompressed, a 1080p pair at 30 fps runs close to 190 MB per second, roughly 670 GB per hour. With good H.265 encoding at 20-40 Mbit/s per eye, the pair lands around 18-36 GB per hour. Device-computed 16-bit depth maps can multiply that several times.

Compression is not neutral for stereo. Heavy compression smears the fine texture that matching relies on. Skin and plain tables suffer most. Record the pilot near-lossless and compute depth twice, once from that and once from the planned production bitrate. Then pick the lowest bitrate that leaves hand-zone depth error unchanged.

The pipeline has to be sized too. Each rig needs local SSDs that hold a full day of capture, and no card gets wiped until a verified copy exists. Checksum every transfer and run QA within a day. As a planning example, a 1,000-hour program at 30 GB per hour is 30 TB before backups and derived data.

Training teams need enough to reproduce the geometry, which the video alone does not give them. MCAP or ROS 2 bags suit teams that replay raw streams; HDF5, RLDS or LeRobot format suit imitation learning pipelines. Many programs take both. Whatever the container, a usable delivery includes:

General video QA still applies. At DataX Power, QA engineers trained on robotics video review every batch. They check temporal consistency and frame quality, and they confirm task coverage. Because we handle hardware, participants, consent, QA and delivery end to end, the stereo checks run inside that same pipeline. Stereo adds four:

Robotics and data leads scoping a stereo egocentric program tend to ask us the same few questions.