Multi-Camera Video Collection for Robotics Training: Rigs, Sync, Calibration

Start from what the deployed robot will see. Supervision views come on top of that, and the specification should list the two groups separately. A policy may train on only the wrist and one external view, while the other cameras exist to produce labels. That split decides which cameras need production-matched lenses and mounting, and which only need good geometry.

Three layouts cover most programs:

Placement is where multi-view programs save or waste the most money. The target is that every point that matters (fingertips, contact points, the gripper) stays visible in at least two cameras through the whole task, and ideally three so one can be occluded. The rules that get you there:

Synchronized multi-camera recording has to solve two problems: exposing frames at the same moment, and stamping them on one clock. Some methods do both; some only the second. The accuracy figures below are typical ranges, and actual results depend on the cameras and network.

Views that disagree on timing or colour are harder to fuse and easier for a model to tell apart. Run every camera at the same constant frame rate, or at integer multiples if one view needs high speed, and reject variable-frame-rate footage from phones or action cameras at ingest. Lock exposure, gain and white balance per session instead of letting each camera adapt on its own.

Mixed camera models complicate colour because their sensors and image processing differ. A colour reference chart captured at session start lets you correct differences later and documents the settings. Choose shutter speed for the fastest motion in the task, then add light until gain is acceptable. Mains power runs at 50 Hz in all four APAC markets where we recruit participants (Vietnam, Thailand, Singapore and Malaysia), so studio lighting should be flicker-free or exposures set in multiples of 10 ms to avoid banding.

Intrinsic calibration fixes each camera's focal length, principal point and lens distortion. Extrinsic calibration fixes where every camera sits relative to the others and to the robot. Between them, Kalibr (which handles camera systems whose fields of view do not all overlap) and OpenCV with ChArUco boards cover most needs. Wrist cameras also need hand-eye calibration to the robot flange.

Healthy calibrations typically show reprojection error under about 0.5 px RMS. The more useful signal is change between start and end checks: a camera that moved mid-session invalidates that session for triangulation even if every image looks fine. Version every calibration and link each episode to the one valid when it was recorded. A workable cadence:

Multi-camera rigs hit bandwidth limits before disk limits. One uncompressed 1080p stream at 30 fps is roughly 750 Mbit/s, so five of them exceed a 1 Gbit link and can saturate a single USB 3.0 controller. Production rigs encode at the camera or capture host and spread cameras across controllers or network ports. Everything writes to local SSDs first.

Storage math is simple and worth doing early. Take a five-camera rig (one wrist, three external, one egocentric), each streaming 1080p30 at about 25 Mbit/s after H.265 encoding. That is 125 Mbit/s, roughly 56 GB per hour. A 1,000-hour program produces about 56 TB of video, and closer to 120 TB once a backup copy and derived files are counted. Depth streams or higher bitrates push those numbers up quickly.

Plan the transfer path too. Capture SSDs get copied to network storage every night with a checksum at every hop, and automated QA runs within a day so a failing camera is caught before the next session.

Multi-view datasets fall apart at the file level more often than at the camera level. Fix the naming scheme and episode structure before the pilot and never rename cameras mid-program.

The three formats robotics teams ask for most all handle multiple cameras, in different ways. LeRobot's v3 dataset format stores state and action data in Parquet and camera frames as MP4 sharded per camera, with metadata mapping episodes into those files; each view appears as its own observation.images key. RLDS stores episodes as sequences of steps, and each step's observation can hold one image entry per camera. MCAP is an open container for timestamped multi-channel logs and the default log format in ROS 2, so each camera gets its own channel, as do its calibration and every other sensor stream.

Many programs take two deliveries. One is raw MCAP or ROS 2 bags for reprocessing; the other is a training-ready copy in LeRobot, RLDS or HDF5, with episodes segmented and calibration attached. Validate both on pilot data in the client's own training code before production starts.

On our side, DataX Power runs third-person arrays with synchronized timestamps for multi-angle 3D pose, wrist-mounted and overhead manipulation views (synchronized with force/torque logging where needed), egocentric rigs, and on-board plus chase-camera navigation capture. We handle hardware, participants, consent, QA and delivery ourselves. A pilot specification typically comes back within five business days, and onboarding takes about two weeks.