Multimodal Sensor Data Collection for Robotics: Integrating RGB, Depth, Force, and Audio (2026)

The early wave of imitation learning research demonstrated that significant robot manipulation capability could be trained from RGB video alone. Policies trained on single-camera RGB demonstrations learned to grasp, place, and sort objects with impressive performance in controlled environments. This created an assumption that camera data is enough.

The assumption breaks down when robots encounter the conditions that matter for deployment: variable object materials (rigid vs. soft, smooth vs. textured), precision insertion tasks that require sub-millimeter force feedback, bimanual coordination where one hand cannot see what the other is doing, and contact-rich manipulation where success depends on force applied rather than position reached. These scenarios require training data that captures what camera data cannot: the tactile and force signals that humans use constantly during dexterous manipulation.

Multimodal sensor data collection combines RGB cameras with depth sensors, force-torque sensors, audio microphones, and proprioceptive data (joint positions, motor currents, end-effector wrench) to create training datasets that give the policy access to the full information state that determines task success. The cost and complexity are higher than RGB-only collection. For tasks where contact dynamics, material properties, or precision force control are critical to deployment success, the multimodal training signal is not optional - it is the difference between a policy that works in the lab and one that works on the production floor.

Not every task requires every sensor. The right multimodal collection design adds sensors where they provide training signal that RGB alone cannot capture, and omits sensors where the cost and synchronization complexity is not justified by the training benefit.

RGB video remains the primary training modality for most robot policies. The camera configuration that matters most is the viewpoint: egocentric (mounted on the robot head or shoulder), wrist-mounted (at the end-effector, pointing at the workspace), and scene (fixed camera observing the full task area). Different policy architectures weight these viewpoints differently.

For VLA models like pi0, a combination of overhead scene camera and wrist camera is the standard configuration, matching the camera setup of the robot at deployment. For UMI-based programs, the wrist camera is the primary modality and should be mounted to match the deployment robot's wrist camera position as closely as possible.

RGB camera specifications that matter for robot training data: global shutter (rolling shutter creates geometric distortion during fast movements), consistent white balance (automatic white balance correction creates color distribution shift between sessions), and sufficient resolution for the task (1080p for scene cameras, 720p acceptable for wrist cameras in most manipulation tasks). Frame rate should match the robot control frequency - typically 25-30fps for manipulation policies, 60fps for higher-speed tasks.

For programs adding depth to RGB, the RGB-D camera pair should be hardware-calibrated so that each pixel in the RGB image has a corresponding depth value. Software registration of separately mounted RGB and depth cameras introduces misalignment artifacts that corrupt the spatial information the depth modality is supposed to provide.

Depth sensors add per-pixel distance information that allows policies to reason about 3D object position and scene geometry rather than inferring depth from monocular visual cues. The decision to add depth should be based on whether 3D spatial reasoning is required for task success, not as a default addition.

Tasks that benefit from depth data: object localization in cluttered scenes where monocular depth estimation fails, pick-and-place with height-variable objects (stacking, bin picking), precise approach trajectories where the robot must stop at a specific standoff distance, and any task where the policy needs to estimate whether its end-effector is at the correct height relative to a surface.

Depth sensor options for robotics programs: the Intel RealSense D435i is the most widely used in academic robotics (good depth range for tabletop tasks, IMU included, well-supported in ROS and Python data pipelines). The Orbbec Femto Bolt is a successor to the Azure Kinect with comparable performance. Stereolabs ZED 2 provides longer range and better outdoor performance. All require calibration per session to account for thermal drift.

Depth data has known failure modes that QA must catch: infrared interference from sunlight or fluorescent lighting creates depth voids, specular surfaces (shiny metals, glass) produce depth noise, and object edges produce depth uncertainty that can be misinterpreted as distance discontinuities. Programs should include depth-specific QA checks that flag frames with excessive depth voids or depth noise in the task region.

Force-torque (F/T) sensors mounted at the robot wrist measure the wrench (forces and torques in all three axes) at the end-effector. This data captures what happens at contact - how hard the robot is pushing, in which direction, and whether it is maintaining stable contact or slipping. F/T data is essential training signal for tasks where correct force application determines success.

Manipulation tasks that require F/T data in training: precision insertion (USB connectors, bolts, PCB components), compliant surface following (wiping, sanding, assembly along a surface), soft object handling (deformable materials, food, medical samples), and bimanual handoff tasks where one arm applies controlled force to allow the other to grasp.

F/T sensor specifications that matter for robot training data: sampling rate (1000Hz is standard for most industrial F/T sensors - sufficient to capture contact transients at manipulation speeds), noise floor (should be below 0.1N/0.01Nm to avoid masking soft contact signals), and force range appropriate for the task (a sensor rated for 200N will have poor resolution for the 0.1-5N range that most manipulation contact events occur in).

F/T data should be synchronized with RGB camera streams at the hardware level. The contact events that F/T captures (contact initiation, slip onset, detachment) are transient - they occur over 10-50ms. If the F/T and camera streams are only loosely synchronized via software timestamps, the policy cannot learn the visual-tactile correlations that the multimodal training data is designed to teach.

Audio is the least commonly used modality in robot training data programs, but it provides unique training signal for tasks where material properties and contact quality can be inferred from sound. The pop of a correctly seated connector, the scraping of metal-on-metal vs. metal-on-plastic contact, and the thud of a dropped object vs. a placed object are all audio signatures that correlate with task outcomes.

Research programs have demonstrated that policies trained with audio alongside visual data outperform visual-only policies for tasks involving material discrimination (sorting objects by material), connector insertion (detecting the click of successful seat), and surface following (detecting material transitions from sound variation).

For production collection programs, audio capture requires microphones mounted close to the task region (ideally wrist-mounted or at the end-effector), with controlled ambient noise conditions. Open-office environments with HVAC noise and human activity produce audio datasets where the task-relevant sound signals are masked by ambient noise. Acoustic isolation panels around the task workspace are a practical solution for programs where audio is a primary modality.

Audio data synchronization is straightforward via hardware trigger - the same trigger that synchronizes cameras can trigger audio recording start. The typical sampling rate for robot audio data is 44.1kHz or 48kHz, which produces manageable file sizes while capturing the frequency range of contact sounds (most task-relevant contact audio is 100Hz-8kHz).

Proprioceptive data - the robot's own joint positions, velocities, motor currents, and computed end-effector pose - is always included in robot training data for teleoperation programs because it is generated by the robot controller as a matter of course. It is the most underappreciated modality because it is always available and never requires a separate sensor.

Joint position data at 50-100Hz provides the action representation for most imitation learning approaches: what the robot did, frame by frame, expressed in joint space or task space. Motor current data provides a low-cost proxy for torque (and therefore force at the end-effector) that does not require a dedicated F/T sensor. For tasks where F/T sensors are not installed, motor current data can serve as an approximate substitute for contact force detection.

End-effector pose (6-DOF Cartesian position and orientation) is derived from joint positions via forward kinematics and is the action representation used by UMI and most VLA models. Proprioceptive data should be recorded at the robot controller sampling rate (not downsampled to camera frame rate), then temporally aligned with camera frames during dataset processing.

The key QA check for proprioceptive data is timestamp verification: all proprioceptive streams should have monotonically increasing, gap-free timestamps across the episode. Timestamp gaps in joint state data indicate controller timeouts or communication failures that corrupt the action representation for the affected episode segment.

Synchronization is the technical challenge that distinguishes production-quality multimodal data from academic collection programs. In practice, every sensor has its own clock. Without hardware synchronization, the only relationship between sensor streams is the wall-clock timestamp assigned by the data collection computer - accurate to perhaps 1-10ms, which at 30fps corresponds to misalignment of up to one full frame between sensors.

Hardware synchronization uses a shared trigger signal distributed to all sensors. A microcontroller (Arduino, Raspberry Pi Pico, or a dedicated trigger box) generates a precise square wave at the desired synchronization rate (30Hz for cameras, with 1000Hz pass-through for F/T sensors and proprioception). Each camera, depth sensor, and F/T sensor receives the trigger and aligns its output to the trigger edge. This achieves sub-millisecond synchronization across all modalities.

Software timestamp synchronization is easier to implement and acceptable for programs where the sensors are all connected to the same data collection computer with low-latency USB or PCIe connections. In this configuration, timestamps are assigned by the data collection software on sensor event receipt. The typical accuracy is 1-5ms, which is sufficient for tasks where actions occur on a timescale of 100ms or more. It is not sufficient for capturing contact transients (10-50ms) or for synchronizing high-speed camera data (>60fps).

For multimodal programs, the recommended synchronization architecture is: hardware trigger for cameras and F/T sensors, software timestamping for proprioceptive data (already synchronized by the robot controller), and post-hoc alignment of proprioception to camera frames using the nearest-neighbor timestamp matching. This approach provides sub-millisecond accuracy for the modalities where timing precision matters and avoids the engineering overhead of hardware triggering proprioceptive data.

Multimodal data is large. A single hour of collection at modest specifications - 2x RGB cameras at 1080p/30fps, 1x depth at 640x480/30fps, 1x F/T at 1000Hz, proprioception at 100Hz - produces approximately 8-15 GB of raw data before compression. At production scale (100+ hours per program), storage architecture and pipeline throughput are real constraints.

RLDS (Reinforcement Learning Datasets) format, used by pi0, OpenVLA, and Octo, stores episodes as TensorFlow Dataset tf.data.Dataset objects with per-step tensors for all observation and action modalities. The format handles multimodal data well: each modality is stored as a separate tensor within the step dictionary, and episodes are stored as variable-length sequences of steps. RLDS provides efficient sequential access that matches the training loop access pattern of most imitation learning frameworks.

For programs delivering in RLDS, the data pipeline should: compress video streams to H.264 or HEVC during collection (not after) to reduce storage bandwidth requirements, store F/T and proprioception data as float32 tensors without compression (the data rates are low enough that compression overhead is not worthwhile), and validate RLDS episode structure before delivery using the RLDS dataset inspection tools.

QA for multimodal datasets requires per-modality checks: verify that each episode contains all expected modalities (no missing F/T stream, no camera dropout), verify that timestamps are monotonically increasing within each modality, and verify cross-modality synchronization using the hardware trigger marker if one was embedded in the data. Episodes that fail any per-modality check should be flagged for human review rather than automatically rejected - the cause of the failure determines whether the episode is salvageable.