Why VLA projects fail before training begins
Vision-Language-Action models - the architecture behind systems like pi-zero, OpenVLA, and Octo - are rewriting what robots can do. A well-trained VLA can take a natural language instruction ("pick up the red cup and place it on the tray") and execute it across physical environments it has never seen. That capability is not magic. It is the product of carefully collected, precisely annotated, temporally consistent training data.
Yet most enterprise robotics teams that engage a VLA training data service arrive with the same gap: they understand the model architecture and the evaluation benchmarks, but they have no clear specification for the data collection pipeline that feeds the model. They know they need egocentric video, depth data, and action labels. They do not know what quality thresholds, synchronization tolerances, or annotation protocols to demand from a vendor.
That gap is expensive. A data collection run that produces misaligned RGB-D streams, inconsistent action segmentation, or footage shot at the wrong camera height can invalidate months of model training. This guide sets out exactly what a VLA training data service must deliver - so you can evaluate vendors before you sign, not after you have burned your pilot budget.
The four data types every VLA training pipeline requires
A production-grade VLA training data service collects four distinct data streams in parallel. Each must be synchronized to sub-100ms precision or the model learns spurious correlations between observation and action.
- Egocentric RGB video - First-person perspective at 30fps minimum, ideally 60fps for fast manipulation tasks. Resolution at 1080p or higher. The camera must be mounted at the operational height of the robot end-effector, not at human eye level. This is the most common specification error in vendor-collected datasets.
- Depth data (RGB-D) - Paired depth frames from a structured-light or time-of-flight sensor (Intel RealSense D435i, Azure Kinect, or ZED 2). Depth must be temporally aligned with RGB to within one frame. Missing or noisy depth frames are the leading cause of poor spatial grounding in VLA outputs.
- Proprioceptive and action labels - Joint angles, end-effector pose (6-DOF), gripper state, and force-torque readings at 100Hz or higher. These labels are what turns video observation into an action policy. A data service that delivers video without synchronized proprioception is delivering footage, not training data.
- Language instruction annotations - Natural language task descriptions paired with each demonstration segment. Annotations must follow a controlled vocabulary agreed before collection begins. Inconsistent instruction phrasing across demonstrations directly degrades language-conditioned policy performance.
Human demonstration data vs. synthetic data for VLA training
The most common question enterprise teams ask when evaluating a VLA training data service is whether they can substitute synthetic data for real human demonstration data. The answer depends on what stage of development you are in.
Synthetic data generated from simulation (Isaac Sim, MuJoCo, Genesis) is effective for pre-training on large-scale diverse scenarios where physical realism is secondary. It is cheap, fast to produce, and does not require a physical data collection team. The problem is sim-to-real transfer: policies trained purely on synthetic data degrade significantly when deployed to physical hardware, particularly for contact-rich manipulation tasks where surface friction, object deformation, and sensor noise matter.
Human demonstration data collected in the target physical environment - or an environment that closely matches it - is what enables final-stage fine-tuning that actually holds in production. The best VLA training pipelines use synthetic data for broad capability acquisition and real human demonstration data for domain-specific grounding. A VLA training data service that offers only one of these is giving you half a pipeline.
- Use synthetic data for: pre-training, rare-event coverage, curriculum diversity at scale.
- Use human demonstration data for: domain grounding, contact-rich tasks, real-sensor calibration, final fine-tuning before deployment.
- Budget split typical for production VLA projects: 70% synthetic pre-training data, 30% real human demonstration data. The real data costs more per hour but drives disproportionate gains in deployment reliability.
Data volume requirements: how much is enough
Volume requirements for VLA training data vary by model architecture, task complexity, and whether you are training from scratch or fine-tuning a pre-trained backbone. These are the reference ranges used by teams shipping production VLA systems in 2026.
- Fine-tuning a pre-trained VLA (pi-zero, OpenVLA, Octo): 500-2,000 demonstrations per task type. Each demonstration is typically 30-90 seconds of activity. At 60fps, that is 1,800-5,400 frames per demo.
- Training a task-specific policy from a pre-trained backbone: 2,000-10,000 demonstrations covering the full distribution of environment configurations, object positions, lighting conditions, and failure recovery scenarios.
- Training a generalist policy from scratch: 50,000-500,000 demonstrations. This is the territory of Ego4D, EPIC-Kitchens, and similar academic datasets. Commercially, this scale requires a dedicated data collection program over 6-18 months.
- Coverage diversity matters more than raw volume. 500 demonstrations across 20 object types in 5 environment configurations will outperform 5,000 demonstrations of the same object in the same configuration. Specify diversity requirements explicitly in your vendor contract.
Synchronization and hardware: the spec your vendor must meet
Temporal synchronization is the most technically demanding aspect of VLA training data collection. RGB, depth, and proprioceptive streams recorded by separate hardware must be aligned in post-processing or - better - synchronized at capture time via hardware trigger. Ask your VLA training data service provider the following questions before signing:
- Hardware sync or software sync? Hardware trigger synchronization (e.g., via GPIO trigger on RealSense) achieves sub-millisecond alignment. Software sync via NTP or ROS timestamps is adequate for slow tasks but introduces 20-80ms jitter for fast manipulation - unacceptable for VLA policies that operate at 10-30Hz.
- What is the RGB-D calibration protocol? Intrinsic and extrinsic camera calibration must be performed at the start of each collection session, not once per deployment. Thermal drift in the camera housing shifts calibration parameters over hours. A vendor that calibrates once per week is producing miscalibrated data.
- How are ROS bags structured? If your training pipeline ingests ROS bag format, confirm the vendor structures bags with standard topic naming (/camera/color/image_raw, /camera/depth/image_rect_raw, /joint_states) and that bag timestamps are consistent. Nonstandard topic structures require pre-processing work that delays your training pipeline.
- What is the failure frame protocol? Every collection session will produce frames with motion blur, occlusion, sensor dropout, or annotator error. A professional data service tags these frames at collection time with a quality flag and provides a clean-frame count in the delivery manifest. If a vendor does not offer this, you are doing their QA for them.
Annotation protocols that actually serve VLA training
Raw video and sensor streams are not training data. They become training data when annotated with action labels, task boundaries, and object state changes. The annotation protocol your VLA training data service uses has a direct effect on policy quality.
The most common annotation gap is temporal action segmentation - defining exactly where one action ends and the next begins within a continuous demonstration video. Inconsistent segmentation boundaries create ambiguous state transitions in the training data, which the model learns as uncertainty rather than skill. Require your vendor to define segmentation criteria in writing before collection begins.
- Action segment boundary definition: provide the vendor with a written protocol specifying how to mark the start and end of each atomic action (e.g., "grasp start = first frame where gripper contacts object; grasp end = first frame where object leaves the surface").
- Inter-annotator agreement (IAA) target for temporal segmentation: require a minimum Cohen's Kappa of 0.75 across annotators on a gold-standard test set before production annotation begins.
- Object state annotation: for manipulation tasks, each object in the scene should carry a state label per segment (e.g., "cup: upright", "cup: inverted", "cup: grasped"). This is the data that enables VLA models to reason about task progress.
- Language instruction pairing: each demonstration segment must be paired with 3-5 natural language instruction variants to improve instruction-following robustness. A single instruction per demo produces a model that is brittle to phrasing variation.
- Failure demonstration inclusion: require 10-15% of collected demonstrations to be intentional failure cases with recovery actions. VLA models trained only on success demonstrations fail silently in deployment when the first unexpected event occurs.
APAC data collection: why geography matters for robotics programs
Enterprise robotics programs that need to collect VLA training data at scale have a geography decision to make: collect in the target deployment market (where the robot will operate), or collect in a lower-cost market with similar physical environment characteristics.
For programs targeting APAC deployment - warehousing in Singapore, manufacturing in Thailand, food service in Japan - Vietnam has emerged as the most practical collection base. Labor costs for trained demonstration operators are 60-75% lower than in Singapore or Japan. English-language instruction fluency is high enough to support multilingual VLA training programs. And the physical environment types available (commercial kitchens, warehouse configurations, lab setups) match the target deployment contexts closely enough for domain transfer.
DataX Power runs data collection programs from Hanoi, with project coordination aligned to Singapore, Australian, and US time zones. Pilot programs of 500-1,000 demonstrations can be completed within 3-4 weeks from contract signature.
What to include in a VLA training data service contract
Before engaging a VLA training data service provider, define these requirements in writing. A vendor that cannot commit to these specifications in the contract is not ready for production-grade data collection work.
- Data format specification: exact file formats, directory structure, topic naming conventions, and timestamp format for all streams. Vague "standard formats" clauses create post-delivery negotiation.
- Synchronization tolerance: the maximum allowed timestamp offset between RGB, depth, and proprioceptive streams. For manipulation tasks operating at 10Hz, a 50ms tolerance is the maximum acceptable; 20ms is preferred.
- Clean-frame rate guarantee: minimum percentage of delivered frames that pass quality checks (motion blur threshold, depth validity percentage, annotation completeness). 92% clean-frame rate is a reasonable floor for production data.
- Demonstration diversity specification: number of unique object instances, environment configurations, and lighting conditions covered. Attach a diversity matrix to the contract - not a paragraph description.
- IAA threshold: minimum inter-annotator agreement score on annotation tasks, measured on a gold standard set agreed before production begins.
- Pilot structure: 200-500 demonstrations delivered and evaluated before committing to full production volume. Any vendor that resists a structured pilot is not confident in their own quality.
- Delivery manifest: each delivery batch must include a manifest listing frame counts per stream, clean-frame rate, annotation completion rate, and a per-session calibration report.
