Why the data collection specification decides the outcome of a video program
Most failed video programs were in trouble before anyone picked up a camera. The problem was already in the data collection specification that went to the vendor. A spec that says "diverse kitchen footage, HD, 200 hours" gets you three quotes that cannot be compared. Later it gets you a dataset that cannot be audited, because nobody wrote down what "diverse" or "usable" means.
This video data collection template is for whoever owns data on an ML or robotics team and needs one document that any vendor can price against the same scope. There are ten blocks. Each one says what to specify, with fill-in lines and example values taken from a typical robot learning program. The last line of every block is the mistake that shows up most often in spec reviews. Copy the structure and swap in your own values; delete what does not apply.
Keep the spec and the capture protocol apart. The spec states what the dataset must contain and how it will be judged. The capture protocol (scenario scripts, rig setup, participant briefing) is the vendor's operational answer, derived from your spec and approved by you. Leave the spec open through the pilot, then freeze it before production.
1. Model objective and success metric - what the data must make possible
What to specify: the model or policy this data trains and the capability it should gain. Then name the metric that will show whether the data worked. Vendors make dozens of small calls in the field, like whether a fumbled grasp counts as a valid episode, and they only make those calls correctly when they know what the model is for.
- Model type: ___ (example: diffusion policy for bimanual tabletop manipulation, fine-tuned from a VLA base model)
- Target capability: ___ (example: pick, fold, and stack 12 garment types from a cluttered bin)
- Offline metric: ___ (example: action prediction error on a split held out by environment)
- Online metric: ___ (example: task success over 50 real-robot trials in 3 unseen rooms)
- Data role: ___ (pretraining mix, fine-tuning set, evaluation set, or a stated split between them)
- Common mistake: leaving this block blank because the project is confidential. Architecture details can stay private. A vendor who cannot tell a pretraining mix from an evaluation set, though, will optimize for volume when you needed precision, or the reverse.
2. Task taxonomy - the activities and where each one starts and stops
What to specify: every task to be captured, with written start and end conditions. Where it helps, break tasks down into sub-tasks and atomic actions. Boundaries are what turn continuous video into learnable episodes, which is why they belong in the spec and not in an annotation brief written later.
- Task list: ___ (example: 8 kitchen tasks, 3 laundry tasks, 4 warehouse tote tasks)
- Start condition: ___ (example: both hands visible and empty, target object at rest)
- End condition: ___ (example: object released in target zone, hands withdrawn at least 20 cm)
- Failed attempts: ___ (example: kept and tagged, capped at 15% of episodes per task)
- Share per task: ___ (example: no single task above 20% of total hours)
- Common mistake: a task list of bare verbs such as "cooking" or "tidying", with no boundaries. Give two participants the same verb and they record episodes of very different length. That footage cannot be segmented consistently without re-annotation.
3. Viewpoints and hardware - where the cameras sit and what they are
What to specify: the capture mode and where each camera is mounted. Say how calibration is done, and whether the hardware has to match your deployment robot. Give every rig type its own lines, whether it is an egocentric rig, a third-person array, robot-view cameras on the wrist or overhead, or a mobile navigation setup.
- Capture mode: ___ (example: egocentric primary, plus one fixed third-person camera for context)
- Device: ___ (example: head-mounted rig or Meta Aria glasses, or "vendor to propose, subject to approval")
- Mounting: ___ (example: wrist camera 12 cm behind gripper tip, overhead camera 1.2 m above table centre)
- Calibration: ___ (example: intrinsics per device; extrinsics recorded every session for multi-camera setups)
- Hardware parity: ___ (example: wrist camera model and field of view match the deployment robot)
- Common mistake: naming a device but not a mounting position. A head camera that sits 3 cm lower on one participant changes how much of the hand is in frame. That variance rarely surfaces in a QA report, but your model trains on it.
4. Video and sensor parameters - resolution, fps, codec, and sync tolerance
What to specify: recording parameters for every stream, each stated as a minimum plus an acceptable range. On multi-sensor programs the most important number here is the synchronization tolerance. Drift you cannot see when watching the video will still corrupt action labels at training time.
- Resolution: ___ (example: 1920x1080 minimum; 3840x2160 for egocentric fine-hand tasks)
- Frame rate: ___ (example: 30 fps constant frame rate; 60 fps for fast manipulation)
- Codec and container: ___ (example: H.264 or H.265 in MP4, in-camera stabilization off)
- Bitrate floor: ___ (example: a stated minimum Mbps per stream, verified on every file)
- Auxiliary streams: ___ (example: IMU at 200 Hz, depth at 30 fps, force/torque where contact matters)
- Sync tolerance: ___ (example: all streams within 10 ms of a shared clock, measured and logged per session)
- Common mistake: accepting variable frame rate footage from phones or action cameras. It breaks timestamp alignment with the other sensors, often silently. Leaving stabilization on is a close second, since it crops and warps frames and invalidates calibration.
5. Environment and scene variation matrix - where diversity is engineered
What to specify: the dimensions the dataset must vary along, and a minimum count for each cell. Lin et al. (2024), studying data scaling in imitation learning, found that environment and object diversity matter far more than the raw number of demonstrations, with little gain once each environment passes a threshold. The matrix is how you plan that diversity in advance.
- Distinct environments: ___ (example: 40 homes, no more than 6 hours from any single home)
- Environment types: ___ (example: apartment, landed-house, and commercial kitchens at 50/30/20)
- Lighting classes: ___ (example: daylight, mixed, artificial-only, each at least 20% of hours)
- Object sets: ___ (example: 150 object instances across 30 categories, rotated every session)
- Clutter levels: ___ (example: 3 levels, each defined with reference photos in an appendix)
- Geography: ___ (example: Vietnam and Thailand, split by environment count rather than hours)
- Common mistake: defining variation in hours instead of distinct instances. 200 hours from 5 kitchens and 200 hours from 40 kitchens are very different datasets, and both satisfy "200 hours of kitchen footage".
6. Participant profile and quotas - who performs the tasks
What to specify: who is eligible and how many unique participants you need, plus caps that stop a few people from dominating the data. In manipulation and egocentric footage the model sees hand size and handedness directly. It also picks up skill level from how each task is done.
- Unique participants: ___ (example: at least 120, none contributing more than 3% of hours)
- Demographic quotas: ___ (example: age bands 18-30, 31-45, 46-65; gender split within 40-60%)
- Physical attributes: ___ (example: at least 10% left-handed; hand size recorded in metadata)
- Skill profile: ___ (example: trained pickers for warehouse tasks, general public for household tasks)
- Qualification: ___ (example: briefing plus 3 practice episodes reviewed before production recording)
- Common mistake: no per-participant cap. The most available people end up recording the most hours, and the model learns their habits. A cap costs little to enforce during collection. Once collection is over, the imbalance cannot be fixed.
7. Episode definition and metadata schema - what every clip carries
What to specify: what counts as one episode and the acceptable length range. List every metadata field, and give each one a type and a set of allowed values. RLDS and LeRobotDataset are both organized around episodes, steps and task descriptions. If the data will end up in either format, design the schema that way from the start so conversion is mechanical.
- Episode unit: ___ (example: one attempt at one task, from start condition to end condition)
- Length range: ___ (example: 20-180 seconds; anything outside flagged for review)
- Required fields: ___ (example: episode_id, pseudonymous participant_id, environment_id, task_id, object_ids, lighting_class, success_flag, device_id, calibration_id, capture_date)
- Language annotation: ___ (example: one short instruction per episode, written before recording)
- Controlled vocabularies: ___ (example: task and object categories drawn from a fixed list supplied with the spec)
- Common mistake: free-text metadata. "Kitchen", "kitchen 2", and "Lan's kitchen" become three environments in your split logic. Any field you will filter or stratify on needs a controlled vocabulary.
8. Consent, privacy, and data residency - what the rights must cover
What to specify: consent scope and how bystanders and personal information are handled. State where data is stored and processed, and how long it is retained. Consent has to cover the real use: commercial model training including derivative models, plus transfer to your jurisdiction. Ego4D, for example, was collected from consenting participants, with de-identification where relevant.
- Consent scope: ___ (example: commercial AI training and evaluation, including derivative models)
- Bystanders and PII: ___ (example: no identifiable bystanders; screens, documents, and mirror reflections blurred or excluded)
- Applicable law: ___ (example: data protection law in each capture country, plus GDPR where EU data subjects are recorded)
- Data residency: ___ (example: raw footage stays in-region until de-identified, then transfers to buyer storage)
- Retention: ___ (example: vendor deletes raw copies 90 days after acceptance; consent records kept for the contract term)
- Common mistake: a consent form that says "research purposes" for a commercial program. Fixing consent after collection means re-contacting every participant, and some will not be reachable.
9. QA acceptance criteria and rejection rules - how "usable" is decided
What to specify: the checks every episode must pass and how much of the footage a human reviews. Then set a batch acceptance threshold and say what happens to rejected footage, including who pays for re-collection. Written down like this, quality becomes a contract term instead of an opinion.
- Automated checks on every file: ___ (example: resolution, frame rate, duration range, file integrity, metadata completeness, sync within tolerance)
- Human review sample: ___ (example: 100% during the pilot; 20% random plus all flagged episodes in production)
- Rejection rules: ___ (example: hands out of frame for more than 10% of the episode, end condition not met, blur on key frames, wrong object set)
- Batch threshold: ___ (example: accepted if the sampled rejection rate is under 5%; otherwise the full batch is re-reviewed)
- Re-collection: ___ (example: rejected episodes replaced at vendor cost in the next delivery cycle)
- Common mistake: letting the vendor's internal QA stand in for the acceptance test. You, or a third party, should be able to re-run your acceptance test on delivered data using the same written rules.
10. Delivery format, file structure, and cadence - how the data arrives
What to specify: output format and directory layout, the naming convention, and how and when files are transferred. Before bulk collection starts, ask for 5-10 episodes in the final format. A format problem found in week one costs hours. Found in week ten, it costs a re-export of everything delivered so far.
With the final delivery, ask for dataset documentation structured along the lines of Datasheets for Datasets or Google's Data Cards. The people who built the dataset will move on, and the documentation is what keeps it auditable after they do.
- Format: ___ (example: LeRobotDataset v3, RLDS, HDF5, ROS 2 bags, or MP4 with JSON sidecars)
- Directory layout: ___ (example: /program/batch/environment_id/episode_id/ with one folder per camera)
- Naming: ___ (example: task_environment_participant_yyyymmdd_seq.mp4)
- Transfer: ___ (example: encrypted transfer to a buyer-owned cloud bucket in a named region)
- Cadence: ___ (example: weekly batches after pilot sign-off, each with a QA report and coverage report)
- Common mistake: agreeing on a format but not a manifest. Without a checksum manifest and a per-batch coverage report, you cannot tell whether a delivery is complete, or whether coverage is on track, until the program ends.
Final review: vague spec lines vs production-ready spec lines
Before sending the spec, go through it line by line and ask: could two vendors read this differently and both be right? If yes, the line needs a number, a definition, or a reference image.
How a vague data collection specification line compares with a production-ready one
| Spec block | Vague line | Production-ready line |
|---|---|---|
| Objective | Data to train a manipulation model | Fine-tuning set for a bimanual policy, judged on 50 real-robot trials in 3 unseen rooms |
| Tasks | Everyday kitchen activities | 8 named tasks with start and end conditions, none above 20% of hours |
| Video | HD video | 1920x1080 minimum, 30 fps constant frame rate, H.265, stabilization off |
| Sync | Sensors synchronized | All streams within 10 ms of a shared clock, logged per session |
| Environments | Diverse environments | 40 distinct kitchens, max 6 hours each, 3 lighting classes at 20% minimum |
| Participants | Diverse participants | 120+ unique participants, max 3% of hours each, 10%+ left-handed |
| QA | High-quality footage | Written rejection rules, batch accepted under 5% sampled rejection, re-collection at vendor cost |
| Delivery | Standard format | LeRobotDataset v3 with checksum manifest, weekly batches with coverage report |
- Every numeric parameter has a minimum and an acceptable range
- Each task has written start and end conditions
- Metadata fields used for splits have controlled vocabularies
- Consent scope matches commercial use, and legal has reviewed the form
- Rejection rules are written so a third party could apply them
- A 5-10 episode format sample is scheduled before bulk collection
- The spec says which blocks may change after the pilot and which are frozen
Send DataX Power your draft data collection specification, however rough. We usually come back within 5 business days with a spec review and a pilot proposal covering capture mode, participant plan, QA rules, and delivery format.
Request a spec review- What is a data collection specification?
- A data collection specification is the document a buyer gives a vendor to define what a dataset must contain. It covers the model objective, task taxonomy, camera viewpoints, recording parameters, environment and participant variation, metadata schema, consent scope, QA acceptance rules, and delivery format. Vendors quote against it and write their capture protocol from it. The buyer later uses it to accept or reject each delivery.
- How is a data collection specification different from a capture protocol?
- The specification states what the dataset must contain and how it will be judged, and the buyer writes and owns it. The capture protocol is the vendor's operational plan for meeting it: scenario scripts, rig setup, participant briefing, in-session checks. The vendor writes that document, and the buyer approves it before recording starts.
- How long should a video data collection specification be?
- Most production-ready specifications run roughly 6-15 pages, including appendices such as reference photos, object lists and controlled vocabularies. Precision matters more than length. A one-page brief is fine for a first conversation, but you cannot price against it or use it for acceptance testing.
- Should the specification change after the pilot?
- Yes. A pilot tests the specification as well as the vendor. Expect pilot data and a quick baseline model to show that some episode length ranges, rejection rules or variation quotas need adjusting. Freeze the specification before production. From then on, handle edits as formal change requests so data collected under different rules can be tracked separately.
- Can I send the same data collection plan template to several vendors?
- Yes, and you should. With a shared template every proposal is priced against the same scope, so you can compare them line by line, QA commitments included. Ask each vendor to flag lines it considers ambiguous or infeasible. How they answer is a useful signal of operational experience.
