Video Data Quality Assurance: 12 Checks Before a Training Dataset Ships

Video data quality assurance is the set of checks that decides whether collected footage is fit to become training data. Annotation QA comes later and asks whether the labels are right. Raw-footage QA asks the earlier question of whether the video can support any label at all. Missing frames or cameras that disagree on time can rule that out, and so can a participant who went off script. Footage of someone who never consented cannot be used, however good it looks.

The cost curve is steep. A dropped-frame problem caught on capture day costs a rig fix and one re-shot session. If it surfaces after annotation, the labelling spend is wasted and you pay again for recollection and relabelling. Found after training, it also burns a model run and weeks of diagnosis. The 12 checks below are the ones that matter most in robot learning and embodied AI programs, multi-camera setups included. Most can be scripted, though a few need trained reviewers.

Each check in a data collection QA checklist should answer four questions. What defect does it catch? How is it measured, and by whom? What number separates pass from fail? What happens to footage that fails? Without a written threshold, pass or fail gets settled by argument at delivery, and without an agreed outcome the next argument is about who pays for the re-shoot.

Failing footage goes one of four ways. Repair means fixing it without new capture, for example by re-muxing a container or rebuilding metadata from logs. A contained defect, like a short blur span, gets a frame-level tag so training can mask it. Episodes that cannot support the intended use are recollected. Anything with an open consent or integrity question is quarantined and kept out of every delivery. The thresholds below are typical starting points rather than standards. Set yours in the pilot and adjust them before production volume makes changes expensive.

Check 1: Resolution, frame rate and codec conformance. The usual failure is footage recorded at the wrong settings: 1080p instead of 4K, 29.97 fps instead of 30, the wrong codec or bit depth, or variable frame rate from a phone camera adapting to low light. ffprobe pulls the stream parameters (resolution, codec, pixel format, frame rate) from every file, and a script compares them to the spec manifest. Pass means an exact match, plus a bitrate floor so compression does not erase fine detail. Container problems can be repaired by re-muxing. Wrong capture settings cannot. Upscaling or frame interpolation would invent pixels for the model to learn from, so those episodes are recollected.

Check 2: Dropped and duplicated frames. Frames go missing when storage cannot keep up or a device throttles under heat. USB cameras sharing bandwidth do it too. Some encoders then repeat frames to hold a constant rate, which hides the gap from a simple frame count. So the check works from timestamps and flags any interval longer than about 1.5 times the nominal frame period. It also compares the decoded frame count (ffprobe -count_frames) with duration times frame rate, while FFmpeg filters such as freezedetect and mpdecimate find frozen or repeated content. A typical limit is under 0.1% dropped frames per episode, with no gap over two or three frames inside the task window. Gaps outside the task window are trimmed. A gap during a grasp means recollection, and a device that keeps dropping frames comes out of the field.

Check 3: Multi-camera and sensor time sync. Two things go wrong here: fixed offsets, whether between cameras or between video and the IMU or force/torque streams, and clock drift over long sessions. A force spike logged frames away from the visible contact teaches a policy the wrong cause and effect. Hardware triggering or PTP is the first defence. QA then verifies it by recording a visible sync event, such as an LED flash, at the start and end of each session and measuring the offset at both. Half a frame period is a common limit (roughly 16 ms at 30 fps, 8 ms at 60 fps), with tighter values for contact-rich work. If the offset is constant and measured, it can be corrected and documented. Drift of unknown shape cannot, and the session either drops out of multi-view use or is recollected.

Check 4: Exposure, motion blur and focus. Fast hands under a slow shutter, a blown-out window, autofocus hunting or a smudged lens can each ruin the frames at the critical moment. Focus and blur are scored per frame using the variance of the Laplacian, a focus measure described by Pech-Pacheco and colleagues that drops as edges soften. For exposure, the score is the share of pixels clipped at black or white. Because Laplacian values depend on resolution and scene texture, the floor is calibrated per rig on reference clips. One typical rule flags spans with more than 5% clipped pixels for over a second, or a blur score under the floor on more than 10% of task frames. Short spans get tagged. If the critical moment itself is unreadable, the episode is recollected.

Check 5: Calibration drift in intrinsics and extrinsics. Intrinsics go stale when a lens or focus setting changes. Extrinsics shift when something moves: a bumped tripod, a loose wrist camera, a head-mounted rig that slips. The standard approach builds on Zhang's planar-target method. A checkerboard or ChArUco board is captured at the start and end of each session, and reprojection error is computed for both. Well-calibrated cameras typically land under about 0.5 pixel RMS. The change between the two readings tells you more. If the camera has moved beyond tolerance (usually a few millimetres and a fraction of a degree for 3D work), the session cannot be trusted for triangulation, though it may still be usable as single-view 2D data.

Stereo rigs need one more check on top of Check 5. Depth comes from disparity, and the baseline and relative rotation between the two cameras are what convert one into the other. QA therefore also measures rectification quality, usually as the vertical offset between matched features in the rectified pair, and flags sessions where it creeps above roughly one pixel. A head-mounted stereo camera can look fine in each view and still have shifted relative to its twin. The depth it then produces is quietly wrong.

Check 6: Episode completeness and boundaries. A flat battery or a full card cuts episodes short, and streams go missing. Boundaries go wrong too: an episode starts mid-reach or stops before the task finishes, or two attempts end up merged in one file. Scripts confirm that every required stream is present for the full episode and that duration falls inside the expected band. A quick human pass then checks that each episode opens with hands at rest and closes after completion, with an idle buffer of one to three seconds. Merged episodes are re-segmented; truncated ones are recollected.

Check 7: Task and protocol adherence. Scripts handle this one worst, because the defects are behavioural: the wrong object or hand, a skipped step, an improvised grasp, an egocentric wearer looking away from the workspace, a failed attempt logged as a success. Trained reviewers work from a protocol checklist, with object detection confirming the right items are in frame. They typically review 100% of pilot episodes and each new participant's first sessions, then drop to a 10-20% stratified sample once adherence is stable. 95% adherence per batch is a common acceptance level, and participants below it are retrained. Off-protocol episodes are recollected. Genuine failed attempts are often kept and labelled when the program wants recovery data.

Check 8: Scene and participant coverage against quota. A dataset can pass every per-episode check and still be wrong in aggregate: too many bright kitchens, too few left-handed participants, or three prolific people contributing a third of all hours. Coverage is computed from metadata against the quota matrix in the spec. Typical tolerances hold each cell within about 10% of target and cap any one participant at a small share of hours, often 2-5%. Footage over quota is held back from delivery, and under-filled cells get targeted sessions in the next schedule.

Check 9: Metadata completeness and schema validity. Typical failures are missing participant IDs, blank environment tags, camera serials that do not match the calibration record, wrong time zones, and free text in fields that should be fixed lists. This check is fully automated. Every sidecar file is validated against a JSON Schema and checked for referential integrity, so participant IDs resolve to the consent registry and device serials to the calibration registry. Nothing ships below 100% validity. Most failures can be repaired from session logs. Episodes whose metadata cannot be rebuilt are quarantined, because untraceable footage cannot be defended in an audit. For the fields buyers will ask about later, Datasheets for Datasets is a useful reference.

Check 10: PII and consent verification. There are two kinds of problem. One is identifiable content in frame: bystanders, faces in mirrors or window reflections, phone and laptop screens, documents, addresses, vehicle plates. The other is footage from participants whose consent is missing, expired, out of scope or withdrawn. Face, text and plate detectors flag candidate frames, but a human reviewer makes the call, since detectors miss partial faces and reflections. Every participant ID is matched to a valid consent record. Delivered footage must contain zero unconsented identifiable people. Where blurring a flagged region leaves the task region intact, it is blurred; otherwise the segment or episode is cut. A withdrawal triggers deletion across every copy, backups included, with a logged confirmation.

Check 11: Duplicate and near-duplicate episodes. Exact duplicates come from double uploads and re-exports. Near-duplicates are harder to spot: a participant repeats an identical setup dozens of times or, in crowdsourced programs, resubmits footage with small edits. Either way the hours go up and the information does not. Checksums catch exact copies. For near-duplicates, perceptual hashes such as pHash or dHash are computed on frames sampled across each episode and compared by Hamming distance, with embedding similarity added for scene-level repetition. A common review trigger is a distance under roughly 8-10 bits out of 64 on most sampled frames. One copy is kept. A participant caught resubmitting deliberately is removed.

Check 12: File integrity and delivery structure. The failures here are mundane: transfer corruption, partial uploads, wrong directory trees, naming errors, missing sidecars. A SHA-256 manifest is generated at ingest and verified after every transfer, including by the buyer on receipt. A checksum only proves the file matches what was ingested. A full decode test, for example ffmpeg -v error -i input -f null -, surfaces the corrupt packets a checksum cannot. A script validates folder structure and file names against the spec. The bar is a 100% checksum match and zero decode errors. Failures are re-transferred from the source copy, or recollected if the source itself is corrupt.

The same 12 checks in one table. The thresholds are defaults to test in your pilot, and none of them is an industry standard.

About eight of the 12 checks run almost entirely as scripts: format, frame integrity, sync, calibration, completeness, coverage, metadata and file integrity. Run them on 100% of footage at ingest, ideally within 24 hours of capture, so a field team hears about a failing memory card or a loose wrist mount before the next session. At that point the participant is still available and the rig and room have not changed.

People are needed for protocol adherence and PII decisions, and to review whatever the scripts flag. A reviewer has to know what a good episode looks like for this specific task. Generic video reviewers do not, and they miss problems like a grasp that succeeds for the wrong reason.

Agree the rework budget before collection starts. When scenario types or participants are new, plan on 10-15% of sessions needing recollection. Once a program is established, the figure falls to single digits.

Questions data and robotics leads ask when setting QA terms for a video data collection program: