Action Segmentation Annotation for Robot Policy Training: Technical Guide (2026)

A robot demonstration - a video of a human or robot completing a task - is a continuous stream of motion. For imitation learning models to learn effectively from this stream, it helps to know where one atomic action ends and the next begins. The reach phase, the grasp phase, the lift phase, the transport phase, and the place phase of a pick-and-place task are distinct action primitives that the model should learn as separable units, not as one continuous blob of motion.

Action segmentation annotation is the process of labeling a demonstration recording with temporal boundaries that indicate where each action primitive begins and ends, plus action class labels that describe what is happening in each segment. The annotation also includes contact state information (what the robot is holding and where contact is occurring), object state changes (before/after states that indicate task progress), and optionally language instruction strings that describe the overall task or individual sub-tasks in natural language.

Models that are trained on unsegmented demonstrations can still learn from action chunk-based approaches (ACT, Diffusion Policy) that handle temporal structure implicitly. But VLA models (pi0, OpenVLA, Octo) and hierarchical policy models benefit substantially from action segmentation annotation, because they learn both high-level task structure and low-level motion primitives. Without action segmentation, the high-level structure that the VLA uses to sequence sub-policies must be inferred entirely from visual state transitions, which is noisier and requires more data to learn reliably.

A consistent action taxonomy is the foundation of action segmentation annotation quality. Inconsistent taxonomies - where different annotators use different granularity levels or different action class definitions - produce datasets with high inter-annotator disagreement that degrades model training.

The taxonomy should be defined before annotation begins, reviewed by at least one robotics engineer who understands the target policy architecture, and documented with explicit examples of edge cases that annotators will encounter.

Temporal boundaries mark the frame where one action primitive ends and the next begins. The critical annotation decision is how to handle transition frames - frames where the robot is mid-motion between two atomic actions.

There are two approaches to handling transitions. The first is clean boundary annotation: define the boundary at the last frame of one action and the first frame of the next, treating transitions as belonging to the subsequent action. This approach is simpler to annotate consistently but underrepresents the transition dynamics that often determine policy robustness at deployment.

The second approach is transition state annotation: define an explicit transition segment between atomic actions, labeled as its own category. This approach captures transition dynamics explicitly and allows models to learn transition-specific policies, but requires annotators to make more judgment calls about where the transition begins and ends - increasing annotation time and inter-annotator variance.

For most VLA training programs, the clean boundary approach at frame precision (marking the exact frame rather than a timestamp rounded to the nearest second) is the recommended starting point. Transition state annotation is worth adding for programs specifically targeting robustness at action transition points.

Action class labels categorize what is happening in each segment. The granularity of the taxonomy must match the level at which the policy will be decomposed: too coarse (only "pick" and "place" for a complex assembly task) loses the structure the model needs, too fine (separate labels for every 10ms of wrist motion) produces a taxonomy that is impossible to annotate consistently.

A practical taxonomy for manipulation tasks has three to five levels of granularity: functional phase (reach, grasp, transport, place), contact state (pre-contact, contact initiation, stable contact, contact release), and optionally task phase (approach, manipulation, completion). For multi-step tasks, add a task step label that identifies which step within the overall task is occurring.

The taxonomy should include an explicit "other" or "transition" category for moments that do not fit cleanly into the primary action classes. Forcing every frame into one of the primary categories when the action is ambiguous produces systematic annotation errors that are more damaging to model training than having explicit uncertainty labels.

Verify taxonomy consistency across annotators before production annotation begins. Run a calibration exercise where 5-10 annotators label the same 10-minute demonstration clip independently, then compare disagreement rates per action class boundary. Taxonomy revisions that reduce inter-annotator disagreement are easier to make before production starts than during it.

Contact state annotation labels whether and where the robot's end-effector is in contact with an object at each timestep. For tasks where contact transitions are critical (grasping, insertion, assembly), contact state annotation provides the policy with explicit supervision on the contact-sensitive moments that raw action label boundaries cannot fully capture.

The minimum useful contact state annotation is binary: contact or no contact. More informative is a four-state taxonomy: pre-contact (approaching but not touching), contact initiation (first frame of contact), stable contact (sustained grasp or surface contact), and contact release (first frame of release). Each state should be annotated at frame precision for the grasp events that determine task success.

Object state annotation labels the state of the task object at key moments: initial state (where the object is and what orientation it has before the demonstration begins), post-grasp state (what the grasp looks like), and final state (where the object is after the demonstration completes). Object state annotation enables models to learn goal-conditioned policies that evaluate whether the task has been completed correctly.

VLA models require language instruction strings associated with each episode. The simplest approach is a single instruction per episode at the task level ("pick up the red cube and place it on the blue plate"). More informative is hierarchical instruction annotation with sub-task instructions aligned to action segments ("reach for the red cube" / "grasp the red cube" / "transport to the blue plate" / "place on the blue plate").

Language instruction strings should be written by annotators who have reviewed the specific demonstration being annotated, not by annotators writing generic instructions based on the task specification. Instructions that do not match the actual execution of the demonstration (e.g., "grasp with your right hand" for a demonstration where the left hand was used) introduce language-action misalignment that confuses policy conditioning.

Language instruction diversity within a dataset improves policy robustness. Programs that annotate every episode with the same instruction string ("pick and place") produce policies that condition on language poorly because there is no variation in language to condition on. Include at least 5-10 distinct phrasings for the same underlying task to provide the language variation the VLA policy needs to learn meaningful language conditioning.

For Chinese-language VLA programs, instruction annotation should be done by native Simplified Chinese speakers with robotics task domain knowledge - not translated from English annotations after the fact. Translation introduces vocabulary inconsistency and tonal ambiguity that degrades Chinese-language VLA performance.

Inter-annotator agreement (IAA) for action segmentation is measured differently from classification tasks. The standard metric is temporal agreement rate: the percentage of frames where two annotators agree on the action class label. A temporal agreement rate above 85% indicates acceptable taxonomy clarity and annotator training. Rates below 80% indicate taxonomy ambiguity or annotator calibration failure that must be resolved before production annotation.

For boundary annotation specifically, the standard measure is boundary tolerance agreement: two annotators are considered to agree on a boundary if they place it within ±N frames of each other, where N is typically 3-5 frames at 30fps. Strict frame-exact agreement on boundaries is unrealistic for human annotators and should not be used as the primary QA metric.

Production QA should include: a 10-15% overlap sample where multiple annotators label the same episodes, automated checks for annotation completeness (every frame labeled, boundaries are at frame boundaries and not between frames), and domain expert review of a 5% sample to catch systematic annotation errors that IAA metrics will not detect (e.g., consistently misidentifying the grasp initiation frame as the grasp completion frame).