Hand Gesture Dataset Collection for Hand Tracking, XR and Robotics (2026)

From the outside, a hand gesture dataset looks like one category: video of hands doing things. Three kinds of product team actually buy it, and each needs different data. XR headset and smart glasses teams want hand tracking training data, so a model can predict 3D joint positions from the device's own cameras, frame after frame. For automotive, kiosk and TV teams the job is gesture recognition data collection, where the model has to decide whether a motion was a command and, if so, which one. Robotics teams are after something else again: hand-object interaction video that shows how people grasp, reorient and release objects, for a policy or retargeting model to learn from.

Those differences drive every collection decision, starting with where the camera sits and how fast it records. Recruitment and scripting change too, and so does the definition of ground truth. Webcam gesture clips carry no 3D information. Lab-captured 3D hand poses rarely include the sensor or lighting found in a car at night. Public data helps with each use case up to a point. Past that point, custom collection becomes the cheaper option.

The public research record is strong and worth using for pretraining and benchmarking. HaGRID covers static gestures at scale: its HaGRIDv2-1M release contains 1,086,158 Full HD images across 33 gesture classes plus a no-gesture class, from 65,977 people, captured 0.5-4 metres from the camera. Jester has 148,092 short webcam clips across 27 gesture classes from 1,376 people. EgoHands covers first-person hands, with pixel-level masks on 4,800 frames from 48 Google Glass videos.

For 3D pose, the reference sets come from instrumented studios. FreiHAND has 32,560 unique training samples recorded against green screen, with 3D pose and shape fitted through a human-in-the-loop process. InterHand2.6M has about 2.6 million annotated frames of single and interacting hands from 26 subjects, captured by 80-140 cameras at 4096 x 2668. ARCTIC is built around bimanual manipulation: 2.1 million frames of 10 subjects handling 11 articulated objects, with 3D hand and object meshes, recorded from 8 static views, one egocentric view and 54 Vicon motion capture cameras. HOI4D is egocentric RGB-D, with 2.4 million frames of 9 participants handling 800 object instances in 610 rooms. HOT3D, from Meta, has 833 minutes of egocentric footage from Project Aria glasses and Quest 3, with 19 subjects and marker-based ground truth.

The gaps show up once you try to build a product on these datasets. The 3D sets have roughly 10-30 participants, which is too few for a consumer device that has to work across skin tones, hand sizes and ages. The large image and clip sets have plenty of participants but no 3D ground truth and little hand-object context. Sensors rarely match either: a model running on a headset's monochrome tracking cameras or a car's near-infrared cabin camera sees a different image than a studio RGB camera does. The gesture vocabulary is whatever the dataset authors chose, which is unlikely to be your product's. Licensing is the last catch. Several widely used hand datasets carry research-only or non-commercial licences, so production use needs a licence review first.

Headset and glasses hand tracking runs from cameras on the device, so training data has to come from the same vantage point. That means egocentric capture: a head-mounted rig, research glasses such as Project Aria, or the target device itself where its sensor streams can be recorded. Many headsets track hands with dedicated monochrome tracking cameras rather than the colour passthrough cameras. If that is the deployment sensor, collection should use it or a close optical match in field of view, lens distortion and infrared sensitivity.

Most of the content is continuous motion rather than posed gestures: pointing, pinching, grabbing virtual objects, typing on surfaces, one hand passing over the other. Smart glasses add micro-gestures, such as a thumb tapping the side of the index finger. These are small and fast, and motion blur wipes them out easily. 60 fps with short exposure is a practical floor. Hands leave a narrow field of view all the time, so a dataset of centred hands is not enough. Script hands entering and leaving the frame, and hands working right at its edge.

Ground truth usually comes from a synchronized third-person camera array around the participant, triangulated and projected into the egocentric view, or from marker-based motion capture as HOT3D used. In both setups the head-mounted rig and the external cameras must share a clock and a calibration.

Gesture recognition classifies a motion into one of a fixed set of commands. The dataset is built around a gesture vocabulary: static poses such as an open palm or thumbs-up, and dynamic gestures such as a swipe, a circular rotate or a push toward the screen. Each gesture needs many performances by many people, because the classifier has to learn the command rather than one performer's style.

The negative classes deserve as much attention as the gestures, which is why Jester includes "no gesture" and "doing other things" classes. In a car, a driver adjusting a mirror, scratching their face, talking with their hands or reaching for a drink must not trigger the infotainment system. These hard negatives should be scripted and collected deliberately, often at a volume comparable to the positive classes, because they set the false-activation rate that decides whether drivers leave the feature on.

In-cabin systems typically use a roof or console camera, often near-infrared with active illumination for night driving. The steering wheel is a constant occluder, and gestures come from both the driver and passenger positions. Collection also has to cover sunlight moving across the cabin, and vibration. Kiosks and TVs face the user from 0.5-3 metres and have to cope with people walking behind the participant. 30 fps is enough for most vocabularies, though 60 fps is worth specifying for fast swipes. Ground truth is usually clip labels with start and end times, plus 2D keypoints for skeleton-based models.

Robotics teams use human hand video in two ways: as pretraining data that shows a model what manipulation looks like, and as demonstrations retargeted onto a robot hand or gripper. Both need hand-object interaction video where you can see the object and the contact, and whether the grasp worked. A hand pose with no object context teaches little about grasping.

That shapes the rig. The egocentric view shows the task as the operator sees it, and a wrist-mounted camera gets the grasp close up, much like the view many robot policies get from their own wrist cameras. Recovering 3D hand and object pose takes a multi-view third-person array. Programs that need contact information add force/torque logging or instrumented objects synchronized to the video. ARCTIC and HOI4D are good examples of what the research-grade version looks like.

Take the task content from the robot's actual deployment rather than a generic list: the objects it will handle and the grasp types it needs. Include bimanual coordination and in-hand reorientation, and record failed grasps with their recoveries. Policies are often trained at 30 fps, but contact transitions are quick, and capturing at 60 fps leaves room to resample.

Typical starting specifications by use case. The exact values depend on the target sensor and model, so confirm them in a pilot.

Hand appearance varies more than most teams plan for, and models fail on whichever groups the dataset under-represents. Write the participant matrix as quotas, not intentions. Then check it against delivered metadata, because the recruitment list tells you who signed up, not whose footage arrived. Recruiting across several markets widens the matrix without multiplying studio time. The dimensions to plan for:

Most hand tracking models predict the common 21-keypoint hand skeleton, often alongside a parametric hand model such as MANO. There are four practical ways to produce that ground truth for collected video. They trade accuracy against cost, and some change how the hand looks on camera.

Multi-view triangulation detects 2D keypoints in each view of a synchronized, calibrated camera array and triangulates them into 3D. InterHand2.6M is the research extreme. Commercial programs usually run much smaller arrays of around 4-12 cameras and accept some loss of accuracy on heavily occluded frames in exchange for throughput. Nothing is attached to the hand, so its appearance stays clean.

Marker-based motion capture, which ARCTIC and HOT3D used, gives the most accurate joint and object trajectories under occlusion. The cost is a mocap volume and small markers on the hand. The markers show up in the footage and, unless the pipeline handles them, can teach a model to expect them.

Model fitting with human review, the approach behind FreiHAND, fits a parametric hand model to multi-view evidence and has annotators correct failures. It produces shape as well as pose. Manual 2D keypoint annotation is enough for many gesture recognition models and is the cheapest route when 3D is not required. Sensor gloves record joint angles directly and help with robot retargeting. They also change how the hand looks, which rules them out as visual ground truth.

When the product itself sees the hands from a head-mounted viewpoint, egocentric stereo is the lighter option. A calibrated stereo pair on the head gives metric depth on every frame, so 3D keypoints can be recovered from the wearer's own view without a full studio array. Accuracy falls with distance and under self-occlusion, which is why many programs pair a head-mounted stereo rig with a small third-person array for a subset of sessions.

Pick the method before collection starts, because camera count, calibration routine, sync hardware and studio layout all follow from it.

Public data is still the right place to start for pretraining and benchmarking, and for comparing architectures. Custom collection earns its cost when the product needs something public data cannot supply. The efficient first step is a paid pilot on the target sensor, with the participant matrix and ground truth method fixed in the specification. A pilot shows whether the capture protocol and QA thresholds hold up, and whether the delivery format works, while changes are still cheap.

Signs that a program needs it:

Questions XR, automotive and robotics teams ask when planning hand video collection: