10,000-Hour Ego-Exo Data Collection Case Study

Recording a single action with five devices requires temporal alignment across views and precise control over the recording duration for each participant and scene. But what if the required volume of data for delivery reaches 10,000 hours? How is this done?

Recently, Nexdata successfully completed a 10,000-hour collection of Ego-Exo multi-view human manipulation data for a leading AI company. This project involved capturing data from factories, warehouses, and other types of locations, combining egocentric, wrist-mounted, and exocentric views to capture human manipulation data for robot learning.

Five Devices Capture Hand–Object Interactions and Full-Body Movements

Each setup comprised one stereo head-mounted device, two wrist-mounted cameras, and two exocentric cameras. As was explicitly required by the project, all five devices had to remain time-synchronized throughout collection so that footage from different views captured the same action at the same moment.

The head-mounted device was used to record the workspace in front of the participant, while the two wrist-mounted cameras recorded grasping actions performed by the left and right hands, respectively. While adjusting the wrist-camera positions, the team needed to make sure that both the hand and the object were visible and the hand–object contact point was not occluded either by the back of the hand or by a sleeve. Two exocentric cameras were set up so that they would cover complementary viewpoints depending on the workstation layout and the participant’s range of motion.

Broader framing allowed the team to capture full-body movements such as bending down, standing up, and turning. The near-frontal viewpoint helped to minimize occlusion of hand–object interactions by the participant’s body. Based on the time-synchronized multi-view footage, the researchers could analyze hand–object contact details in relation to body posture and object motion.

Different Work Scenarios Preserve Variations in Actual Task Execution

The same object retrieval task can be executed in different ways. Picking up a part from a factory workbench may involve reaching around a fixture and grasping it from the side. Different shelf heights and depths in warehouses can affect the reaching distance and body posture needed to retrieve an item. For example, picking an item from a high shelf may require raising an arm, while picking one from a low shelf may require squatting. This kind of variation caused by real-world factors was precisely what the client wished to preserve.

To ensure scene diversity, the collection was conducted across multiple factory and warehouse workflows, with no more than eight hours of cumulative recording per scene.

In factories, the collection included picking and placing parts, assembling components, and transferring finished products along production lines; sorting parts, organizing materials, and packaging at workbenches; retrieving, returning, and storing tools at tool cabinets; and transporting materials, moving carts, and stacking totes between material preparation and temporary storage areas.

In warehouses, the collection included unpacking, counting, and sorting goods in receiving areas; putaway, picking, replenishment, and removal of goods in shelving areas; boxing, sealing, and packing in packing areas; and shipment consolidation, pallet stacking, and loading/unloading in shipping areas.

Tasks similar to those mentioned above could also be performed at shelves of different heights and depths, at workbenches with different layouts, and in aisles of different widths, thus adding to the diversity of environments where the tasks were performed.

Multiple Rounds of Quality Control Ensure Multi-View Data Quality

Prior to the project start, Nexdata and the client jointly defined the data collection and acceptance criteria. The Nexdata team conducted multiple rounds of review, using both AI-assisted quality checks and manual review.

Scene validity. First of all, the project managers checked the proposed workstations against the existing collection records, considering the workstation layout, object placement, and operating conditions to establish whether a scene was a duplicate. For already recorded scenes, they checked the cumulative recording duration before scheduling additional collection within the remaining time limit.
Task authenticity and action validity. The reviewers checked if the collection environments and the tasks being performed were consistent with actual working practices, screening for staged behavior and actions without a clear task objective. Movements like turning, bending, or squatting had to occur naturally during task execution.
Multi-view video quality. The reviewers checked if the head-mounted footage fully showed both hands of the participant during task execution, if the wrist-mounted footage clearly showed hand–object contact points, and if the exocentric footage captured full-body movements and hand–object interactions. In addition to checking temporal synchronization across camera views, the reviewers checked for occlusion, subjects moving out of frame, loss of focus, motion blur, and exposure issues.
Task completeness and annotation accuracy. The reviewers made sure that each video clip contained the complete task sequence and its final outcome. Moreover, they checked if the objects, actions, and outcomes described in the verbal narration and annotations corresponded to the footage. Finally, before delivery, the team checked that video files from all camera views were present and correctly labeled, ensuring a complete set of corresponding recordings for each session.
Nexdata successfully delivered 10,000 hours of Ego-Exo multi-view human manipulation data with a data quality pass rate of up to 98%.