Simulation is cheap, scalable and systematically wrong in the ways that matter. Lighting that never flickers, objects that never deform, floors that are never wet: the long tail of physical reality is precisely what simulators cannot enumerate, and precisely where robots fail.
Robotics teams know this. That is why the field has moved from pure simulation toward foundation models trained on large volumes of real-world data, and why the bottleneck has shifted from algorithms to data collection. The question is no longer whether real data is needed but how to collect it at scale in a form a model can actually learn from.
01Where simulation breaks
Simulators are excellent at the parts of the world that are easy to describe with equations: rigid bodies, gravity, collisions between simple shapes. They are poor at everything else, and everything else is most of a real environment:
- Deformable and articulated objects. Cloth, cables, food, bags, drawers that stick and lids that need a twist.
- Visual mess. Clutter, reflections, motion blur, glare from a window at 4pm, a camera lens with a smudge on it.
- Human environments. Kitchens arranged by people, not by a scene generator, with objects in places no one would think to randomise.
- Contact. The friction, slip and compliance that decide whether a grasp holds.
Domain randomisation helps, but it only randomises the variables someone thought of. The long tail is, by definition, the set of things nobody thought of. The only reliable way to cover it is to record it.
02What real-world collection captures
We collect real-world sensor data at scale, with more than 200,000 hours of first-person video and more than 15 million sensor and visual assets processed across physical AI projects. A typical session combines several synchronised streams:
Head-mounted video
First-person footage of people performing manipulation and workflow tasks in real environments. The egocentric viewpoint matters: it is the closest human analogue to a robot's wrist or head camera, and it captures where attention goes during a task, not just what happens.
Audio and motion telemetry
Ambient and spoken audio aligned to every frame, plus accelerometer and gyroscope streams at 30 Hz. Audio carries information video misses, like the click of a latch or the change in pitch as a container fills. IMU data gives a direct signal of motion and effort.
Multi-camera rigs
Homes, kitchens and industrial settings captured from several viewpoints at once, so the same action can be learned from the actor's perspective and from the outside.
03Designing for training utility
Volume alone does not make a useful dataset. Ten thousand hours of the same person making the same coffee in the same kitchen teaches a model about that kitchen. We design collection around what a model needs to generalise:
- Task diversity. Many different tasks, and many variations of each, rather than repeated demonstrations of a few.
- Viewpoint and environment diversity. Different rooms, layouts, lighting and object sets.
- Participant diversity. Different hands, heights, habits and ways of doing the same task.
- Failure cases, on purpose. Grasps that slip, objects that tumble, a lid that does not come off the first time.
A dataset of only successful grasps teaches a model less than one that shows where grip slips and objects tumble.
Failures are the part most collection programmes edit out, because they look like bad data. They are the opposite. Recovery from a failed attempt is one of the hardest things for a robot to learn and one of the most important for deployment.
04Labels a robot can use
Raw footage becomes training data when it is labelled at the level a policy needs. We annotate verb-noun actions, object identities and hand-object interactions with timing, so a model can learn not just that someone placed a cup on a tray, but when the grasp began, which hand was used and when contact ended.
Labels are delivered in formats that fit existing pipelines, including COCO and JSONL, or a custom schema, and the label taxonomy is agreed during scoping so it matches the action space of the target robot.
05Rights in the physical world
Recording real people in real places raises questions that synthetic data avoids. Participants consent to collection and use, rights are documented per session, and faces, screens and bystanders are blurred before delivery. Industrial sites often have their own confidentiality requirements, which are agreed during scoping.
Research datasets raise a separate question. Large academic egocentric collections are usually released under licence agreements written for research, so a team shipping a commercial robot has to check whether its use is covered before training on them.
This matters for the same reason it matters in every other domain: a robotics foundation model trained on footage of unknown origin carries that uncertainty into every commercial deployment.
06Simulation and reality together
None of this argues for abandoning simulation. The strongest physical AI programmes use both: simulation for scale and safe exploration, real data to anchor the model in the world it will actually operate in, and real-world benchmarks to measure the gap honestly.
Human video and robot data play different roles too. Teleoperated demonstrations record the robot's own actions, so they train a policy directly, but every hour needs the robot, an operator and a setup. First-person video of people is far cheaper to collect at scale and covers many more tasks and environments. Models learn what to do and how objects behave from it, then use a smaller amount of robot data to map that onto the robot's own body.
When real collection is designed around the robot, with tasks, environments, participants and sensor rigs specified to match, it becomes the most efficient way to close that gap. Our egocentric video vertical covers the catalogue and custom collection.
