The enormous advance in AI capabilities over the last several years has in large part been driven by the availability of vast amounts of training data. Gwern’s influential 2020 essay “The Scaling Hypothesis,” written after the release of GPT-3, argues that recent advances in AI have been the result of scale: bigger models, bigger datasets, and more compute spent training them. Gwern writes:
The scaling hypothesis regards the blessings of scale as the secret of AGI: intelligence is ‘just’ simple neural units & learning algorithms applied to diverse experiences at a (currently) unreachable scale. As increasing computational resources permit running such algorithms at the necessary scale, the neural networks will get ever more intelligent.
Likewise, pseudonymous AI researcher Nostalgebraist wrote in 2024 that “the entire current landscape dates back to the realization that ML models improve steadily and predictably as you train them on more data. And that everything since then consists of training bigger models on larger datasets, enabled by years of rapid improvement in hardware.” Epoch AI found that since the early 2010s, the size of training datasets for AI language models has doubled about every six months. The biggest AI models today are trained on tens of trillions of tokens.
As more and more effort gets devoted to applying modern AI to robotics, an important question is “where will the training data come from?” Language models could be trained on huge existing corpuses of text. GPT-3, for instance, was trained on about 300 billion tokens’ worth of data scraped from the internet. In 2024 Epoch AI estimated the total existing stock of quality, accessible text data is around 300 trillion tokens. But there’s no equivalent stock of good, public robot training data. Figuring out where the data for robotics will come from and how much of it we’ll need has become a key problem for the field.
What is robot training data?
Training an AI model, whether it’s an LLM or a model used for controlling robots, fundamentally requires lots and lots of examples of something being done correctly (or enough knowledge of what a correct action looks like that you can score an attempt). You feed a model some set of inputs, you compare the model’s output to the actual, correct output, and you use the difference between the two to modify the parameters of the model to reduce this difference via some flavor of gradient descent.
With text, a training example can be just an existing chunk of text; you take the chunk, feed the model part of it, and have the model predict what comes next in the sequence. One chunk of text can thus generate many training examples by feeding the model different portions of the text.
To train a robot, an example is typically some sort of recording of a task being accomplished or attempted. There are a variety of different methods used to create these recordings and turn them into training data.
Historically, one of the most common methods for creating these recordings is called “teleoperation.” With teleoperation, a person manipulates some controls that are directly connected to a robot, and uses them to move a robot through some particular task (folding a shirt, picking up and moving objects, and so on).
As the person performs the task, data of the robot’s state is recorded, along with images from cameras that are recording the task (often mounted to the robot itself), which are combined to create a training example. An example might consist of feeding a model some data on the robot’s state and what the cameras can see at some particular point in the sequence. The model will try to predict what the next robot actions are (in the form of something like joint angles), and then compare that to what the actual actions were. This sort of training strategy, using examples generated by an expert (a human) and then training a model to mimic them, is known as behavioral cloning, and it’s part of the broader category of “imitation learning.”
The major benefit of this method is that the data it generates very closely matches the robot’s actual movement capabilities, because it’s produced from the actual robot you want to train, and it remains widely used today.
But teleoperation has drawbacks. The biggest one is probably expense: to get the data, you need to have an actual robot, along with a person to sit there and manipulate it, which makes it hard to scale this method to produce very large amounts of data. Another drawback is that it’s somewhat clunky for a person to control a robot this way; they often can’t get any sort of tactile feedback from the object being manipulated to guide their motions, so this won’t necessarily produce really good data on dexterous manipulation.
It’s also possible to generate training data using an existing robot by having it move around on its own, recording its actions, and using some sort of system to categorize these behavioral sequences as successful or not. This sort of strategy, where the robot moves autonomously and you score its actions in some way, is known as “reinforcement learning.”
This sort of arm farm doesn’t seem to be widely used today, but using real-world deployed robots will probably eventually be the largest source of robot training data. You often hear robotics folks talk about the need for a “data flywheel,” where you have robots deployed in the real world, you use the data from them to train even better models, resulting in more robot deployments, resulting in better models, and so on.
Another very common strategy for generating robotic training data is simulation: instead of running a physical robot in the real world, run a digital one in a simulated world instead. This is also commonly done with reinforcement learning, in which you have the robot try to navigate some simulated space, after which you score it on how well it does. Another flavor of this is the teacher/student paradigm where you have a “teacher” model that has full access to all the data within the simulated environment, and a “student” model that can only see what the real robot would see (images from its cameras and its own internal state), and you try to get the student to mimic the teacher.
The benefits of simulation are that it’s much cheaper and faster to simulate a robot than it is to deploy one in the real world, and that you can get very large amounts of data by simulating the behavior of thousands of robots.
But simulation also has drawbacks. The biggest one is that the quality of the data it produces is a function of the accuracy of the simulation. For some things, such as locomotion (having a robot move through some physical space), simulation is fairly accurate, because the behavior is mostly well-characterized enough that we can represent it digitally. But for things such as object manipulation (particularly for soft or deformable objects), it’s much harder to build a simulation that accurately captures real-world behavior, and using it as a source of training data is much more difficult. This is generally referred to as the “sim-to-real gap.”
Because of simulation inaccuracies, one popular simulation technique is called “domain randomization.” Since you don’t know what various parameters like friction will be in the real world, you decide to vary the parameters randomly when running the simulation, in the hopes of creating a training envelope that encompasses the parameters that will be encountered in the real world.
Another way to use simulation is to create synthetic training data: taking real training data as an input and using it to create lots of similar training examples.
We can also create robot training data by recording humans doing tasks and creating some sort of mapping from the recorded task data to a robot. This might be as simple as fitting a person with a head-mounted camera (creating what’s known as “egocentric video”), and using it to record the person doing various tasks with their hands. The video is then processed so the person’s motions are interpreted as robot motions, and used to create training examples.
You can also add more sensors to collect more elaborate human task data. Another flavor of this strategy is to use what’s called a Universal Manipulation Interface, or UMI. With UMI, a person holds a pair of specially designed manipulators, that typically have a simple pincer grip at the end. The person does various tasks with the manipulators, which have cameras mounted to them that record exactly what the manipulators are doing. This video is then used to create training examples for robots that have similar manipulators.
The benefit of using human-collected task data is that it’s much cheaper to collect than teleoperation data: all you need is a camera, or some camera-equipped manipulators, rather than an entire robot. And it’s probably much easier to get a wide variety of different task recordings by just having a person walk around and do different things, rather than constantly repositioning a robot to do them. And because the data is gathered in the real world, by people manipulating real objects, it’s more accurate than simulation.
It’s possible to use multiple training strategies in a single robot model. It’s also often useful to train a robot model using data other than task completion data. Vision-Language-Action models are built from Vision-Language models trained on text and images. And because of the ubiquity of general video data on things like YouTube or TikTok, there’s a great deal of interest in finding ways to use generic web videos as a source of training data.
Robot dataset sizes
How much of this action-relevant training data have we gathered so far, and how much do we need?
As with AI models more generally, robotic training datasets seem to be getting bigger. It’s not amazingly easy to compare these datasets to the text datasets used to train LLMs. For one, different robotic training sets have different levels of quality. And video data for robotics training is informationally different from text data: for one, video data tends to be highly correlated — on short timescales, one frame is pretty similar to another frame. Thus even though video recordings contain a lot of data, they’re probably less informationally dense for training purposes than text data.
But we can still try. One very rough way to compare dataset sizes is to estimate how long a dataset of text tokens would take for a person to read, and compare that to the hours of robot training data we have. Under this metric, even several-year-old AI models were trained on something like 1,000 times the data that current robot models are trained on. Goldberg calls this the “100,000-year data gap.”
If you assume that we need about 1,000 times as much real-world robot training data as we actually have — on the order of 1 billion hours of data — how much would that cost to generate? A naive estimate suggests that getting enough data to get robot models to a similar level of capability as recent AI models might cost on the order of $10 billion to $100 billion.
How much data we actually need, however, seems to be something of an open question. If we don’t care about generalization, and only want to train robot models to perform various specific tasks, the data requirements might be substantially less. Alternatively, it might turn out that frontier general AI models are extremely capable robot controllers after being fed a comparatively small amount of robotic training data. Likewise, we might find robotic AI model architectures that are much more sample efficient — learning much more from each training example — than current models.
On the other hand, it could be the case that moving around and manipulating objects in the real world is a fundamentally more difficult task than manipulating text on a computer, and that models require a lot more training to become competent at this real-world behavior.
Conclusion
LLMs showed the benefits of scale when training AI models — bigger models, more data, trained using much more computation. We’re now in the process of seeing how well this scaling works for robotics, and robotics researchers are working hard to gather enormous amounts of robot task training data in the hopes of seeing the same sort of trends with robotic capabilities that we have with AI more generally.
