What's special about data for training world models?

Hello everyone, my name is Julian Lopez, and I am the AI Operations Lead here at Reka.

Hi, and I'm Fedor, and I'm responsible for the data.

One of our main focus areas here at Reka is on building a World Language Action Model. That must take a lot of data. What do you need? What kind of data do you need in order to build this kind of model?

We work with, indeed, a very large amounts of data. And we work with very large amounts of multimodal data. So, we work with tons of petabytes of video data. We work with billions of images and huge amounts of text data as well.

Of course, we need to process all of the data to make sure it's suitable for model training. But that's basically the essence of what kind of data we use.

We don't really focus on a very specific style of, let's say, videos or images we get. We do try to get as much egocentric data as possible, but for the model to learn, we need all sorts of different scenarios.

So, the more diverse and rich our data set is, the better our world models gonna become.

Yeah, makes perfect sense. So, it, it seems to me like we're looking for volume and diversity.

Now, when you, when we look at data from a data intelligence perspective, what does that look like, and how does that impact data quality or other aspects of the data?

We have a sub team in a, in the research organization which is focused on data intelligence. And there's really three, three key topics the team is working on.

The first one is understanding what's in the data. So, let's say you have hundreds of millions of video clips. How do you know how many of them have humans? How do you know how many of them have humans running with dogs? How many of them are egocentric? How many of them are exocentric? Do you have a lot of night videos? Do you have a lot of videos with snow in them?

It's all of these questions need to be asked and answered before, before you train the model. Otherwise, you don't really know what you're training your model on, and so therefore you don't know what it knows and what it doesn't know.

So, this, this aspect is very important, very important in the charter of the team.

Another aspect which is as important is, is of course the quality of the data you feed, you feed into the world model. It depends thoroughly on how the data intelligence team executes on achieving that, that quality.

We have a lot of different quality parameters, quality characteristics that we check for every data point that, that we provide to training. And achieving that quality is, is another topic that the team is focusing on.

Finally, last but not least, one of the, the third topic is the topic of, of actually modelling some of the properties that your model want, that your world model needs to know in the data.

World models these days have to know and learn how to react to certain, to certain actionable commands. So, let's say you want the character in a, in your generated video to turn right and maybe pick up a cup. How do you structure the data? How do you annotate the data? How do you enrich your data in a way that your model can learn the right thing?

So, that is, that is the aspect that the, the third, that is the third aspect the data intelligence team is working on and is focusing on. Is trying to think about how to properly model these different, different signals such that, such that the model learns, learns what we wanted to learn.

Interesting. So, it's about modelling the data to understand, so the model can understand the right signals, about improving the data quality, and about understanding what data is going into the model.

Yeah. I'd say that, that's a good, that's a good summary.

Yeah. You've talked about a vast volume of data, which seems like processing all that seems so daunting. How do we do that here at Reka?

Successfully.

We did, we, of course, it's a, it's a huge challenge to process such volumes of data with a small, in a small startup environment. But we leverage a lot of the AI tools.

We have a huge separate, wonderful separate video about, about the, how data platform thinks about data processing, how it's built to process such volumes of data. So, I direct the viewer towards that video.

Excellent. We'll add a link to it somewhere.

You've spoken about different types of data that we need to look at throughout the training process. Now, as we look at pre training, we have multiple stages. Then we're looking at supervised fine tuning, sometimes we're training LoRAs.

How do you know what kind of data is most useful at different stages? Like, what, what are you looking for? What are you focused on? Is there a specific data quality resolution, or is there a specific focus in the different stages, in the latter stages, let's say, of training?

You know, the way we approach this is, is that because we're training models from scratch, we actually have to properly, properly think about what kind of data and with what quality comes at which stage of the training.

Early on in, early on for training, for training video generation models, we need a very large amounts of image data, and we need very large amounts of video data.

But the quality requirements, maybe the resolution requirements for that very early stage of pre training, are not as high as for the later stages of pre training. And of course, not as high as for the post training stage, such as SFT or some other stages, some other types of post training.

So, early on, we do have some very strict quality, quality barriers and, and quality gates for the data to get through. But we increase those, increase the requirements for quality the further we go into the training.

And of course, the volumes that are, that the model is trained on early on are much larger than the volumes the model is trained later on.

Now, another big difference between pre training and post training stages in terms of the data is that the data at post training stages often has more signals, more, I would even call it control signals, so that the model can learn very specific ways of how to react, how to operate when, when asked to do so.

So, in the early stages, in pre training, the model learns how the world works. It, it learns how objects move, how it learns about the physics of, of the, of a different interactions between, between objects and between subjects in the, in the videos.

Whereas, in the post training stage, you learn about, the model learns how to react to certain controls we ask it to do. And therefore, you need more signals at the, at the later stage.

You may be need some signals such as motion trajectories, or where you would like the subject or the object to move. Or maybe you need to pass, like, a depth mask to actually create a scene consistent with, with the desired characteristics of your scene.

So, those are the sort of controls that we might, we might pass in at the later stages.

Interesting. So, we're looking at different volumes of data in different stages, looking at diversity, data quality, control signals that we're looking for in supervised fine tuning.

Now, you've spoken about different modalities of data. One thing we haven't touched on below is synthetic data. How does synthetic data used? When is it used? What do you see the value of synthetic data as opposed to real world data?

Yeah, so I think that one of the ways to think about this is that if you want to learn how the real world behaves, you better use real world data.

So, we do want to use as much real world data as we can. However, it's very hard to find a lot of real world data about rare scenarios, and, or rare characteristics that you want to capture.

So, for those, often it's quite useful to use synthetic data. And we try to use synthetic data for, for covering some of the scenarios that we cannot model with real data.

However, of course, it can lead to some artifacts that the model doesn't generalize. So, every time we use synthetic data, we need to check whether the model generalizes to, to the real data scenario.

So, for example, if you want to learn the physics, the physics of the real world, you better use the real world. However, some of the characteristics might not be, actually some of the videos with the, with some characteristics might not be available in the real world.

You know, you don't have objects passing, passing through each other usually. And so, if you want the model to learn that this is a wrong thing to do, then you need to pass that sort of a data in a, in a special way. And for that, we generate, we generate synthetic data. That's an example.

Interesting. So, to summarize, you, at Reka, we try and get as much real data as possible, but when it comes to synthetic data, we try and leverage it for those edge cases that are harder to find.

In real data, we might be able to find it, but there might not be as much data on that. So, we want the model to learn it. Synthetic data is a good way to, to create it and to supplement already the vast amount of real data that we have in the model.

Yeah, I think that's a good way to frame this.

Now, in order for the model to really understand what the data is and what's in the videos and the images, we have a lot of annotations, we have captions.

What does that look like here? Do we use, at Reka, do we use human annotators? Do we mostly use models like Reka Edge to do some of our captioning? What does that look like in the, in the data pipeline?

That's another, that's another very interesting question, I'd say. Because often you do need a, you do need human annotations, as well as even human generated data, again, for either covering the edge cases that maybe you covered with synthetic data, but maybe your generalization to real data just is not good enough.

So, you do need some amounts of human generated data, human generated, let's say, videos or, or images, for those scenarios.

And I think a similar, a similar picture you can see in the annotations world as well. These days, you can use VLMs and vision, and VLAs to, to annotate a lot of the different aspects of images and videos.

However, not everything. For some of the scenarios, for example, for the videos with a lot of temporal dynamics, it might be hard to get reliable annotations, you know, even out of the best models out there.

So, we do leverage, we do leverage reliable human data pipelines to refine, to refine what we get out of the, out of the VLMs, as well as to provide ground truth for some of the classifiers and some of the detectors that, that we use internally.

We use a variety of different models, of course. Reka, Reka models, as well as other open source, as well as open source models, open weight models that are out there.

I think what's really interesting about Reka, now that you mentioned it, is that we have our internal annotators as well, who can help us with captioning, understanding what the ground truth is, and help us out with a lot of the experiments that we are running internally.

Now, we've spoken a lot about data, data quality, all the work that we have to do, the pipeline, diversity, modalities, and there's so much more we could be talking about.

But if you were to just tell me what is, like, the best thing about working for the data team here at Reka, what makes it unique compared to other data teams and other companies, what would you say?

First of all, Reka overall as a company is creating a future of AI. Is, is betting that the world models are going to become incredibly useful for all sorts of different applications, but they're not there yet.

And we are betting that we will be able to actually create those world models that will be useful. And so, that state-of-the-art work of creating such models is, is incredibly exciting, because there's, there's very few companies in the world that are doing similar style of, that face the most styles of the challenges.

And if you talk specifically about the data, abstracting away, what, what is the most important for the, for the model to be good? Of course, it's a huge aspect of architecture of the model and how you train that model. But I'd say data is equally important.

And so, you basically have working in the area that contributes a huge amount in how successful your world model is going to be. Is it going to execute on what you wanted to execute on? Is it going to create the videos that you like?

And so, those, everything, every, almost every single aspect that you'd ask about, that you'd want out of a world model, needs to be first thought through in the data team, and model through the type of data and the quality of the data that we feed into the model.

And so, that is an incredibly exciting part of working for the data team.

I couldn't agree more. I really think it's the vision at the company. And then it's all the challenges that we have on a daily basis to make sure that we get the best data possible, and that we process it in such a way that the training team can leverage it to build the best model.

Thank you very much.

Thank you, great talk.

Thank you, thank you, good chat.