How do we prepare data for a World Language Action model?

Hi everyone. My name is Julian Lopez. I am a program manager here at Reka.

And I am Konrad Jamrozik. I'm a tech lead of the data platform team at Reka.

Fantastic. Konrad, why don't you tell us a little bit about what you do here at Reka?

So, my team is responsible for preparing data for our World Language Action Model training team. Basically, the way you can think about that is that, to train this world model, you need a lot of data. And by a lot of data, I mean hundreds of thousands of video hours.

And it needs to be prepared by a quite elaborate data pipeline. The way you can think about that is we are obtaining data from different sources, including our own solution and data marketplace, Claru. We'll talk about this in a second.

And once we get this data, it needs to go through over a dozen of different stages, including standardizing the metadata, figuring out how long are the videos, what is the resolution, and so on, cutting it into clips of the right length, sizing them, making sure they are encoded correctly, then doing quality evaluation, transformations, annotating it, and so on and so on, to finally obtain a standardized set of data with, with appropriate characteristics and quality that is then given to the model training team, so they can use it to train the model.

Sounds really hectic. How do you ensure that all the requirements are neatly documented and that you're giving your training team exactly what they want?

Yeah, so we are a fully distributed team, full remote team, all across the planet. So, the way we are trying to solve this problem is we, of course, spend a lot of time on Slack, collaborating intensely, but we also have our knowledge base based on Notion.

And this is, like, the base easy stuff that what we do, which is, like, not unlike other companies. But also, we are trying to leverage our agentic infrastructure to help us with, to deal with this problem.

Maybe, actually, you could talk a little bit more about that, because I know you've been tackling this problem.

Sure. Well, one of our priorities here at Reka is to ensure that we can automate as much as possible and that we can leverage agentic workflows in our road mapping, in our data sourcing, data processing, and monitoring how data is moving throughout the pipelines.

That is one of the big focus areas for us. And when we leverage all the, like, those agentic solutions, we're able to focus more on thinking strategically about what data it is that we need and what are the best requirements or the best data that we can give to the training team, so they can build the best model possible.

Could you also tell us a little bit more about our data sourcing solution at Claru Marketplace?

Sure. So, we get data from a variety of different places. We have some licensed data that we use for our Marey model. We also have open source solutions, and we have the Claru Data Marketplace.

This marketplace is an end-to-end solution for Reka to collect and annotate data from around the world. So, this makes it easier for us, as we're building world models, to ensure that we're getting egocentric data from different corners of the world, to ensure that we not only have the right quality, but we also have the diversity that we need to build the best model.

What do you think is the biggest challenge that you face in your job here at Reka?

The biggest challenge is that we have a lot of challenges that are all big, which is that, first of all, we need to prepare a lot of data, like I mentioned, hundreds of thousands of hours.

Second thing, because we are a research startup, we have a very strong AI research muscle. We have multiple researchers originally in Google DeepMind, senior staff researchers, staff researchers, which means that we are still constantly trying to improve how we train the model, which means improving what we do with the data, how we filter it, how we transform it, what kind of data we exactly need.

Which means that not only we need to prepare a lot of volume, we also have to make, we also have to make sure that we are constantly able to augment the pipeline and adjust it, the data preparation pipeline, so that the data is constantly being processed in a different way.

So, often what happens is we, we know we already have some datasets, and they are very different. They require, each of them requires, slightly different kind of processing. And our research team tells us, "Okay, now we need to this additional thing. There is another ML model that is doing another kind of transformation of the data."

So, we need to incorporated in the, into the pipeline while we already have requirement of preparing a lot of volume, like, you know, two weeks from now.

So, it's basically taking everything together. Yes, we have large volume, a lot of different heterogeneous datasets, and constantly improving, and that data pipeline that is constantly being augmented and improved.

So, doing all of this together at the same time is indeed quite challenging challenge problem to solve.

No, I can imagine. I think what's very interesting about Reka is that we have a very small but mighty team around the world. Everyone has such a unique background, very technical background, and even though we're a company of less than 100 people, the output that we have is immense due to the ownership that we have throughout the company.

Yeah, and we're also, we're also managing to survive with such a small team because of what you said. And also, we are also really on the, I would say, we're on the bleeding edge of leveraging AI infrastructure.

We are constantly, you know, we are, of course, staying on top of all the developments. And I would say we are on the forefront of figuring out how to use all the, you know, different models that are out there to augment our work and basically do more with less people, and also lesser coordination overhead. That definitely helps.

Yeah, you know, I completely agree. I feel like throughout my time of, at Reka, I've really noticed that everyone in the team pushes each other to get the best out of them.

And we do ensure that we're not just looking inward, but we're looking at the market. So, we're seeing all the different developments that other models produce and other competitors produce, in a sense, to then be able to understand how can we make our model better, how can we produce better data that we can then feed into the model itself.

So, what do you like the most about working at Reka?

It's gonna sound very cliche, but I would say the people.

It's not the agents?

People, not the agents. Not the, the agents are very helpful. Agents are what allow me to do my work much faster. But no, I, I would say it's the people.

I've worked in a variety of places in my life, and this is one of those companies where you just know that everyone you work with is working just as hard as you are, and everyone's committed to this, to the same mission.

We want to make sure that we develop the best models, that we achieve our vision together, and that fosters collaboration, which you won't necessarily see in other larger tech companies.

So, we're small, we're mighty, we move quickly, we're not afraid to break things, but at the same time, because of the ownership that we have throughout the company, we're able to really work together towards reaching our goals.