A key ingredient of training a new world model is the data. At Reka Labs, we have a dedicated data team responsible for preparing petabytes of video and image data for our model training. Here we outline key components of our data pipeline and some challenges we face. In follow-up blog posts, we will delve deeper into specific parts of the data team’s charter.

Data Pipeline Outline & Key Challenges

Preparing data for world model training requires many processing stages, starting with sourcing, then moving through various steps of analyzing, filtering, transforming, and annotating video or image data. When the final artifact is delivered, it is ready to be used by the research team training our “omni” world model from scratch.

The key challenge with the data pipeline is the tyranny of the project management iron triangle: cost, time, and quality — pick two. On one hand, we want deep analysis of the data to ensure only high-quality videos reach the model (quality), but, on the other hand, it is computationally expensive (cost). While training our models, we are quickly learning what works and what doesn’t, resulting in fast turnaround and the need for rapid adjustments to the data pipeline, shortening our timelines (time).

Managing these three goals is a fine balancing act.

Furthermore, we are operating both on a few large datasets and on hundreds of small, heterogeneous datasets, and different modalities.

Because of the large volume of data and short timelines, we must approach the data processing problem with a scalability mindset. We constantly analyze where the next bottleneck is and put just the right amount of effort into addressing it until something else becomes the most pressing issue.

On the other hand, the heterogeneity of the datasets increases the challenge of automation. Each new dataset brings new corner cases, requiring us to be ever vigilant about tooling reliability issues, and to constantly adjust our AI harnesses.

Stage 1: Sourcing

The first stage of our data pipeline is data sourcing. This stage immediately poses a series of challenges.

First, we must discover datasets with the right characteristics required by model training, and in the appropriate volume.

Second, we need to standardize the dataset’s metadata, format, and location, so that its processing through the data pipeline can start in earnest. We developed a formal specification of what the data pipeline tooling expects, and each dataset must be converted to match the specification. We call this process data ingestion.

One of the key data providers for us is Reka’s Claru, a data marketplace that provides purpose-built datasets for frontier robotics, embodied AI, and world models.

Overall, the sourcing stage is in many ways the hardest to automate, and requires a lot of human review. We developed detailed guidance for our AI agents to help deal with the dataset heterogeneity, but pretty much every single new dataset poses a new corner case we haven’t covered yet, requiring us to constantly update our data ingestion AI harness.

Stage 2: Analysis

There are multiple layers of data analysis. During ingestion, while we are reformatting a given dataset’s metadata to match our formal spec, we are also analyzing it with ffmpeg to produce a standard set of basic metadata. We obtain information about each input video such as width, height, number of frames, fps, codec used, and whether the video is interlaced. We also do a series of “health checks,” ensuring that e.g., the video has correct key frames and is not corrupted. This process is very fast and compute-efficient.

Later on in the pipeline, we do more heavyweight analysis that inspects the video contents and requires GPU computation. For example, we run shot boundary detection or compute various other quality scores.

We also leverage Claru for datasets with strict quality requirements, such as those used for supervised fine-tuning.

The aforementioned iron triangle of competing goals fully manifests itself in the analysis stage of our pipeline. Often, we must rapidly deploy new, augmented data analysis, and it is extremely expensive to perform a full analysis on every input video.

Stage 3: Filtering

One of the key benefits of doing analysis is data filtering — corrupted or poor-quality videos never reach the model. What counts as “poor quality” is decided by a bank of 9+ Quality Analysis models, each scoring every clip on a different aspect such as aesthetic quality, presence of text, and others; the combined scores bin each clip into a quality rank. The same stage also computes video embeddings, which we use to deduplicate near-identical clips. This is an interesting chicken-and-egg problem: to save processing costs, we want to filter out videos, but to filter out a video, we must analyze it. Since running a full analysis is computationally expensive, we first analyze each given video “only a little bit” to determine which videos even deserve full analysis. Overall, this poses an important challenge of proper sequencing of the various processing steps to maximize cost savings.

Stage 4: Transformation

Training a model on videos requires GPU efficiency, which in turn requires frame-level precision of the input video lengths and distribution, as well as appropriate resolution. This means input videos must be segmented into appropriate lengths, also taking into account the scene boundaries detected during the analysis stage.

Furthermore, some videos may require post-processing, such as removal of black bar padding on the sides, resolution scaling, or re-encoding to an appropriate codec.

Transforming input videos at our scale ultimately runs into the same challenges as the analysis stage, but even more severe ones. We are not only inspecting the videos, but also their re-encodings.

It is also worth noting that many of the video transformations we do rely on ffmpeg, and the exact ffmpeg incantations we developed to transform videos at scale, at low cost, and to handle all the myriad corner cases coming from our varied datasets are one of the many key elements of our pipeline.

Stage 5: Annotation

World models cannot be trained only on videos and images; they must also be annotated with appropriate information. For example, we require a description of what’s happening in the video, or a taxonomy of what is in the video, e.g., people, cars, nature. At our scale the only feasible way of producing such annotations is to run them through a dedicated vision-language model. We have many different kinds of annotations, produced based on numerous prompts, all depending on the exact use case we are aiming for, like video shot extension.

Annotation is also the stage where our pipeline forks. The same refined and annotated video trains two different kinds of models: generation models, which learn to synthesize physically plausible video from a description, and understanding models, which learn the reverse direction of reading, localizing, and reasoning about the physics, motion, and objects in a video. The class of models that combine both capabilities is called omni models, and they are central to our pursuit of physical AI: a model cannot act in the physical world without both visualizing it and understanding it, and both abilities are mined from the same data. Feeding generation and understanding from a single pipeline means every improvement in data quality or annotation richness pays off twice, which is a big part of why we invest heavily in this stage.

Stage 6: Deployment

Once the results from all aforementioned stages are ready, the data must all be assembled into a standardized format and delivered to the training cluster storage, ready to be ingested by the model training team. As such, the data pipeline both starts and ends with a formal specification: the expected inputs and the expected output. We also deliver in stages rather than as one big final drop: training starts on early increments while we process the rest, and feedback from the research team flows straight back into how we handle the remaining data.

Every delivered sample can also be traced back to its source video through auditing metadata. This lineage is particularly essential whenever we are working with licensed data and attribution.

This stage is the main interface step between the data team and the research team, the place where the “rubber hits the road,” so to speak — in other words, a typical integration point between multiple teams. The key challenge here is that sometimes we discover issues that ideally should have been caught upstream. The reasons for that include the fact that the requirements for what exactly the data pipeline is supposed to produce are constantly in flux, due to our rapid iteration.

Infrastructure

All internal metadata owned by our data team is stored in the ubiquitous Parquet format, together with the media content it describes, both of which live mostly in AWS S3 buckets. 

We have two types of jobs. Everything involving raw video data runs on Ray on Kubernetes: ffmpeg transformations, scene detection, and GPU model inference are all submitted as Ray jobs to our autoscaling clusters. We built an in-house distributed processing framework on top of Ray that splits datasets and processes metadata rows in parallel together with their video payloads, before merging the results back into our data lake. This framework provides several benefits:

• Crash recovery: Enables automatic retries of failed rows and per-step resource configuration.

• Scalability: Supports processing hundreds of thousands of videos that scales linearly with more GPU nodes.

• Auditability: Records input data, code versions, and exact logic to simplify targeted re-computation.

• Standardization: Abstracts away data-processing boilerplate to simplify new processing steps.

Ray Data is a workhorse here: the QA model bank uses Ray Data batch inference, streaming clips through multiple GPU-hosted models with backpressure keeping memory in check. As our volumes grow, we expect to lean on Ray Data even harder.

Pure metadata operations do not require accessing the media files, so we run them on a different engine: Spark on AWS Glue. We chose serverless Spark on AWS Glue because it gives us a battle-tested runtime without the burden of maintaining Spark infrastructure ourselves. This step is where the outputs of all pipeline stages get merged into the megajoin.

The megajoin parquet is then sharded, filtered and otherwise processed by our Ray engine into the WebDataset format with contents, metadata and format fields laid out according to our internal specification establishing the data format required for model training. We chose WebDataset for its widespread adoption, tooling support, flexibility and training input streaming performance.

A Case Study

To illustrate the rapid feedback loop between the research and data teams, consider the following case study.

We discovered that our model had begun generating videos with unexpected blockiness artifacts. While these artifacts were difficult for an untrained eye to notice, once seen, they were impossible to ignore. We immediately investigated the pipeline and traced the issue to inadequate detection of the blockiness artifacts during the analysis stage and tainted datasets, specifically those containing MPEG artifacts.  Next, we had to determine how to manage datasets at various stages of processing. For each partially processed dataset, we had to decide whether to rerun an improved analysis, assume it was clean, or discard it entirely. This required rapid execution, as model training was already underway, and any delay risked feeding further compromised data into the model.

Another similar case was when we detected our model was generating interlaced videos. It turned out that a very small percentage of the videos were interlaced. At first we didn’t have a detector for interlacing at all. Due to the rarity of such videos, even manual spot-checking didn’t discover them at first.

These processes take us only a few days. We often have to do them in parallel with other high-priority issues — for example, when the launch of a new model training cycle uncovers unforeseen problems." 

Over time, we have built expertise in handling these scenarios, enabling us to gradually reinforce our toolchain by pushing more robust validation upstream while maintaining cost-efficiency.

Closing Thoughts & What’s Next

Every model we train is downstream of this pipeline, and the data it produces sets the ceiling on what those models can learn. The goal is not less data, it is better data at scale: more high-quality volume per dollar, delivered fast enough to keep pace with research iteration. We iterate and augment the many steps of the pipeline rapidly according to feedback from the research team, while continuously ingesting and processing new datasets. We constantly scale up the pipeline, tackling the next bottleneck and handling the next corner case that broke last night’s job.

This post provided a high-level overview of what we do. In future posts, we’ll dive deeper into individual processing stages, covering our tech stack, the technical challenges we faced, and the solutions we developed along the way.