Hello everyone. My name is Mateusz Malinowski. I'm the head of the world models at Reka. I'm a research scientist. Hamid, can you introduce yourself?

Hi everyone. I'm Hamid. I'm leading the robotics team at Reka.

So, Hamid, we are here to introduce our two benchmarks. Can you tell a few words about it?

Yes, so we have the physical realism benchmark. So, we're trying to understand if video language models can understand physics in the way that we do. Our benchmark is designed to provide these models with specific violations of physical laws and then ask them if they can detect those violations and they can reason about their predictions.

So, not only they need to understand which laws of physics are being violated, but they also need to provide enough reasoning to justify their answers: which object violated which physical laws at exactly what frame in the video, so we know that the answers are not hallucinated.

Hamid, tell us why we are building this.

The video language models are increasingly used for critics to judge the generative models' realism, and if they're not accurate enough in understanding physics, then their judgment cannot be trusted. So, that's something fundamental that we need to fix in the video language models, and we're building this benchmark because we want to have more control over how the models are being evaluated.

Physics is a very broad term, so can you share a little bit what kind of physics this benchmark is focusing on?

Yeah. We're targeting certain laws of physics to start with that are very intuitive for us and also very important for real-world deployment. For example, the conservation of mass of objects, so the objects should not suddenly appear or disappear or change shape without a reason; the law of gravity; the conservation of momentum, that if the object suddenly starts moving or stops, there needs to be a physical reason for it; and also the impenetrability of different objects.

For example, these mugs, they shouldn't suddenly go to each other. That's not the way that our real physical world behaves.

What are the biggest surprises when you are working on this project?

Yeah, so we're finding even the best models out there, they have up to 50% accuracy in our current benchmark. They hallucinate a lot. They don't understand the motion in the intuitive way that we expect them to. They hallucinate when the objects are near the border of the videos or in different scenarios.

So, we already have a very good understanding of what are the shortcomings of these video language models and how we should improve them.

Okay, so if I understand correctly, with this example that you show about those two objects that pass through, so it means that the existing VLMs, they still make the mistakes, and they're thinking that the objects are not colliding, but actually they are passing through. Is that correct?

Yes. So, we want a model that can understand our physical world, how the realistic behavior of different objects work, and it's able to then help us to create a better generative models that would create outputs that can be deployed in the real world and they can be trusted.

It can be on robots, it can be self-driving cars, it can be any, any platform that interacts with the real world, and we want those models to understand the real-world dynamics and how they, how it behaves.

Yeah, it makes sense. Okay, so to build those models, we have to qualitatively and quantitatively understand where we stand on.

Yes, exactly. We want to have a strong evidence on the performance of these models, not just trust their judgment, but be able to verify their accuracy.

Physical realism is the first benchmark that we release. The other benchmark is called WorldModelGym. Hamid, can you share a little bit some details about the WorldModelGym and how is it different from the physical realism bench?

Sure. So, first, world models are simulators of the worlds. So, they understand how the world transitions. Say we have a current state in the world and we have some actions that are happening, and the world model will understand how the next state of the world happen depending on the consequences of the actions that are happening in the world.

So, in this benchmark, we want to focus on the behavior of the agent. So, we ask the agent that trained in a ground truth environment, that knows exactly what is happening in the ground truth, and it can operate and act and with a high accuracy.

We ask that agent to now switch to its own internal representation of the world, which is the world model, and use that to predict the consequences of its actions, then acts in the real world, and then we measure how well that agent behaves and how much of the performance that it had in the real world can be conserved when it's switching to its internal representation of the world.

Okay, sounds good. My understanding is that it differs from the physical bench because it's not evaluating VLM anymore. It's evaluating the world models.

Yes. It's evaluating how faithful is the world model representation compared to the ground truth environments that they're trained on.

Why do we need it?

Well, we need the world models to understand the dynamic of the world, and we need the agents to be able to rely on them to predict the consequences of their actions so they can behave safely and reliably for real-world deployment. And for that, we need a world model that can accurately understand how the world evolves.

Hamid, tell us what is the most unique about this benchmark.

Yeah, so what is unique about our benchmark for world models is understanding how the behavior of the agent changes when it switches to using its own internal simulation of the world to act in the ground truth environment compared, like, for example, other benchmarks out there usually measure if the agent succeeds in a task, but they don't actually try to understand the behavior of the agent and the divergence of the sequence of actions that it executes in the world when it's using its internal imagination versus just observing the world itself.

What is the scale of this benchmark?

Yeah, so we have over 100 different environments across different families of benchmarks, including Atari games, a DeepMind Control Suite, robot manipulation tasks, and so on.

Why two benchmarks?

We think that it's good that we separate these problems into different benchmarks so we can accurately measure how they perform and how will they understand these different concepts, and then hopefully in the future we can bring them together into one cohesive framework.

Right now, we're keeping them separate so we can measure one thing at a time, but they can definitely be mixed, and that is going to be our actual deployment in the real world, where we combine the physical understanding and the world model that brings the transition understanding of the world.

Yeah, makes sense, makes sense. I think that is good idea to have a multiple different benchmarks. Each benchmark focuses on some feature of the model. Because of this, it can also guide us how to build the next versions of the models correctly.

Well, yes. As we said, the world model is trying to predict how the world is evolving, and the physical realism comes into picture when it wants to critic its own imagination, when it wants to understand if its imagination is accurate enough and can you improve over time.

Yes. So, Hamid, what's next?

So, we like to collaborate with the community. That's why we're releasing the benchmark publicly, so everyone can participate and we can see what are the best-performing world models out there.