How well does the model understand the physical world?

Hey everyone, I'm Marianna, a research scientist at Reka AI. Along with Roberto, I was working on a physical realism benchmark, and today we're going to dive deeper into it and explore it.

So, first of all, Roberto, can you introduce yourself?

Sure. Hello everyone, I'm Roberto. I'm also a research scientist here at Reka, and I've been leading the research on the physical realism benchmark.

So, today we're going to learn about physical realism benchmark and dive deeper into several questions. And yeah, Roberto, I'm curious, what was your motivation for creating this benchmark?

Yeah, so here at Reka, I was working on evaluating video generations coming from a text-to-video model. And for that, I was using VLM, so a vision language model. And let me show you what I experienced there.

So, here on the left, we will see a real example coming from a text-to-video model. And obviously, we see that there's a truck that's moving towards the camera. So, there's clearly some motion.

Now, when I was using the VLM to assess the video, and when I was asking it and to assess the motion quality, what the VLM was actually saying, that it was a state-of-the-art model, it was saying that the motion quality is very bad because there is no motion at all. It's just a static frame. It doesn't see any motion.

That's apparently very wrong. Everyone can see that there is some motion happening. So, that motivated me into looking more closely into this topic of how well VLMs can understand motion.

And that's why we can see another example on the right, where it's also fading. So, here we see a video where, on the left part, there is an object that suddenly disappears. Let's have a look at it again.

So, at the left part, there's this object that suddenly disappears. Now, I was using an LLM and asked it to describe the motion that it sees in the video. And it was saying that the object that just disappears is smoothly going out of the frame. So, it's moving, and then suddenly it's out of the visual part.

That's apparently wrong again. Here we have an object that just suddenly disappears, which is something that you want to avoid. But the model, the VLM, is not able to really understand what's happening here. And that really motivated me into looking more closely into this topic.

VLMs seem to be good at understanding visual content, good at understanding captioning images, what's happening in an image. They are maybe also good at understanding if an image generation or regeneration is well aligned to a textual instruction, but they seem to struggle with understanding physics, understanding if the videos or the content of the video is aligned with our understanding of the physical world.

So, it's important for VLMs to understand not only visual quality, but also the physical correctness. And yeah, that's the, as far as as far as I understand, the main motivation.

So yeah, and what's the physical realism benchmark in simple terms?

Yeah, so as the name says, it is a benchmark. So, we are evaluating different vision language models, different VLMs, regarding how well they understand physics in a video. So, if they're able to recognize if a video is physically realistic or not.

And to do that as a proxy, we collected a set of fundamental physical laws, like the law of gravity, impenetrability, conservation of mass, conservation of momentum. And now we created a data set of videos, videos that are physically correct, that follow all fundamental physical laws, and videos that have a violation that we injected into the video.

And now we use the VLMs to evaluate these videos, ask them to tell us if the video shows a violation or not. And then we create a metric from that. Based on that metric, we form the benchmark where we can evaluate and rank the different VLM methods.

Thank you. So, you said you have collected the correct data with corresponding physical laws. Can you tell us how you get these videos and accurate labels?

Sure. That's actually a very crucial part because if you just come up with into-the-wild videos and you want to label them, it can be very ambiguous creating the labels there.

So, our approach is to use synthetic videos. Let me show you an example.

So, we have videos that are coming from the render engine. So, these videos are synthetically generated. So, in this video, we see that there's a wood stick that is floating around in the air for some time before it's falling down. So, the wood stick is violating the law of gravity.

And this is now the video that we have synthetically generated. In contrast, we have also a video on the right side that follows all the physical laws. So, here everything is correct.

Both videos show the same scene. The only difference is that one time we injected programmatically a violation of physical laws, and on the other side we did not.

And now, since we're using a render engine, this means that we have full control over the violations that we're injecting, but also that we have very clean labels because the labels are coming from the renderer engines or from our programmatic injection.

So, that means they're all unambiguous. We know exactly which object is involved in the violation, when exactly is the violation happening, and which type of violation we see in the video.

Let's see another example of our dataset.

So, here we see an object that suddenly is moving without that there's some other object that's pushing it. So, that's a violation of the conservation of momentum. While on the right, we see again the same scene with the object adjusting on the ground, and nothing happens.

So, we have these pairings of videos that are physically correct and physically incorrect. And since we have this very clean structure, where there's at most one violation in the video, we avoid any compounding factors.

So, we can really analyze for each model what are the weaknesses and strength regarding the different fundamental laws, so we haven't, we can get an understanding what is their physical understanding in videos.

Yeah, I see. The correct data seems a key to any research. Was there anything surprising you came across during the research process?

Well, yes. First of all, to be honest, the surprising part was to me that it is actually necessary to create this benchmark.

So, what I mean with that is that, for us humans, the errors, the physical laws that we injected, are very obvious. So, they seem to be very simple for us to detect. While for us it is simple, it is very difficult for current VLMs, for current state-of-the-art models.

They are struggling in understanding or detecting these issues. So, that's something that I was not expecting and motivated us in creating this benchmark.

And then another finding that was quite interesting is that, in our benchmark, the models that were performing best were the models that don't have native video support. So, there are models where you can evaluate a sequence of images, but not a video natively.

And those were the ones that were performing better, much better than the ones that had native video support.

Yeah, I see. Yeah, and now, if I would like to explore the benchmark itself after, yeah, watching this video, can you tell me, like, where I should start and what's available today?

Yes. So, we have a web page where you can find all the information. Let me open the page and show you.

So, we have here a central page which covers all the information about the benchmark. So, you see some examples of the data set with the physical violations that we injected.

So, for example, gravity, where we have objects that are hovering around before they're falling, or impenetrability, where objects are intersecting each other, violating physical laws.

And then, most importantly, we have also a blog post, we have a leaderboard, and we have a way how others can participate in the benchmark.

So, the blog post contains all the information, all the motivation of the benchmark, how our benchmark relates to other existing benchmarks that use VLMs for video understanding and video assessment.

We also provide details about the data set, and then finally some examples, and also how we are actually computing the metric in order to rank the different models.

Then we have the leaderboard. So, on the leaderboard, we see now the performance of the different state-of-the-art models. We see that the best-performing model has a realism score of sixty percent. So, it's still far away from really understanding reliably physical realism in videos.

And when you click on one of these models, you get all the details. So, you get, per law, what is the accuracy of that model detecting violations for that law.

When you scroll down, you see also some examples of the videos that the models had to assess. So, different categories, like the impenetrability, and there was without violation. That one is with violation, where the one object just goes through the other.

And when you scroll down, you also see what the model was responding and if that's aligned with the ground truth or not.

So, we see, for example, that the VLM is saying that in that video impenetrability is obeyed, while the ground truth is actually that there's a violation that we can clearly see here.

So, that's very insightful. We can get all the information about the different models.

And then finally, you can also participate by yourself in the benchmark. You just need an API access. And when you fill out the form here, then we will run our server on your model. And then it will run over the entire benchmark, and you will get the score of our benchmark, and you will have a submission row here in the leaderboard.

So, that's a very central page, where, or a very convenient way, since we have the central page where everyone can compete with each other, and we can understand what is the best model regarding physical realism, regarding understanding physical realism.

Wow, that's really helpful to have everything on one page. So yeah, if I'm gonna have a new model and would like to test it out, I will come to this page and, yeah, submit my model.

So yeah, is there any directions you think this benchmark can move forward? Like, any other new physics concepts or scenarios you'd like to add to the future versions?

Yes, certainly. So, I mean, there are different axes how we plan to improve it. The one is, of course, the coverage.

So, as you say, more scenes, more diverse scenes, but also more fundamental physical laws that we want to cover, more probes of these fundamental laws, more different ways how we test if the model understands these laws.

But also, we are planning a training set that is designed for these problems, for understanding physical realism. And then it would be very interesting to see if models can improve their understanding of physical behaviors when they are trained on this dataset.

Thanks a lot, Roberto, for explaining physical realism benchmark during this short session.

So, the thing I've learned is that VLMs are not able to assess physical realism of the videos. And that's the main reason why you came up with the physical realism benchmark.

And I hope this benchmark will support all the future researches. So yeah, hope you found this short session helpful. Thank you for joining us.

And if you'd like to explore more about Reka Labs, just click on the link below.

Thank you.