Why your AI can read but can’t watch (yet)

You probably have heard AI absolutely everywhere: in newspapers, in thanksgiving, in your dates. You’re probably a bit saturated with it, but the biggest question still remains: how can I make money from this? Is this actually useful? Is this just the trend? So did you know that now AI can actually understand videos? But importantly, how does this affect you? Is this something that you should actually pay attention to. I’m Josemaria Soriano, I’m Growth Product Lead at Reka, an AI Research Lab.

My name is Mateusz Malinowski. I’m one of the Co-founders and Head of World Models.

And today we’re going to talk about vision language models, which are AI that can truly understand anything in a video. But let’s start from scratch. Mateusz, what is a vision language model? Is it the same thing as a chatbot? How is it different? Is there something new with a VLM? Why should we care?

So Visual Language Models, which are sometimes abbreviated to VLMs, are a new class of multimodal models. So chatbots originally, they were operating on something that we call large language models, LLMs. Large language models are trained purely on texts, and because of this, they can reason about the text, but not necessarily reason about the real world, like the images or the videos and so on.

So what they are lacking is the spatial understanding. Then you can ask questions about the motion, the intuitive physics and so on, and then you can see that most of them, they will fail in the way that we humans wouldn’t fail at all. So even though it’s a very mature and impressive technology, they still lack this kind of intuitive understanding of the world that we humans do have.

VLMs could also be used in the chatbots, of course, there’s nothing that limits. They can also reason about the text, but they can also, in addition to the text, they can reason about the images and the videos. And it has very important applications in the real business as well, for instance in marketing or in branding. So maybe you can share a little bit more your perspective on this.

One of the biggest issues that happening right now in the industry is that companies that are doing different consumer items, they’re all marketing with creators. But the problem is that they are using hundreds of creators for just one campaign, and then it’s really hard to actually verify. One, if those videos have identified the creators are compliant with regulations. Two, if they are actually in line with marketing briefs. And three, if they have actually mentioned the correct disclaimers.

Think about, for example, any protein bar or any health care supplement, you need to be really good at all of the specific details, because this is something that is, that this could be a health concern. The advantage of VLMs, and what we’re seeing and what I see on a day-to-day, is how can we use these VLMs to actually analyze hundreds of videos at scale. How would you simplify it for someone that is not technical at all? What’s the difference between computer vision, a VLM, and why that gap is so important?

Classical computer vision tends not to generalize pretty well, so it’s working with the fixed vocabulary of categories. So let’s maybe think about sports. Which club is your favorite one, like the football club?

I’m a Liverpool fan actually.

Liverpool, yes, okay, sounds good. So let’s say that you use the traditional computer vision methods for Liverpool, okay. So the traditional method could detect players, that’s fine. The issue is that let’s say that you want to go beyond that, so you would like to also understand the strategy, what kind of strategy is happening in the team. And for this you need to understand the relationship between the players and also the players themselves. It should also generalize better when swap the players as well. It might not recognize the player because it doesn’t have the name in the system, but it still shouldn’t break catastrophically in terms of global configuration in reasoning what is the strategy and so on.

There’s usually three things I’ve seen that most VLMs need some help with. The first one is specializations. I would love for you to chat a bit on how different specializations for different fields improve significantly the output and why our audience should be concerned on that.

The thing though is that specialization brings the performance much higher, and this is actually something that is very different, very important for many different business use cases, right? Because okay, you can have the VLM, the generalist VLM, that can achieve let’s say 80 percent the performance, but actually what you want to have is that you want to have a model that is achieving 95% accuracy, right? So this gives you much, much bigger revenue.

Technically speaking, we are often distinguished between two or even maybe three phases of training. So have pre-training, and then we have post-training, and then perhaps also the customization. And in the pre-training we are building the generalist model, so the model that can understand the world in general and in a little bit abstract sense. But then the role of the post-training, and crucially the customization, is to make it much more specific to your use cases and to your business use cases. And this is something that we are also offering here at Reka. We can build those customized models for you.

I can also imagine that in the emerging field of robotics, that this could also be important when we start thinking about embodiment. If we want to have a robot that acts like a dog or acts like a human, so they have different embodiments, they have different configuration of the joints. And I guess for this we also need to have models that are much more customized. So even though that the generalist model can understand the world, it might not understand the body that we put the model into and then ship. So that’s, very important aspect.

You have the big labs that do VLMs. One of them is Google, right? Gemini has also some vision understanding capabilities. There’s a specialized companies that do some VLMs for specific fields. One of ours is ours, Reka, we focus a lot on security, sports, entertainment, media. Another one is Twelve Labs, And then you have also the open source models or open weights models. One of them it’s, Qwen for example, that I believe from Alibaba.

Let’s say for example that I’m a Hollywood executive. I want to specialize all my VLM to understand all of the Marvel Universe, for example. We just had the new Spider-Man movie that just came out. What would be the route if I use a Gemini versus if I use a specialized model or a specialized lab like Reka that can do some customizations, or if I use an open weight model like Qwen — I believe Kimi is also is the open weights as well. And what are the pros and cons of each scenario?

The good thing about building models on-premise, is that the companies don’t have to send data to third parties. So, for instance we could take our model, the base model, then we can do the customization of the model for the customer’s needs, let’s say the Marvel universe as you mentioned. And then we can actually ship this on the cluster that they believe it’s a safe cluster for them, so it’s a trusted cluster. And for some players, for some business use cases, this is critical. Not always you are okay to ship your data. Especially imagine that there’s a new movie that doesn’t exist yet, and then if you send this data over, then there is the possibility for the leak.

But if you have a trusted third party it doesn’t have to be us. It can be like some arbitrary another company or maybe internally in the company itself. I think that the benefit of having the cloud is that to some extent is the convenience, because then you don’t have to think about the cluster at all. Like if you are type of the customer that doesn’t want to think about cluster and infrastructure then I think that the cloud solution is the simplest one. I’m pretty sure that for many customers that would be to go solutions, so there’s also nothing wrong with that.

I love that. And I think that for our viewership on LinkedIn or YouTube and other platforms, a good rule of thumb is that if you need a simple VLM that will detect basic stuff, for example chairs, people, or clothing like your sky blue shirt. You don’t need it to run on-prem here, meaning privacy isn’t a concern and can run it on the cloud, honestly, we have seen Gemini work pretty well.

If you need a significant level of customization, right, for example either for sports, media, public safety, security, or, and then you need that 95%, and you’re talking, and you’re talking about a level of context that needs some customization, you probably are going to be better if you work with either company like ours or other companies that are offered specialized models, because not only is going, we’re going to provide you the VLMs as is, but we’re going to do some customizations. That can be fine-tuning, can be prompt engineering, can be some computer vision scaffolding to improve that accuracy to that 95% that we targeting.

But if you actually have privacy considerations, if you are in healthcare, or either if you’re in security, if you’re in the government, right, and you need on-prem, that’s when we go into a third level complexity, third level of, I would say, a VLM scenario, in which you need not only a company that can do customizations but can also run on-prem. One of the issues of running on the cloud is that when you actually look at your storage cost on either AWS, Azure, GCP, for all of this video understanding, it starts getting very expensive very fast. Can you quickly explain to our audience.

So of course there are pros and cons. So there’s no universal model that serves everyone, but frankly speaking, in many scenarios, in many cases the domain is more contrived. So for instance, if you are thinking about the live stream and the sports, you don’t necessarily need to have a super potent model that can solve the problems in quantum mechanics. But at the same time, the benefit of having small models is the cost and the speed, they are much more efficient. But for this kind of settings that you described, the system has to be very reactive.

Yes.

And also very interactive, the physical world is interactive. It’s not like everyone is doing all the decisions in turns, it’s not like a multi-turn, kind of strategy game. It’s something that happens, a lot of things are happening at once, and then the model has to react very quickly. So maybe you can describe a little bit more about this kind of use cases.

Yeah, absolutely, yes. So I mean we all watch the World Cup. One of the interesting things is that now everyone wants all of their highlights as soon as they happen. People have now short attention spans because of social media. And whenever Mbappé would do either a goal or, then, Messi would do a very nice dribble people are expecting this in TikTok and Instagram as soon as it happened.

And the first either company or YouTuber that can put that on social media is one, is to the most views. And one the biggest things that are actually getting from enterprise customers is that they are like, hey, I have this league, I own this league, so how can I get it as fast as possible? And that’s where we’re seeing lot live stream clipping, that’s very big thing that actually get a lot of traction for.

Let’s say that I’m an executive and I’m in charge of AI procurement in organization, I don’t have much of technical background but really know what my organization wants. What would be the two or three things that you would suggest me to evaluate whenever I’m evaluating these different VLM providers or these different systems that say that do vision understanding? What would be the rule thumb, the rubric, that you think that I as a security leader should implement in organization?

Yeah, sounds good. So I think that the first one is that you have to have a good understanding what are benchmarking. And often in many business use cases the internal benchmarks are different benchmarks than those general benchmarks that we tend to measure the performance of the models. And I think that this interactive real-time use case is very interesting one, because the model can be a bit worse on the, let’s say, some kind of question answering benchmark, but this benchmark doesn’t necessarily benchmark another aspect that is critical for your application, which is how reactive the system is, how quickly it can answer your question, as opposed, how accurately it can answer this question. Again, both aspects are very important, but there are slightly different use cases, and often there’s a trade-off between the accuracy and the speed.

The second thing is that to think about the customization as well, and I think that this is kind of related to the benchmarks, because if you know what are your use cases, you know how to benchmark, then you can also start thinking, oh, what kind of data are actually interesting, what kind of domain is interesting. And then we can do the customization. And this is the second aspect that is very important for execs to think about this.

Yeah, I love that. And so let’s do a quick TL;DR. I know we’ve talked about a lot of things. Number one, a VLM is an AI model that can understand video and understand context within a video. They are different from computer vision, traditional computer vision, mainly because you don’t need to pre-define everything that want to test. They can derive insights from videos. And very different than chatbots, so whatever you’re thinking is the best models on the text side is a whole different thing on the video side.

And finally, to wrap it up, if you actually want to test these models, grab 20, 30 videos that you would actually want this VLM to understand and get insights from, and then manually find the answers that you’re looking for this AI to get, for you. Did I miss anything, Mateusz?

I think that is well said. I would say that all the benchmarks are important, and of course academic benchmarks are also important, but the ultimate benchmark is the real world.

Awesome. Well, hope you really enjoyed this episode. Stay tuned, to learn more about VLMs, what AI can do with videos. And thank you so much, this was really good.

It’s a pleasure, thank you, bye-bye.