Can we trust how AI understands video?

Hello, it's another episode of The Working Notes. I'm Zhen Wei, a Forward Deployed Engineer at Reka, and my role at Reka allows me to support enterprise with the adoption of Reka's multimodal models, like Reka Edge, Reka Flash.

Today, with me, I have Thong.

Hello everyone, I'm Thong. I'm a Research Scientist at Reka. So, my work on visual language models revolve around one question: can the model really reason through the video content, or does it just describe it?

So, today we're gonna explore this question by looking at four things: where, when, how, and why things matter. And this connects to world models, not just generating video, but understanding it.

That's right. So, Thong, yesterday I watch a demo where our model was tracking bikes at over 300 km per an hour. My first reaction was, okay, it sees a bike, but what, what's next?

Fair enough. So, seeing a motorbike now is pretty easy. What really matters comes after that. Where exactly, when things happen, and does it matter? That, that's really important in the existing world now.

Great, this sounds really helpful for race analysts. Could you walk me through how the model watches the race? How does it understand the bikes, the riders, and track them?

Oh, sure, yeah. So, I would say there are two levels of precision. The first level is what I call the bounding box level. It's like when you type a prompt such as "detect red car" into Reka Edge, and it will give you a box for every red car.

But bounding boxes will only loosely capture the objects. The second, but harder, level is what I call the segmentation. This is necessary when you need to understand the video content in a more fine-grained way. So, this way, the model will decide whether every pixel belongs to the object, or it's just a background.

Wait, but a single photo can't tell you what happened. How does the model learn when the bikes actually overtake each other, and when they crash, for example?

Oh, sure. So, one frame tells you what is visible, but only a sequence can tells you what is happening. So, we try to make the model reason across the time, such as we can find a long activity over hours of footage.

So then, the, when the answer happen between two timestamps, and we turn a video into a raw timeline of every subject.

Yeah, this is the part that I would like to slow down a bit more, maybe for our viewers. So, let's say that we have 20 riders. They all have the same bike, they look identical, but they maybe have just different stickers, different motifs, but they're all going at 300 km an hour.

So, how does the model even begin to understand and identify which bike and which position in the race they're at?

Yeah, oh, so, you know, because a single frame actually cannot reveal a lot of details. So, we try to combine multiple frames together to extract useful signals, such as we extract appearance details.

Like, when you have two bikers at two different timestamps, but they possess similar features, then you know that they are likely to be the same person.

We also extract contextual information. Like, in a roster, there are 20 riders. So, you know, two different riders in, in the same scene cannot be the same person.

Okay, so the model can see precisely which rider it is, follow the race sequence, and know who is who. So, what's left from this? Where can we go from here?

Oh, sure. So, after recognizing the objects, what we need to have after that is the judgement. So, we need to tell the difference, like a door opening at 9:00 a.m. and a door opening at 3:00 a.m.

A door opening at 9:00 a.m. is pretty normal, when you have to go to work. When a door opening at 3:00 a.m., with a person hiding his face, is worth flagging.

So, we need, in order to tell these moments apart, we need spatial, temporal, and identity reasoning together.

Yeah, definitely. These are, like, new capabilities that our VLM brings across, and I certainly do not want a stranger opening my door at 3:00 a.m.

So, could you show me an example? Do you have benchmarks?

Yeah, so this is from SmartHome-Bench, a benchmark built specifically for home security. There are seven categories, spanning from urban to wildlife.

Yeah, that sounds like a really, a big dataset. And it, so this isn't just a home security thing?

Oh, not at all. The same logic can apply anywhere the cameras watch people. A weapon in the crowd, a fight breaking out, a person climbing into a restricted area.

So, each of those is pretty rare and happens really fast, but if we miss it, it's pretty expensive.

Definitely. I help, I have heard from our enterprise customers as well. You know, when there are safety, time-sensitive events like a knife attack or someone falling down, you definitely want to make sure that you catch those events as they happen, so that you can respond quickly.

These demos and benchmarks are great in theory, but in practical use, our enterprise customers have always looked for on-premise deployment of the Reka Edge model or other faster models.

So, it doesn't really matter if your model can only run on the data center, because sensitive camera footage needs to always stay within the premises, or stay within the country.

So, what is the trade-off to getting the model small and running efficiently?

Oh, sure. So, that's why we try to build our models where cameras run. Our Reka Edge model has 7 billion parameters, and we have made its visual encoder pretty efficient.

For example, we produce only 64 tokens per image tile, which is only about one third of what existing models produce. That's why we can process up to 6 images per second.

And in order to achieve more efficiency, we have quantized our model to the 4-bit regime, so the memory footprint shrinks from 13 gigabytes to only 5 gigabytes.

So, you can run your model on an NVIDIA Jetson, Apple Silicon, or even Snapdragon phone.

That's great. This efficiency gain is really important, because if you are deploying the Reka Edge model on a large-scale deployment, like at the train station, they have so many cameras, and we need to definitely be able to run and capture these dangerous events that happen in real time.

Well, this has been a really insightful conversation with Thong. Thank you for sharing.

So, yeah, I hope you and the viewers found this helpful. And our team at Reka is constantly publishing such kind of work, even now as we speak.

And yes, we encourage you to join our Discord link below, subscribe to our YouTube channel, or follow us along on the Reka via the links below.

Thank you for joining us. Bye-bye.