Large Language Models (LLMs) have become a mature technology that can serve multiple products. However, LLMs were never the endgame; they are just the intermediate step. Physical-world intelligence demands an architecture that unifies all possible inputs and outputs, where all the sensory signals, language, images, audio, or videos, can be processed by exactly the same architecture.
Even though the recent AI progress mainly meant training bigger models on more textual data, that direction isn’t sufficient if we consider the physical world, where the rules of motion and geometry need to be understood.
The gap between what LLMs offer and what’s needed to successfully operate in the real world begs the question of what the next model could look like. Therefore, we’re building “omni world models” that can simulate, and reason about the real world.
What “omni” actually means
Our omni model is a single architecture that handles many kinds of input, e.g., language, images or videos, and also produces many kinds of output, using one unified system rather than an orchestration of different agents. To operate with arbitrary known outputs, it has to handle both discrete (like the words in a sentence) and continuous signals (like smooth waveforms of a spoken sentence, or the pixel values in the image).[1] Therefore, our long-standing objective extends beyond generating images, video, and audio within a single architecture. Since an image can be interpreted as a single-frame video, and audio can be represented by a spectrogram (an image), a single diffusion-based model should be capable of handling these modalities.
Optimally modelling both types of the signal is the difficult part, and most existing architectures model discrete signals autoregressively or use diffusion-like approach to model continuous ones. While early hybrid models have shown that both approaches can co-exist, building a unified world model that is production-ready is the next frontier of AI. Getting such a model is what makes “omni” special.
How “omni” differs from the multimodal models
You might be asking then: don’t existing VLMs already handle images and language? Yes, but the key lies in multimodal outputs. VLMs typically transform input signals from various modalities such as images or language into a single output modality (usually language). Reasoning about visual data is what sets them apart from LLMs and enables applications such as OCR, healthcare or quality control. While VLMs can also be applied to robotics, their intuitive understanding of the real-world physics remains a potential constraint. Omni models generalises VLMs to work with potentially arbitrary output modalities, though our initial version specifically focuses on images, videos, language, and actions. In other words, while VLMs usually operate on language, image -> language, our omni model operates on language, image, video, actions -> language, image, video, actions.
Omni implies many new possibilities:
Unification of generation and understanding. Most existing approaches rely on agentic methods, where images are understood by one component, but they are generated by another component. Omni folds both into a single process, therefore avoiding costly “swapping” between the models, or cumbersome communication between both systems that have never been trained to live together.
Shared representations. Similar to VLMs, image, video or language tokens are all projected into the interconnected representation where the model can reason across all of them at once. With a rough analogy from biology: our brains do not separate “eye language” and “ear language.” Light hitting retina, and wave hitting cochlea both get converted into the same shared representation: electrochemical signals. Thus, perception is unified at the input side. What’s interesting is that generation may also work in reverse. Omni acts as a computational realization of functional integration, using a shared representation to connect diverse inputs and outputs; that is, one shared room, with many doors into that room, and many doors out of that room.
Handling continuous and discrete signals. Discrete signals represent language, or logic; both are at the core of communication and reasoning. On the other hand, continuous signals are closer to the physical world: audio, images, videos, or arm joint configurations. The ability to model both within a single model is creating a universal canvas where the model could potentially operate on all domains in science or engineering.
“Omni” differs from agentic AI
There is a fair objection to the omni approach. Can’t we achieve exactly the same by chaining different agents (either LLMs or VLMs or text-to-video generations)? The current trend of chaining different agents to operate on various modalities is a temporary patch rather than a proper solution. The clear drawback of the chaining is the incurred costs on swapping models: it’s slower as each model adds its own latency, it’s more expensive as each model has to be paid separately. Such approach is also arguably suboptimal as each agent was optimised for its own objective rather than the collaboration. A unified omni architecture structurally resolves these inefficiencies by processing multimodal reasoning within a single, shared representation instead of the API calls. Note that both concepts are also not mutually exclusive as agents can also run omni-world models internally.
Technically, an omni architecture may issue multiple model invocations internally depending on the task. However, because memory is shared at the hidden-state level, input context is processed exactly once without requiring cache refilling. As a result, token billing is amortized across outputs and zero hand-off latency is incurred between modalities.
Autonomous systems
Nowhere is this unification more important than in robotics, where a system needs to see, understand, imagine, plan, and physically act, often in real-time. Nowadays, a derivative of VLMs, so called vision-language-action (VLA) models, integrate perception and language understanding (as inputs) with robotic actions (as outputs); thus robots can perform actions in unstructured and nondeterministic environments rather than relying on rigid rules or a deterministic setting. As the VLM derivative, VLAs naturally offer reasoning over inputs.
World-action models (WAMs) offer an alternative paradigm to the methods mentioned above. Rather than mapping inputs directly into actions, WAMs use foundational video generation models to predict future dynamics by generating video sequences from an initial frame together with corresponding actions. In summary, while VLAs map visual inputs into actions, WAMs forecast the underlying physical evolution of what happens next.
Our omni-world model unifies VLA and WAM architectures within a single backbone; thus offering a generalized framework that integrates the core capabilities of both. It employs a shared interface – accepting images, arm joint configurations, and language instructions as inputs, while producing actions such as motor commands or joint configurations as outputs. Through this design, the model combines environment-simulation capabilities with reasoning and multimodal understanding.
Training on Videos. Unlike traditional VLAs, where first LLMs are fine-tuned to work with images (or individual video frames, e.g., at 1fps) and next tuned to output actions, ours is trained to represent whole videos. This means the model is exposed to the time component to better understand geometry, motion or physics. In contrast, many VLAs are mostly limited to high-level concepts present in videos such as recognising various categories (e.g., “walking with a dog” or “picking up an object”).
Generations. The model can extend the current state, (e.g. input image, instruction, joint configuration, etc) into plausible futures (next frames, joint configurations, etc). It also offers pixel-level decoding of the next frame that can be helpful in interpreting the results or using that as the inputs for other models such as an external critic. Generations can also be used to scale up training data, where long-tail, rare cases can be generated, e.g., navigating London streets during heavy blizzards or encountering highly unusual obstacles such as godzilla crossing the street (which could happen during Halloween). This ensures the physical model is more robust against out-of-distribution events. To maintain high-quality outputs, the model employs structured inputs, e.g. depth masks, which capture scene motion and geometry, and prevents the model from drifting into hallucination. With this approach, the model generates varied visuals of the same scene, akin to the digital cousin concept.
Simulations. Omni deviates from VLAs as it doesn’t just perceive and act but also predicts what the world will look like next. That’s essentially enabling robots to “imagine”, certainly an important feature of safer planning – predicting consequences of their own actions grounded in the observed world. Furthermore, our world model can serve as an engine for building simulated environments to train agents inside world models and contributing to the advancements of simulation 2.0 or digital nomads.
Planning. By simulating the future followed by judging by outcomes, the model can “mentally rehearse” an action and see how the world would respond, before committing to it physically. The same ability of constructing hypothetical situations in one’s imagination, prior to acting, is also a crucial component in human planning where “imagination” is the core component of past and future thinking. Together it creates a closed loop: Observe -> Simulate -> Criticise -> Act. With parallel simulations the model can find actions and plans that are safest and most optimal. By construction, the same model can be both a critic and a simulator, thus avoiding chaining two different agents together and benefiting from the same shared representation and reduced latency.
Beyond robotics
While our primary objective remains the development of foundational architectures for physical AGI - with robotics representing the ultimate domain for omni systems that unify perception, reasoning, planning and physical act; all of which have to work together in real time - the versatile nature of this unified interface allows for much broader implementation across diverse modalities. Much like the trajectory of LLMs that first started as simple chatbots, but later became the universal engine capable of solving complex problems expressed in the digital domain.
In the immediate future, these architectures will likely serve as a sophisticated extension of existing agents, providing a vastly more versatile interface for diverse outputs. Another opportunity lies in the creative landscape of media and entertainment. Here, such systems can act as a powerful utility for artists, enabling rapid iteration of concepts or serving as a convenient tool for digital reshooting.
Looking further ahead, beyond robotics, real-time omni models open up compelling long-term opportunities across interactive media, smart wearables, edge computing, and user interfaces, which we briefly describe below:
Interactive Media & Storytelling: On-the-fly generation can lay the groundwork for next-generation entertainment, merging classic storytelling with real-time interactivity generated frame-by-frame as the story unfolds
Wearable Tech & AR: Integrating these models into devices like smart glasses enables instant environmental reasoning and real-time augmented reality visuals.
Edge Deployment: Unified omni world models avoid the latency and heavy computational overhead of multi-agent chaining, which is a necessary prerequisite for real-time intelligence. However, to meet the strict hardware and memory constraints, edge models will rather rely on distilled variants of the base model.
Next-Generation Operating Systems: By combining inputs, internal computation, and rendering into a single framework, future operating systems could interpret gestures, speech, and visual context natively, and communicate with us using the same modalities. Incidentally, by fostering more natural human interaction, these systems promise a far more intuitive and richer user experience.
The versatile nature of the omni unified interface allows for much broader implementation, as the same shared representation that plans robotic arm movements can equally well interpret gestures or synthesize high-fidelity media. However, even though the base model is a generalist, different applications may require specialised models developed in the customisation phase of post-training.
The catch: building omni-world models isn’t easy
Building frontier models from scratch is very challenging. It requires the combination of a large volume of data, top talent, and computation. This is why Moonvalley and Reka have consolidated their individual talents in VLMs and text-to-video generation to form Labs, and together are building their first instance of the omni-world model that can simulate, reason, and predict actions. This way it could be seen as the extension of both World Models and vision-language-action models (VLAs). The joint entity is also what differentiates us from other startups.
Our model is trained from scratch, and we are expecting to release the first prototype in the next two months, just after around eight months of working together.
Data. Training a model that is good at generation and understanding requires data that satisfies both jobs at once. Large volumes of pre-training data, mid-size volumes of higher quality mid-training data focused on egocentric videos, and small-size volumes of highest quality data targeted to specific use cases. This requires building pipelines, processing data at scale, and sourcing videos accordingly. A sample of our dataset was released publicly.
Architecture. Building a versatile system that can efficiently be trained on understanding and generation tasks, where none of the modality is degrading any other one, is an open research problem. That is, there is no settled consensus on how that architecture would look like, and requires experimentation from our side. Training models with different objectives may also cause training instabilities. We address that by bringing together experts in generation and understanding; and by building benchmarks that monitor our progress.
World consistency. In the physical world when you rotate by 360 degrees, you will end up in the same spot. But this may not be the case with the text-to-video models, where the model always keeps generating new content. Keeping long consistency is also challenging for video generations as it requires operating on a long context.
Real-time generation and understanding. Interacting with the environment requires the system to operate in the low-latency regime. Here, the trade-off between speed and potency becomes more clear. Models that are better at generation and understanding tend to be bigger and operate on the larger number of tokens, thus are slower. Finding the right balance is an open problem.
Intuitive physics. LLMs can be very strong on formal physics or mathematics but the same models may struggle in intuitive understanding of physics required to successfully operate in the physical world. Building models with strong foundations in understanding physics the same way as we do is the next challenge in building omni-world models.
The bigger picture
It’s tempting to treat the current wave of agents that rely either on LLMs or VLMs as the destination of the AI story. However, they are closer to the mid-point. It’s a mature technology that operates in language or symbolic space that ignited the most recent revolution, but they struggle with motion and intuitive physics, and are limited by the outputs. In this spirit, our omni-world model continues that revolution by building models with more generic interfaces and broader capabilities. Furthermore, by imagining the real-world impact of its own actions, the model offers a better foundation for more tractable alignment with our values.
The analogy to brains also holds water, where there is a single system that can see, hear, touch, speak, write, and act in the physical world; all draw from the shared and interconnected representation. Getting the architecture right, scaling that up on more data, will take some time. But the direction is telling: “one mind, many senses” that operates in the physical world rather than “one mind per modality” focused on the digital domain only.
This points toward a future where one unified model observes a messy room, grasps physical dynamics of falling objects, and directly executes physical interventions avoiding any reliance on isolated components dedicated to perception, reasoning, or physical motion.
