JoyAI-VL-Interaction

Introduction

The most important moments rarely wait for you to ask. A pot boils over while your hands are full. A toddler wanders toward the stove. The best moment of the game is gone before you can react. By the time you'd think to ask an AI, the moment has already passed, because the real world doesn't pause.

Today's AI can't help with moments like these, and it isn't a matter of speed. These models are turn-based by design: they sit quietly until you address them, then answer the question you asked. Even the video-call features in today's apps are question-and-answer underneath, reacting only when polled or asked. They were built for conversation, not for being present in a world that keeps moving.

We think the next step is a model that's present like a person: one that watches what's happening now, decides on its own when a moment is worth a word, speaks up when it matters and stays quiet when it doesn't, and hands off to a stronger model when a problem is hard. Thinking Machines Lab recently named this an interaction model. We believe it's the right direction, and we wanted to make it something anyone can build on.

So today we're releasing JoyAI-VL-Interaction: an 8B-scale, vision-first interaction model, released together with its training recipe, its data, and a complete, deployable system, all fully OPEN. Point a webcam or a livestream at it and it's immediately present in the scene, watching and responding in real time. Because the model is compact and the system runs on standard infrastructure, anyone can stand up their own always-present assistant from a single repository.

In head-to-head tests against the in-app video-call assistants of Doubao and Gemini, across 58 recorded visual-interaction cases from live commentary and monitoring to real-time memory, human raters preferred JoyAI-VL-Interaction by a wide margin on both what it says and when it says it. And along the way, the same recipe gave rise to abilities we never trained for, like guiding a shopper through changing app screens or improvising a lecture from a slide deck.

Our goal is simple: to take what interaction models make possible and put it in everyone's hands, helping move multimodal AI from turn-based dialogue toward genuine, real-time presence, openly and together. Here's what that looks like.

Capabilities

Once interactivity is trained into the model itself, rather than bolted on by an external harness, a whole class of capabilities comes naturally. These are exactly the things a turn-based assistant can't do well, however fast it answers: being present, acting at the right moment, sensing time, and remembering across a long stream. Here are nine of them, each a natural advantage of building interaction into the model. And for every demo below, we include real screen recordings of Doubao's and Gemini's video-call assistants alongside ours, so the difference in interaction style between an interaction model and a turn-based one is plain to see.

Click the buttons below to access the corresponding demos.

Our Approach

At the core of JoyAI-VL-Interaction is one decision the model makes on its own, every second: speak, stay silent, or delegate. We build it on our visual-language instruct model, JoyAI-VL-8B, and keep speech as pluggable input and output rather than fusing it into the model, so the model's only job is to watch and judge the right moment to act. To stay real-time over long streams, a predictive video codec (AdaCodec) spends only a handful of tokens on each predictable frame and saves full detail for the moments the scene actually changes, so the token budget grows slowly instead of with every frame. The behavior is learned rather than scripted: we train the model on more than four million time-aligned clips labeled second by second for when to speak, stay silent, or delegate, and refine it with reinforcement learning. We release the data and the full recipe so the result can be reproduced and extended.

Around this model we build a complete, deployable system so it works out of the box. The model is the only part that decides when to act; everything else is a pluggable component arranged around it: streaming ASR and TTS for speech, a long-horizon memory that keeps useful detail across hours, a visualization UI, and a bridge that lets the model hand hard subtasks to any background model, API, or agent and fold the answer back while it keeps watching. The whole stack runs on standard vLLM infrastructure to stay real-time over long sessions, and any component can be swapped for a deployment's own without rebuilding the rest.

We are open-sourcing all of it, the 8B model, its training recipe, the data, and the deployable system, so anyone can stand up a real-time, always-present assistant and carry the interaction-model direction forward in the open. We expect the full release to be complete by June 20, 2026, at https://github.com/jd-opensource/JoyAI-VL-Interaction.

Evaluation

We evaluate the model in 58 real, event-driven visual interaction settings. Each item is recorded as a live video interaction with JoyAI-VL-Interaction and the corresponding in-app video-call assistant, then judged pairwise by human raters for both response quality and timing. This keeps the evaluation close to the product setting we care about: whether the assistant says the right thing at the right moment while the scene is actually moving.

Each aspect corresponds to a scenario category in the evaluation ledger. The memory category here is a minute-scale visual recall setting, with cases mostly spanning several minutes to a little over ten minutes. The tables report the percentage of pairwise comparisons where human raters preferred JoyAI-VL-Interaction, found a tie, or preferred the baseline.

In the final aggregate, JoyAI-VL-Interaction wins 77.6% of pairwise comparisons against Doubao and 87.9% against Gemini, with ties included separately in the tables below. Its strongest margins fall on the most time-critical settings: it wins every comparison in monitoring and alerting against both systems and never loses one in real-time translation or counting, exactly the event-driven, act-at-the-right-moment tasks that turn-based products structurally miss.

Conclusion

Limitations We want to be upfront about scale. The video-call assistants we compare against, Doubao and Gemini, are backed by far larger models and polished through years of product iteration against real users; they are comprehensive, broadly knowledgeable, and hard to beat on open-ended chat, personal style, and the long tail of everyday requests. JoyAI-VL-Interaction is a compact 8B model, and we don't claim to match them everywhere. What we have done is pry open a door: in the advantage zone of a vision-language interaction model, real-time presence, vision-triggered proactivity, and a sense of time across a stream, a far smaller open model already comes out ahead. That a compact, open model can do this against large, heavily optimized products is exactly why we're excited to put this work in front of the community.

What's next, and an invitation And we think this is only the beginning. The interaction data we trained on is still small, yet even this much was enough for capabilities we never explicitly taught, like guiding a shopper through changing app screens, to emerge on their own; we're convinced the headroom is large, and that scaling this kind of time-aligned data, together with the recipe and the system, will take the model much further. The moment we are reaching for is an everyday one: you come home worn out after a long day, and before you have said a word, a quiet voice notices and offers, "I can see you're tired; today must have been hard on you." Presence like that, given unasked, is what an interaction model makes possible and a turn-based one, waiting to be addressed, never can. We have released the whole stack openly, the 8B model, the time-aligned data, the training recipe, and the deployable system, to lower the barrier for everyone working in this direction. We'd love for you to explore, with us, what a model that is truly present in the world can become.

Authors

Citation

@techreport{joyai2026vlinteraction,
  title        = {JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence},
  author       = {{Video Understanding Team of JoyAI-VL @ Joy Future Academy, JD}},
  institution  = {Joy Future Academy, JD},
  year         = {2026},
  month        = {June}
}