VideoJev Turns Video Understanding into a Reusable Decision Interface

Introduction

A video agent must do more than describe what it sees. It needs to identify when an event happens, answer targeted questions, track progress, and decide what to inspect or do next. Conventional multimodal large language models often handle these tasks through autoregressive text generation. That approach is flexible, but repeatedly generating answers for the same video can duplicate expensive visual computation. VideoJev takes a different route: it frames video understanding as a structured decision problem and scores candidate outputs directly.

How it works

VideoJev is built by adapting pretrained multimodal large language models. Instead of using the language decoder to generate an answer token by token, the model adds a shared head that measures how well temporal intervals and textual options match the evidence in a video.

For temporal grounding, the model scores candidate intervals and identifies the span corresponding to an event. For multiple-choice video question answering, it compares answer choices and selects the option best supported by the video. Both tasks therefore share the same basic interface: evaluate structured candidates against visual evidence. This preserves the semantic capabilities of an MLLM while producing outputs that an agent can consume directly.

The system also targets repeated interaction with the same visual history. Cached inference stores the visual computation already performed, allowing later queries to reuse it instead of processing the video from scratch. Joint inference addresses a different scenario: multiple questions can be evaluated in a single forward pass, which is useful for batch queries or parallel decision branches.

Reported results

The paper presents VideoJev-4B and VideoJev-35B-A3B. Across three temporal grounding benchmarks, the two variants improve mean mIoU over their respective Qwen backbones by 6.7 and 10.8 points. Their measured model forward passes are approximately 2.8 to 3.1 times faster. On four video QA benchmarks, both models achieve performance comparable to their corresponding Qwen Thinking models, while reported forward-pass speedups exceed 100 times.

These speed figures concern model forward computation rather than guaranteed end-to-end application latency. Factors such as video decoding, preprocessing, scheduling, and downstream actions can still affect the total response time. Even so, the results suggest that direct scoring can be substantially more efficient than repeated autoregressive decoding when the task has a structured answer space.

Why it matters

VideoJev offers an interface that fits the way visual agents actually operate. An agent may revisit the same recording many times, ask new questions after receiving an intermediate result, or evaluate several possible events in parallel. Reusing visual computation can make such workflows more practical, while interval and option scores are easier for planners and tools to consume than unconstrained text.

The current evidence focuses on temporal grounding and multiple-choice QA, so it does not establish that structured scoring can replace open-ended generation in every video task. Still, VideoJev illustrates a useful System One direction for video: reserve expensive reasoning for the decisions that matter, and return compact, composable judgments that can be combined into longer workflows.

Source: Hugging Face Daily Papers