Gemini’s Agentic Video Understanding Lets the Model Decide What to Watch

Long-form video analysis has always involved a trade-off. Sampling more frames can preserve detail, but it raises token usage and inference costs. Sampling less often is cheaper, yet it can miss a brief action, a subtle visual change or an important sentence. Google DeepMind’s new Agentic Video Understanding is designed to address that trade-off by changing how Gemini decides what to inspect.

From fixed sampling to active inspection

Conventional video processing generally feeds a model frames at a fixed rate. Gemini’s default is 1 frame per second, although developers can adjust the rate through the API. This approach is predictable, but it treats every part of a video similarly, even when only a small section is relevant to the user’s question. On long recordings, that can create high costs and still leave gaps around fast events.

With the agentic mode, Gemini can choose which time ranges to examine, how densely to sample them and which signal to prioritize: visual frames, audio or transcripts. It uses an internal video tool in an iterative loop. The model can first locate a likely section, then load it again at a higher frame rate or inspect another modality for confirmation. Developers could build similar pipelines themselves, but the new feature moves much of that orchestration into the model.

Reported performance and target workloads

Google says that, across standard video-understanding benchmarks, agentic processing can reduce token consumption by up to 88%, cut analysis costs by up to 66% and improve accuracy by up to 7%. These are upper-bound results from tested scenarios, not a guarantee for every video or prompt. The impact is expected to vary with the model, the length and structure of the video, and the task being evaluated.

The company highlights several use cases:

• Sub-second moment retrieval: Finding brief state changes, tight edit boundaries or short events for automated video editing.
• Long-form search: Answering questions about lectures, tutorials and multi-hour recordings without scanning every moment at the same level of detail.
• Anomaly detection: Resampling a suspicious time window at a higher frame rate to inspect subtle artifacts or unusual motion.
• Action and object counting: Tracking repeated physical movements or distinct objects more accurately over time.

Why it matters for developers

The main change is not simply lower token consumption. The feature makes the model’s observation strategy part of the reasoning process. A video is treated less like a single block of media and more like a time-indexed source that can be queried as needed. This could reduce the amount of custom code required for segmentation, adaptive sampling and coordination between visual, audio and transcript inputs.

Agentic Video Understanding is available through the Gemini API in Google AI Studio and the Gemini Enterprise Agent Platform. The launch covers Gemini 3.7 Flash, 3.6 Flash and 3.5 Flash-Lite. Developers can enable it by setting video processing to “agentic”; Google says it uses standard Gemini API token pricing without an additional feature fee.

Google also plans to bring the efficiency and quality improvements to the Gemini app and, in the coming months, to YouTube’s Ask YouTube feature on video watch pages. If the rollout performs as intended, video assistants could move beyond broad summaries and retrieve evidence tailored to a specific question. Still, benchmark gains should be interpreted carefully: real-world results will depend on video format, audio quality, question formulation and model selection.

Source: Google DeepMind