Back to articles

NexSight Makes Long-Video Agents’ Evidence Search Measurable

3 min read

Long-video question answering is not simply a matter of recognizing what appears in a frame. A system may need to find brief clues scattered across hours of footage to count repeated actions, order events, or pinpoint when a state changed. Finding one clip that supports an answer is not enough to establish a total count: the agent must also check for missed instances and avoid counting the same event twice. NexSight sets out to make this evidence-search process measurable and trainable.

The work has three main components:

  • SightForge constructs tasks. Rather than treating a draft answer and a few supporting clips as sufficient, it requires a verifier who is blind to the draft answer to reconstruct the task from evidence densely reacquired from the raw video. Counterexample checks must also fail to refute the task.
  • SightLine is the resulting benchmark: 310 evidence-traceable tasks drawn from 108 long videos, designed to test fine-grained temporal understanding.
  • SightLoop provides a shared interactive evaluation harness. Agents can explore the timeline, inspect local segments, integrate evidence, and revisit earlier moments; the harness records the observations returned to them.

This setup targets a blind spot in answer-only scoring. A correct final answer does not reveal where a model looked, whether its evidence was sufficient, or whether it checked information that could overturn its conclusion. Those distinctions matter especially for counting and timeline questions. One clip may establish that an action occurred, but not that it occurred only that many times. Establishing event order likewise requires checking the relevant moments against the broader timeline.

The strongest model reaches 54.52% accuracy on SightLine, compared with 79.35% when it is given annotated evidence. That gap suggests that interpreting evidence already in hand is not the only challenge: choosing where to look and when to revisit also matter. The researchers turn solver trajectories on verified tasks into step-level supervision for observation decisions, using videos disjoint from the evaluation set. They report that with fewer than 6,000 training examples, Sight-35B improves interactive accuracy from 21.29% to 42.26%.

NexSight’s contribution is to bring the search process into the evaluation of long-video agents, rather than judging only their final answers. That makes it easier to distinguish failures of visual understanding from failures to find, cover, or recheck relevant moments. The gap between benchmark performance and performance with annotated evidence also points to unfinished work: reliable long-video reasoning remains difficult. Progress will depend not only on higher answer scores, but on whether agents can consistently build complete evidence trails across videos and task types.

Source: Hugging Face Daily Papers

Comments

Checking sign-in status...

Loading comments...

Related articles