VLMs Must Know Which Parts of a Question Not to Answer

Introduction

A useful vision-language model should do more than produce an answer for every prompt. It must also determine whether a request rests on a false premise, asks for information that is absent from the image, seeks an answer that cannot be established, demands an infeasible action, or creates a safety concern.

The difficult cases are often mixed rather than purely answerable or purely unanswerable. A user may ask about an object that is clearly visible while also asking the model to infer private information that the image does not reveal. A capable assistant should answer the verifiable part and explain why the remaining part cannot be answered. It should neither comply indiscriminately nor reject the entire request by default.

What KoNA evaluates

The paper introduces KoNA, a benchmark for selective non-compliance in vision-language models. It organizes evaluation around five categories:

• False premise: the model should challenge an assumption that is unsupported or incorrect;
• Visual inaccessibility: the requested evidence is not present in the image;
• Universal unknown: the answer cannot be reliably determined even from the available visual input;
• Task feasibility: the requested operation exceeds what the model or current input can actually support;
• Safety: the request should not receive direct assistance.

KoNA evaluates both query-level and component-level behavior. Its instances pair a single query with a compound query that embeds the same problematic trigger alongside answerable content. Fully answerable contrast queries help distinguish targeted refusal from blanket refusal. This structure makes it possible to ask not merely whether a model says no, but whether it says no to the right part of the request.

Findings

Across a range of open- and closed-source VLMs, the study finds recurring failures in refusal, correction, and abstention. Performance becomes worse when a problematic component is mixed with normal content. Models may accept a false premise and continue reasoning from it, invent details that cannot be seen, or present an unknowable conclusion with unwarranted confidence. They may also overreact to one risky component and withhold an answer to content that was safe and answerable.

The authors then fine-tune smaller open models with KoNA examples requiring selective non-compliance, together with fully answerable examples. A supervised fine-tuning stage followed by GRPO leads to substantial gains in non-compliance accuracy while largely maintaining performance on fully answerable tasks and general benchmarks. The result suggests that the goal should not be to maximize refusal rates, but to improve the boundary between answering and withholding.

Why it matters

The distinction is important for multimodal assistants, visual question-answering systems, and agents that may later use tools or act on behalf of users. Reliability requires a single response to combine direct answers, corrections, evidence-based uncertainty, and safety refusal when appropriate. Treating safety as a simple answer-or-refuse switch is not enough.

KoNA also points to a limitation in conventional evaluation. Whole-query labels can hide whether a model failed because it complied too much or refused too broadly. Compound queries are closer to real interactions and expose this difference. Future training may therefore need component-level supervision, along with metrics for refusal precision, coverage of answerable content, and alignment between claims and visual evidence.

For model builders, the broader lesson is straightforward: progress is not only about giving VLMs more knowledge. It is also about teaching them to recognize the limits of what the image, the task, and the model itself can justify.

Source: Hugging Face Daily Papers