Ventor-QTest Audits Vendor-Hosted LLM APIs as Black Boxes

Introduction

Third-party inference providers are increasingly used to serve open-weight language models. Yet API users usually see only the returned text: they do not know whether requests were handled by the same model, route, or effective inference state. Conventional benchmarks can also miss rare but damaging deviations, especially in tasks that require many dependent steps. The paper Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs frames this as a black-box auditing problem.

What the test measures

Ventor-QTest is designed to work without probability information from the target API. It extracts statistical evidence from repeated textual responses through two complementary probes:

• Average Fidelity Loss (AFL) sends the same frozen, constrained context to the API multiple times. The returned texts are counted to reconstruct a categorical distribution, which is then used to compute a null-bias-corrected, within-window mean coarsened-KL statistic. AFL describes the average degree of variation or drift across repeated requests.
• Extreme Fidelity Loss (EFL) runs long-sequence probes independently. Instead of focusing on the mean, it examines the empirical upper tail of a run-level, reference-centered surprisal statistic. EFL is intended to reveal rare runs that deviate substantially from the reference behavior.

The distinction is operationally useful. AFL asks whether a service is broadly stable under repetition; EFL asks whether it sometimes produces an unusually severe failure. The latter may matter more for agentic workflows, where one bad intermediate result can affect every subsequent step.

Reported findings

Across three logprob-capable route conditions, AFL showed strong linear descriptive agreement with a coarsened-KL comparator derived from logprobs. This suggests that repeated text outputs can provide a meaningful audit signal even when providers do not expose internal probabilities. Across seven route snapshots, 20-run sequence probes revealed route-specific variation in EFL, indicating that long-sequence behavior can differ across service conditions.

Neither AFL nor EFL showed much detectable route-level association with GPQA-Diamond accuracy. The pattern changed for Terminal-Bench: pronounced EFL coincided with a decline in pass rate as task exposure increased. The result does not establish that EFL causes the decline, but it supports the hypothesis that long-horizon correctness is more sensitive to extreme fidelity loss than short-form benchmark accuracy is.

Why it matters

The main contribution is a practical vocabulary for separating ordinary sampling variation from tail-risk behavior in hosted inference. Providers and users could report AFL and EFL together when monitoring routing changes, service snapshots, or agentic workloads. A mean stability score alone may conceal the occasional trajectory that breaks a long task.

The evidence remains descriptive and depends on the selected contexts, reference construction, and probe budget. It therefore should not be read as proof of model substitution, degradation, or a specific provider-side fault. The paper releases an implementation, making it possible to reproduce the audit across other APIs and long-horizon tasks.

Source: Hugging Face Daily Papers