Selective Risk Control for Document Extraction Has a Validity Problem

Introduction

For document extraction, the useful promise is not simply that a model is accurate on average. It is that the system accepts only fields it can trust, sends the rest to review, and keeps the error rate among accepted fields below a budget α. This is the practical meaning of selective risk control. A paper featured in Hugging Face Daily Papers argues that the usual confidence-thresholding recipe can silently break that promise on real documents.

The evaluation uses 13,859 genuine fields from 800 CORD receipts. Only 49.0% of the fields are correct, making the setting substantially harder than a polished benchmark headline might suggest. The contribution is not a new conformal-prediction theory. Instead, it is a careful diagnosis of where familiar calibration procedures fail when their statistical assumptions meet structured documents.

Three failure modes

• Document clustering. Fields from one receipt share layout, image quality, and failure causes. They are therefore not interchangeable observations. The measured design effect is 1.84–2.45, so the effective calibration sample can be much smaller than the raw field count suggests.
• Score-refit leakage. If a high-capacity score and its acceptance threshold are fitted on the same fields, score optimism can flow directly into threshold selection. Separating score fitting from calibration helps, but does not by itself solve document-level dependence.
• Tie-mass pathology. A score with only a few distinct values produces a collapsed threshold grid. In the reported example, coverage changes from 0.030 to 0.001, leaving little room for smooth risk-coverage tradeoffs.

A ladder of validity

The study distinguishes four levels of claims. The first two control expected selective risk, not the probability that every repeated split will respect α. With a fit/validation split, a learned fusion reaches 0.318 coverage at 0.096 risk for nominal α=0.10. Yet its realized risk exceeds the target in 47.5% of resplits. This is an average-controlled operating point, not a certificate.

The third level uses Mondrian Learn-then-Test with exact binomial tails to provide group-wise PAC certificates. Under a field-iid calculation, coverage is 0.171 at risk 0.068; cluster correction reduces coverage to 0.140. The fourth level uses a document-iid assumption, which better matches how documents are sampled: coverage falls to 0.060 at risk 0.020, and 19 of 40 splits certify nothing. It is the most honest tier for the data structure, but is nearly powerless with only 800 documents.

When conditioning pays

More conditioning is not automatically safer. With a learned score, relevant covariates can already be used inside the score. Additional Mondrian partitioning then fragments the threshold sample and reduces coverage. With a weak or frozen score, covariates can instead define a useful taxonomy, especially when a pooled threshold cannot obtain a rigorous certificate. On the Sonnet CORD capture, the prespecified support-bin taxonomy reaches 0.171 coverage at a rigorous tier, versus roughly 0.091–0.098 for pooled thresholds.

The result has a clear boundary. On the same documents, the provenance advantage disappears with haiku or Qwen2.5-14B; field-type taxonomies become the practical winners instead. Conditioning is therefore not a universal recipe. Its value depends on how much signal the score already contains and whether the proposed groups reflect stable error differences.

Why it matters

The paper asks production teams to state exactly what “risk below 10%” means: an average operating expectation or a high-probability, document-level certificate. A frozen configuration was also run on selection-untouched captures for haiku and an open-weights Qwen model without a risk-contract failure, although coverage varied with signal quality. In a blind human review of 149 accepted fields, 2 were judged incorrect, or 1.3% selective risk. That supports the practical direction, but does not remove the need for document-aware sampling and uncertainty accounting.

The accompanying VerifyDoc artifact is released under Apache-2.0. For practitioners, the immediate checklist is straightforward: split by document, isolate score fitting from threshold calibration, inspect score ties, and report coverage, realized risk, and the type of guarantee separately. The harder open problem is obtaining powered document-level bounds that remain honest without becoming nearly vacuous at this sample size.

Source: Hugging Face Daily Papers