ExtractBench Benchmarks Schema-Guided Extraction for Enterprise Documents

Introduction

Enterprise document extraction is moving beyond OCR pipelines and fixed templates. Increasingly, teams expect agents to read a document, follow a user-defined schema, produce structured outputs, and attach evidence showing where each answer came from. ExtractBench is designed to evaluate exactly this kind of schema-guided extraction in realistic business settings.

Key Points

• A benchmark built around real workflow requirements: The task is not just to answer a question or extract one field. A system receives a document and a schema, then returns the requested structure with grounding metadata. This better reflects how enterprises handle invoices, forms, lists, contracts, and operational records.
• Broad enterprise coverage: The evaluation set includes 4,869 pages from 370 enterprise documents, spanning 8 business domains and 67 document types. The documents are tagged by challenge scenario, making it easier to see where systems fail, such as on long documents, tables, lists, or forms.
• More than value accuracy: ExtractBench reports order-insensitive value F1 to measure whether extracted values are correct. It also includes word-level and page-level F1 to evaluate source traceability. Record completeness and measured cost are considered alongside these metrics, which is important for production decisions.
• Scalable ground-truth construction: The authors combine independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. This approach is meant to balance scale with reliability.
• Clear differences between system types: Commercial VLMs perform well on short documents, but they often truncate record lists when documents become long. Coding agents maintain higher accuracy, yet at much higher cost. LlamaExtract Agentic Plus ranks first across the three main metrics, with accuracy comparable to coding agents at a fraction of the cost.

Why It Matters

ExtractBench is useful because it shifts enterprise extraction evaluation from isolated demos to measurable trade-offs. In production, a system must not only return correct values; it must also avoid missing records, provide verifiable evidence, and stay within cost constraints. These are the factors that determine whether a document agent can be trusted in workflows such as compliance review, expense processing, procurement, or contract operations.

The benchmark also highlights a practical failure mode: long record lists are difficult for many VLM-based systems. That suggests the next generation of document agents will need stronger context management, iterative reading strategies, output validation, and record-level self-checking. For enterprise AI teams, ExtractBench offers a more grounded way to compare tools before deployment.

Source: Hugging Face Daily Papers