Benchmark Radar Turns AI Benchmark Research into a Traceable Workflow

Introduction

Finding the right benchmark can be harder than running it. Relevant information is scattered across research papers, code repositories, dataset pages, model cards, and technical reports. A benchmark name alone is rarely enough: researchers may also need to verify the dataset version, implementation, prompting setup, evaluation protocol, and origin of a reported number. Benchmark Radar addresses this discovery problem with a living, searchable catalog that keeps links to the evidence behind each record.

What the project provides

• Broad evaluation coverage. The catalog targets LLM evaluation as well as agentic and tool-use benchmarks, coding, reasoning, safety, and domain-specific tests.
• Daily discovery. The system collects signals from 37 public sources every day, including 13 direct connectors and 24 first-party research or engineering feeds. The inputs include benchmark papers, repositories, datasets, and releases.
• Evidence-aware retrieval. Benchmark entries are connected to mentions in model cards and technical reports. Source identities and citations are retained so users can inspect the underlying material instead of treating an extracted score as an isolated fact.
• History and trend analysis. The current catalog contains 1,283 source records and 12,916 numeric observations across 790 records. The dashboard offers a leaderboard, score histories, saturation and trend views, and a Pareto frontier comparing scores with measured use.
• Multiple research workflows. Users can search the web dashboard, download evidence, or query the collection offline through a command-line interface.

Why this matters

As the number of AI benchmarks grows, the central question is no longer simply which model has the highest score. Scores may reflect different dataset versions, prompts, sample filters, tool configurations, or evaluation protocols. Two numbers with the same benchmark label may therefore carry different meanings. Benchmark Radar does not remove that comparability problem, but it can make the relevant context easier to locate.

The paper includes a worked prior-art search showing how to query the catalog and inspect benchmark evidence when designing a new evaluation. This is particularly useful for researchers deciding whether a proposed benchmark addresses an under-measured capability or merely duplicates an already crowded area. Adoption history and saturation views can provide additional context before a new test is introduced.

A tool, not an automatic judge

A searchable database should not be confused with a definitive ranking of benchmark quality. Records may come from sources with different reporting standards, and measured use is not the same as scientific validity. Likewise, a leaderboard can support discovery without proving that its entries were evaluated under equivalent conditions. Researchers still need to return to the original papers, code, and datasets to verify the protocol and interpret the numbers.

Benchmark Radar’s broader contribution is organizational. It turns a fragmented web-search task into a structured evidence-retrieval workflow. If its source coverage and update practices remain reliable, the project could help benchmark researchers spend less time locating scattered references and more time analyzing whether an evaluation is appropriate. In an increasingly saturated benchmark landscape, knowing what was measured matters—but knowing how it was measured and where the evidence came from matters just as much.

Source: Hugging Face Daily Papers