FlavourBench Reframes LLM Evaluation with Executable Culinary Ground Truth

Introduction

Many language-model benchmarks inherit the weaknesses of their judge. Human panels can be expensive and subjective, another model may introduce a second system’s preferences, and exact-match keys are often too rigid for open-ended answers. FlavourBench explores a different design: convert a domain task into a finite choice problem and let a versioned, executable culinary system provide the reference signal.

How the benchmark works

Each prompt presents eight ingredients and asks a model to select a three-ingredient portfolio. There are 56 possible portfolios for each prompt. Before model execution, Epicure assigns scores to all of them, creating a frozen map against which the response can be evaluated. The result is a dense task-level signal rather than a binary match or an improvised judgment after the fact.

The core set contains 534 tasks spanning substitution, pairing, and constrained composition. The study evaluates 27 frontier endpoints. To reduce leaderboard distortion from uneven response failures, every ranked model has exactly 89 valid responses for each panel and family. Across the evaluation, that produces 14,418 model-task cells with a common validity structure. The final FlavourBench Score is defined as the equal-family mean of the frozen task scores.

The paper also places unusual emphasis on uncertainty and reproducibility. Two independently compiled panels show a Pearson correlation of 0.89 and a rank correlation of 0.80. The authors use 50,000 anchor-cluster bootstrap replicates for simultaneous 95% score bands. For the 351 paired model contrasts, they run 100,000 sign-flip draws and apply Holm correction to account for multiple testing.

Results and implications

Grok 4.6 records the largest point estimate, 65.1, with a simultaneous 95% confidence interval of 61.0 to 69.2. Yet the result should not be read as a declaration that every neighboring model is meaningfully different. Only 101 of the 351 pairwise comparisons are resolved by the reported statistical procedure. This distinction matters: a leaderboard can impose a strict order even when the evidence supports ties across many pairs.

The broader contribution is methodological. FlavourBench shows how a domain-specific executable evaluator can make open-ended assessment more systematic without forcing every answer into a single reference string. Precomputing the candidate space, freezing the scoring map, balancing valid responses, and publishing the full reconstruction path all make the benchmark easier to audit.

There are also clear boundaries. A score reflects performance under Epicure’s culinary representation and task construction; it is not a direct measure of general reasoning or real-world cooking skill. Extending the approach to other domains would require a credible executable evaluator, a well-defined candidate space, and careful version control. FlavourBench releases the prompts, portfolio maps, raw responses, exact routes, content hashes, and an offline verifier, giving outside researchers the materials needed to reproduce and scrutinize the reported results.

Source: Hugging Face Daily Papers