Static Protein Recovery Is Not Dynamic Stability: BILBY Raises the Evaluation Bar

Introduction

Protein design benchmarks often ask whether a generated scaffold folds into a structure that places a functional motif in the right geometry. That is a useful first test, but it treats the protein as if it were a frozen object. In reality, function can depend on conformational ensembles, local flexibility, and transitions between states. A design that looks correct in one predicted structure may therefore fail to preserve the relevant interactions over time.

The study titled Static Recovery Is Not Dynamic Stability addresses this gap through BILBY, a pair of resources designed to improve both benchmark coverage and the definition of success.

Two complementary resources

• BILBY-PRO contains 220 systematically derived motif-scaffolding cases from experimentally determined structures linked to PROSITE. The cases represent 174 PROSITE motif-pattern entries, providing a broader and more traceable alternative to small hand-curated motif collections.
• BILBY-MD is a reference-based, all-atom molecular dynamics evaluation framework. It compares generated designs with reference simulations rather than judging only a single static structure.

The framework examines motif drift, scaffold drift, preservation of motif contacts, and motif-dihedral free-energy landscapes. Together, these measurements ask whether the design remains physically consistent with the reference during motion, not merely whether its starting coordinates appear plausible.

Main findings

The authors evaluate six recent scaffold-generation models and observe different trade-offs between target coverage and unique-solution yield. RFDiffusion and Proteina solve more targets, while ESM3 generates more unique successful scaffolds on the targets it does solve. The result illustrates why a single leaderboard metric can hide important differences in model behavior.

The dynamics analysis is performed separately, on designs from the original MotifBench benchmark that had already passed static checks. Of 55 statically solved design–problem pairs, 31 satisfy the study’s MD-supported criterion, while only 8 pass all of the proposed checks. Compact, self-contained motifs tend to receive dynamic support more often. Interface-like motifs and motifs whose behavior depends heavily on surrounding context fail more frequently in simulation.

Why it matters

The work does not make static structure prediction irrelevant. Instead, it positions static recovery as an initial filter rather than a complete definition of success. Future evaluations of generative protein-design systems may need to report target coverage, solution diversity, local geometric accuracy, contact preservation, and fidelity to conformational ensembles together.

For researchers, BILBY-PRO offers a way to reduce dependence on a small number of manually selected examples, while BILBY-MD provides a reusable route for testing whether apparently successful designs remain stable under physically meaningful motion. The broader lesson is straightforward: a protein design should not be called functional merely because one structure looks right. When activity depends on flexibility or transitions, the gap between static recovery and dynamic stability may be decisive.

Source: Hugging Face Daily Papers