How Can Agentic Capability Be Evaluated Before Post-Training?

Introduction

Agentic capability is not the same as being able to produce a plausible answer. A useful coding or research agent must interpret a goal, operate within an environment, use available tools, maintain progress, test its work, and respond to failure. These properties are difficult to observe during pretraining, when a base language model has not yet been shaped into a chat assistant or explicitly trained to follow agent protocols.

The paper introduces ASAP, or Agent Scaffold Adaptation for Pretraining Evaluation, as a lightweight way to study this problem. Its goal is to evaluate pretrained models in end-to-end tasks while avoiding an assumption that they already behave like post-trained conversational agents.

What ASAP changes

Rather than redesigning a base model around a chat interface, ASAP keeps the completion-style interface and adapts existing agent scaffolds. The framework relies on three broad ideas:

• Structured trajectory continuation: the model continues from an organized partial trajectory, giving it an entry point into an ongoing task instead of demanding a complete agent response from an empty prompt.
• Short reference prefixes: concise examples or prefixes provide enough information about the expected interaction pattern without requiring a full conversion to an instruction-following system.
• Minimal recovery mechanisms: limited recovery support keeps evaluation running when an output does not immediately fit the execution protocol, while attempting to preserve the model’s own behavior as the object of study.

The authors apply the approach to software-engineering and deep-research benchmarks, comparing base models and checkpoints from different points in pretraining. According to the supplied abstract, ASAP produces discriminative end-to-end results across these models and checkpoints. The framework also records the trajectories generated during execution rather than reducing every run to a single final score.

Why behavior traces matter

Conventional pretraining metrics, such as likelihood-based measurements or static task accuracy, can reveal important information about language modeling. They are less informative about how a model behaves when a task unfolds over multiple steps. Does it explore the environment? Does it edit an attempted solution? Does it test the result? When something fails, does it recover or simply continue along a broken path?

Trajectory-level inspection makes these questions visible. Two models may receive similar final outcomes while following very different paths. One may explore deliberately and test intermediate results; another may make a short sequence of unsupported actions. Likewise, a failed run can be separated into different failure modes, including weak exploration, poor editing, missing verification, or ineffective recovery. This gives researchers a behavioral lens on how agent-relevant capabilities develop across pretraining stages and how execution patterns change after post-training.

Implications and open questions

ASAP does not claim that end-to-end agent evaluation should replace standard pretraining benchmarks. Its stronger claim is complementary: interactive tasks can expose capabilities that static scores and likelihood proxies leave hidden. For model developers, this may offer an earlier signal of whether a base model is ready for agentic post-training and a way to compare behavior before and after alignment.

The abstract does not provide detailed numerical results or isolate the contribution of every scaffold and recovery choice. Therefore, ASAP is best understood as a practical diagnostic framework rather than a complete definition of agentic intelligence. Its contribution is to move evaluation from asking whether a model predicts the right text to examining how it acts, adapts, and completes a task.

Source: Hugging Face Daily Papers