CaSKG Calibrates Skill Graphs for More Reliable Agent Retrieval

Introduction

For an LLM agent, learning how to act is not simply a matter of storing successful examples. When a new task arrives, the agent must retrieve the relevant skills and compose them in an executable order. Larger skill libraries improve coverage, but they also increase retrieval difficulty and context consumption. CaSKG, short for Counterfactual-Causal Skill Graphs, targets this bottleneck by asking whether a relationship between two skills is procedurally useful, rather than merely similar in wording.

How the framework works

Several common retrieval strategies expose different trade-offs. Prompting an entire library preserves coverage at a high context cost. Vector retrieval is more compact, but generally treats skills as independent pieces of text. Graph retrieval can restore workflow context, yet its value depends on whether the graph edges are reliable. CaSKG therefore makes edge-confidence calibration the central part of its design:

• Build a high-recall candidate graph. The system combines semantic and lexical evidence with skill input/output compatibility and structural signals to propose directed relationships. Repair evidence and an optional LLM judge can further refine the candidate scores.
• Test the relationships counterfactually. For a pair of skills, the method creates direction-conditioned textual probes that remove one skill, substitute it, or reorder the pair. The resulting evidence is aggregated with Bayesian smoothing to estimate how strongly one skill supports another.
• Publish a state-filtered weighted graph. The calibrated graph is built offline. At runtime, the agent expands retrieval according to the current task and state, without changing the downstream policy or task interface.

The distinction is important. A pair of skills may appear related because they mention the same objects or actions, while still failing to represent a useful execution dependency. By testing omissions, substitutions, and order changes, CaSKG attempts to identify relations that preserve prerequisites, state-changing actions, verification routines, and completion steps.

Results and implications

The paper evaluates the framework on ALFWorld ID-140 and ScienceWorld U211 with six LLM backbones. The authors report that CaSKG achieved the highest task score in all twelve model-and-benchmark combinations. Against Graph-of-Skills, its six-model macro-average ScienceWorld score increased from 72.62 to 80.50, while ALFWorld success rose from 80.01% to 86.79%. Mean environment steps were also reduced on both benchmarks. Qualitative and ablation analyses further indicate that calibrated edges helped retrieval retain important parts of multi-step procedures.

The broader contribution is to treat the quality of connections between skills as a first-class retrieval problem. In household tasks, scientific procedures, and tool-use workflows, missing a prerequisite or reversing two actions can make an otherwise relevant retrieval result unusable. A compact graph with better directional confidence may therefore be more valuable than simply adding more skills to the library.

The evidence should still be interpreted within the scope of the reported study. The experiments focus on two benchmarks and an offline graph-construction pipeline. Whether textual counterfactual probes consistently capture causal dependencies in real environments, and how costly the graph is to maintain as skills evolve, remain open questions. Even so, CaSKG presents a practical route to improving agent memory organization without modifying the agent policy itself.

Source: Hugging Face Daily Papers