Can LLM Agents Stay on Script? A New Benchmark for Long-Horizon Consistency

In interactive storytelling, the hardest problem is often not generating a good next line, but preserving the world over time. This paper defines that challenge as Narrative Commitment Preservation (NCP): when a user intervenes freely and repeatedly, can an LLM agent keep earlier facts intact and still advance the story’s must-hit plot points?

To study this, the authors release NCP-Bench, a benchmark built from 100 movie synopses. Each environment includes a structured narrative specification with a trajectory, commitments, and initial facts that can be checked automatically during the interaction between a player agent and a narrator agent.

What the benchmark tests

• whether the model contradicts established facts
• whether it genuinely responds to the player’s actions
• whether required plot commitments are still completed later
• whether performance holds up over long horizons, not just a few turns

Main findings

• Strong language quality does not imply strong commitment preservation.
• Under adversarial interventions, even capable models frequently produce logically conflicting content.
• The best-performing model reported here, GPT-5.2, reaches only a 42% survival rate after 20 turns.
• Fact conflict rates range from 40% to 68% across models.
• Only a small number of runs manage to satisfy all achievement commitments within the 100-turn limit.

The broader message is important for AI games, role-play systems, and any agent that must reason over state across many turns: short-term fluency is easy to admire, but long-horizon reliability is much harder to achieve. Benchmarks like NCP-Bench help expose the gap between surface-level naturalness and actual narrative discipline.

For developers, this is a reminder that memory, planning, and state tracking are not optional extras. If future agents are expected to act inside open-ended worlds, they need more than plausible language—they need mechanisms that preserve commitments under pressure.

Source: Hugging Face Daily Papers