PlannerForge Connects Autonomous Driving Tests with LLM Agents

Introduction

Testing an autonomous driving system is more than generating an unusual traffic scene. A useful test must translate a requirement into an executable scenario, select or alter the right environment, run the motion planner, and analyze whether the resulting behavior reveals a meaningful failure. In many existing pipelines, these jobs are handled by separate tools with limited communication between them. PlannerForge, accepted to EMNLP 2026, proposes an LLM-agent framework designed to connect the process.

What the framework does

• A broader testing workflow. The framework covers scenario generation, scenario selection, scenario modification, module routing, planner testing, and assessment. This positions the language model as an orchestration layer rather than merely a prompt-based scenario writer.
• Two additional stages. PlannerForge extends the conventional testing pipeline with ADS Enhancement and ADS Benchmarking. The first is intended to support improvements to an autonomous driving system, while the second organizes comparisons across systems or configurations.
• A multi-model evaluation. The study tests 10 off-the-shelf LLMs under five prompt conditions. Best-per-task scores range from 0.88 to 1.00. According to the supplied results, open-source 20B–35B backends match commercial APIs on most tasks, and Qwen3.6:35B reaches comparable performance on three of the five reported tasks.
• End-to-end performance matters. Chaining the modules retains 83% of seed queries in the commercial-model setting and 78% with open models. The gap between isolated task scores and pipeline retention highlights the difficulty of preserving intent while requirements pass through several agents.
• More executable scenarios from language requests. In the reported comparison with Scenario Factory 2.0, PlannerForge produced 193 executable scenarios from 200 natural-language requests, compared with 144 for the baseline. It also realized 92%–96% of requested city, road, and vehicle attributes.

Why it matters—and what remains open

The main contribution is architectural. Instead of treating generation, simulation, and analysis as independent utilities, PlannerForge makes the LLM agent responsible for coordinating them. That could make it easier for engineers to express complex test requirements in natural language while keeping the resulting scenarios tied to executable checks.

The results should not be read as proof that autonomous driving safety has been solved. They describe framework-level task performance, and the usefulness of the final assessment still depends on the simulator, scenario realism, evaluation criteria, and the agent’s ability to detect its own mistakes. The end-to-end retention figures also show that routing and format conversion remain important failure points. Further validation would need to examine richer interactions among road users, real-world data, and independent safety audits. The central question is whether an LLM-based testing loop can turn discovered failures into changes that are both effective and verifiable.

Source: Hugging Face Daily Papers