How DoorDash Uses Multi-Agent LLMs to Remove 60,000 Feature Flags

Introduction

Feature flags make staged releases and experiments easier, but inactive flags gradually become code debt. DoorDash is addressing that problem with a multi-agent LLM system that connects flag metadata, code search, human review, isolated execution, and automated validation into one cleanup workflow.

Key points

• DoorDash’s experimentation platform spans about 623 repositories and manages more than 60,000 feature flags.
• Around 2,300 new flags are added each month, while more than 1,000 have already been classified as stale.
• A flag is considered stale when it has not been modified for 90 days, is still referenced by code, and has not been archived, retired, or explicitly excluded.
• In a test involving 50 stale flags, the system generated 45 usable pull requests, with an average cleanup time of 13.8 minutes and a cost of $4.79 per task.

Why flag removal is difficult

DoorDash uses dependency-injection wrappers around its flags. Definitions, client calls, business logic, and tests may be distributed across many files. Even a simple boolean flag can therefore require changes in five to 20 files.

This structure also exposes the limits of syntax-only automation. Uber’s open-source Piranha uses abstract syntax tree transformations and rule-based detection to remove stale flags. DoorDash found that this approach could not fully handle relationships created by dependency injection, where the important connection is semantic rather than directly visible in syntax.

The LLM-based approach is intended to reason across code search results, usage context, and cleanup strategy. It does not eliminate the need for deterministic checks, but it can handle a wider range of repository structures than a fixed transformation rule.

A two-stage agent workflow

The first stage is an orchestration agent powered by Claude Sonnet. It retrieves stale-flag tickets from Jira, searches the relevant repositories, and uses the Model Context Protocol to query the experimentation platform. The resulting report includes metadata such as rollout percentages and target values. An engineer reviews the report and confirms the target value before any source code is changed.

The second stage uses a Claude Opus cleanup agent. Each task runs in an isolated Git Worktree, and a repository can run up to four agents concurrently. The agent locates all references, selects a removal strategy, updates production and test code, and runs builds, tests, JaCoCo patch-coverage checks, and Detekt static analysis. A pull request is created only after the checks pass. Agents time out after one hour, while Gradle runs with its Daemon disabled to reduce shared-state problems across worktrees.

Results and remaining limits

Of the 50 evaluation tasks, 31 pull requests were merged after the first submission, 14 required changes, and five required engineer intervention. One-shot success reached 100% for simple flags, 94% for medium-complexity flags, and 85% for complex flags. All five manual interventions involved deeper call chains or parameter passing across interfaces. DoorDash reported no bugs or regressions in the 50 code changes.

The result suggests that agent automation is most useful when it is treated as an engineering system rather than a code-generation feature. The agent performs discovery and edits, but human approval controls ambiguous experiment semantics, while worktree isolation and automated checks limit operational risk.

DoorDash plans to add confidence scores for lower-risk cleanup tasks and a post-cleanup quality review to catch issues such as misleading variable names left behind after a flag is removed. For other engineering organizations, stale-flag removal is a practical agent use case: the goal is well defined, the repository changes are reviewable, and correctness can be tested. The transferable pattern is not simply “ask a model to delete code,” but to combine machine execution, human confirmation, isolated changes, and automated acceptance.

Source: InfoQ Chinese