CADWorld Tests Whether Computer-Use Agents Can Handle Real CAD Workflows

Introduction

Computer-use agents are increasingly tested in realistic desktop environments, but many existing evaluations focus on whether an agent can click through an interface or reach a visible end state. Professional engineering software imposes a stricter standard. A CAD task is only truly complete when the result is saved as a native project, remains editable, and preserves the geometric, parametric, and downstream engineering state required by the task.

CADWorld introduces a benchmark designed around that standard. It places agents in FreeCAD and evaluates long-horizon mechanical computer-aided design workflows rather than isolated interface actions.

Key points

• Broad workflow coverage: CADWorld includes 200 tasks spanning 11 categories, including sketching, part modeling, assembly, CAM, finite-element analysis, measurement, mesh processing, and technical drawing.
• Screenshot-and-GUI interaction: Agents work through screenshots and graphical actions. They must interpret the current application state, perform sequences of operations, and respond to the consequences of earlier decisions.
• Artifact-based evaluation: Success is determined by executable, task-specific checks over saved FreeCAD artifacts and auxiliary outputs. These checks cover geometric properties, parametric structure, constraints, manufacturing state, and simulation results.
• A large performance gap: Across seven current agents on the full benchmark, the strongest system achieves 17.5% success, while the expert reference pass rate is 87.0%.

Why CAD exposes a deeper problem

Mechanical CAD workflows are dependency-heavy. A missing or incorrect sketch constraint can affect every later feature. A model may look plausible while having the wrong dimensions, an invalid construction sequence, or a parametric structure that cannot be edited as required. Assembly relationships, manufacturing settings, measurements, and simulation results add further state that may not be apparent from a screenshot.

This makes CAD fundamentally different from a benchmark that judges only visual similarity. The agent must maintain consistency across the interface, the action history, the feature tree, and the saved project. Recovery is also more demanding: fixing a failure may require tracing a dependency, rebuilding a feature, or reconstructing constraints instead of simply clicking an undo button.

The reported failure pattern reflects this progression. Weaker agents often fail before producing a valid artifact at all. Stronger agents are increasingly able to leave behind a project, but their failures move toward structural, geometric, and construction-process requirements. In other words, better GUI competence does not automatically produce reliable engineering execution.

Significance and impact

CADWorld offers a useful test for the gap between general computer use and professional automation. It suggests that future agents need more than accurate visual grounding and action selection. They also need persistent state tracking, long-horizon planning, knowledge of parametric modeling, and validation procedures that can detect when a project has silently become invalid.

For researchers, the benchmark helps separate interface failures from artifact failures. For engineering users, it highlights the standard an agent must meet before it can be trusted with production-oriented workflows: the output must be native, inspectable, structurally correct, and ready for further editing or downstream analysis.

The low success rate should therefore be read less as a verdict on GUI agents than as evidence that engineering software is a demanding frontier for them. CADWorld turns that challenge into a measurable evaluation target.

Source: Hugging Face Daily Papers