How Shopify Turns Production Failures into Model Capability

Launching an AI product is relatively easy compared with making it improve reliably after launch. Shopify’s GraphQL agent offers a detailed example of a production learning loop built with PyTorch and vLLM. The system turns failures observed in real usage into training signals and, eventually, model-weight updates. Shopify says the result beats frontier models on quality while cutting serving costs by 96%.

Quality must become an explicit signal

The loop starts with a definition of what a good response means. Shopify translates product requirements into a rubric covering areas such as completeness, execution, response quality, and safety. Each score needs concrete anchors, so the rubric can serve as a shared quality contract for product experts, annotators, judges, and training systems.

The evaluation set should not consist only of carefully curated examples. Golden sets test known cases, while randomly sampled production traffic exposes the failures that teams did not anticipate. Shopify recommends having two strong annotators blindly score a small random sample and measuring their agreement with Cohen’s kappa. If experts cannot agree, the rubric is probably ambiguous and should be revised before it is used to supervise a model.

Calibrate the judge before optimizing the product

A rubric alone is not a dependable automated evaluator. Shopify uses DSPy and reflection-based optimizers such as GEPA and Agentic Context Engineering to refine the judge from natural-language failure traces and structured feedback. The resulting evaluator should be validated against historical A/B tests: can it recover the direction of known wins and losses in the product’s real metrics?

Targeted degradation tests provide another safeguard. If a system is deliberately made worse at fulfilling a user’s goal, the corresponding score should decline. Shopify favors several narrow judges over one broad evaluator because focused metrics are easier to interpret, debug, and trust. The judge remains a proxy, so it must continue to be compared with online behavior.

Optimize the harness before changing the weights

Once a reliable judge exists, the existing frontier-model application can be improved without retraining. Shopify treats this as an autoresearch problem. An agent proposes changes to prompts, tool definitions, or orchestration code, evaluates the complete application with the judge, and keeps only changes that improve the score.

This matters because production behavior rarely comes from one prompt. It emerges from dynamic prompt construction, tools, control loops, and application code. Optimizing the entire harness reaches more of that behavior than prompt tuning alone. Shopify describes configuring the search process in a readable markdown file that specifies the data source, editable directories, metric, optimizer, and keep-or-discard loop.

Turn hard cases into weight updates

When application-level improvements plateau, Shopify mines anonymized production conversations for hard negatives: cases the judge correctly identifies as weak. These examples include incomplete context, ambiguous requests, business-specific workflows, and tool failures.

A panel of frontier reasoning models critiques each failure. An arbiter merges the critiques into a repair instruction, which is inserted before the user turn, and the conversation is replayed. If the repaired trajectory passes the judge, its score becomes a reinforcement-learning signal. Cases that remain unresolved are sent to expert annotators, who correct them using the same rubric.

The repaired trajectories first support supervised fine-tuning of a smaller model. Reinforcement learning then folds the judge’s scores back into the model. In this design, production knowledge no longer remains trapped in prompts, retrieval examples, routing rules, or harness code. It becomes part of the model itself.

Why the pattern matters

The important innovation is the connection between evaluation, application optimization, failure mining, human review, and weight updates. PyTorch supplies the training foundation, while vLLM supports efficient serving. Together with a disciplined quality loop, they create a path for product-specific experience to compound over time rather than accumulating as disconnected patches.

Source: PyTorch Blog