Google DeepMind Pilots Double-Blind Evaluations for AI Models

As AI systems become more capable, the credibility of their test results increasingly depends on whether the model has seen the test material beforehand. Google DeepMind says it is piloting a “double-blind” evaluation in which neither the model provider nor the external evaluator can access the other party’s most sensitive assets.

The project, announced on August 27, 2026, brings together Google DeepMind, the Singapore AI Safety Institute, OpenMined, AVERI and MLCommons. The partners are testing a Gemini Flash Lite model against confidential benchmarks inside a privacy-preserving environment.

Why the evaluation problem matters

Benchmark contamination occurs when a model has encountered evaluation questions, prompts or closely related material before testing. A contaminated benchmark may produce an impressive score without showing how well the model performs on genuinely unseen tasks. This is particularly concerning for evaluations used to assess safety, cybersecurity capabilities or systems relevant to public-sector decision-making.

External organizations have traditionally relied on contractual restrictions, zero-logging procedures and operational controls to protect their prompts. Those measures can help, but they leave an important trust question: how can both sides verify that the other has not accessed sensitive information?

How the pilot works

• The evaluator keeps its prompts private. Google does not receive the external test questions in readable form.
• The model remains proprietary. The evaluator does not obtain Gemini’s model weights.
• The computation runs in a protected environment. The pilot uses Confidential Space, part of Google Cloud’s confidential computing portfolio.
• Cryptographic evidence supports the separation. The stated goal is to verify that the model and evaluation data remain confined to the intended environment.

This arrangement removes a familiar compromise. Evaluators no longer need the provider to expose model weights, while providers do not need to receive a complete set of questions that could later inform model optimization.

What it could change

The main contribution is not a new benchmark, but a new way to establish confidence in benchmark administration. If external organizations can test advanced systems without surrendering their data, more independent assessments may become practical. The approach may be especially useful when prompts contain sensitive cybersecurity scenarios or information controlled by government bodies.

Still, cryptographic isolation cannot by itself make an evaluation objective. The quality of the test set, the choice of metrics, the execution configuration, result interpretation and the independence of participating organizations remain important. The announcement describes a pilot rather than a complete industry standard, so further technical reporting and validation across models and evaluation types will be necessary.

If those hurdles can be addressed, double-blind evaluation could become part of the infrastructure for credible AI oversight. It would give researchers, policymakers and businesses a stronger basis for distinguishing genuine model performance from results shaped by prior access to the test.

Source: Google DeepMind