Who Should Write the Rules for AI Safety?

Introduction

Artificial intelligence is moving into finance, telecommunications, defense, energy, and other sectors where failures can affect institutions and critical services at scale. As a result, AI safety is no longer only a product-development question. It is also a question of market structure, public accountability, and national security. In this essay, Aidan Gomez, Cohere’s co-founder and CEO, asks who should decide the boundaries for a technology expected to reshape society: a small group of leading laboratories, or a more open and reviewable governance process?

The source is an advocacy piece written by an executive of Cohere. Its immediate context is a roadmap from Anthropic CEO Dario Amodei that calls for coordination among leading AI labs and seeks a limited antitrust exemption for that work. The article therefore reflects both a policy position and a competitive perspective. Even so, its central institutional question is important beyond the rivalry between companies.

Key points

• The dispute is not about whether guardrails are necessary. Gomez acknowledges that AI can be used to identify vulnerabilities in software and systems, while offensive cyber capabilities may become cheaper faster than defenses improve. He also emphasizes that Cohere’s systems are used in high-stakes environments, making independent scrutiny valuable.
• The central issue is who defines risk. If a few large laboratories set the standards, the definition of a dangerous system, the evaluation thresholds, and the pace of development may reflect their existing capabilities. Other developers, governments, researchers, and affected communities may have little opportunity to shape the result.
• An antitrust waiver could entrench incumbents. Coordination might give companies time to conduct safety work without racing for market share. But the same arrangement could turn the resources and scale of today’s leaders into the baseline for lawful competition, making entry more difficult.
• Model size is not the whole risk surface. The article argues that smaller models can acquire significant capabilities through tools, orchestration, and verification. Systems composed of multiple models may also create risks that are not captured by rules based mainly on compute or model-scale thresholds.

Lessons from previous regulation

To illustrate how safety goals can produce protected market structures, Gomez points to two historical examples. In the United States, government recognition of a small number of bond-rating organizations created a durable barrier around outside evaluators. Those institutions later assigned top ratings to complex subprime securities, contributing to a wider financial crisis. In Europe, automobile manufacturers received broad influence over standards for sales and servicing in the name of safety and reliability. The resulting structure required years of reform to unwind.

The lesson is not that strict standards are misguided. It is that safety standards do not need to be controlled by incumbents. A credible AI regime would include independent evaluators, researchers, governments, open-source developers, affected industries, and the public. It would publish the evidence behind its thresholds, provide ways to challenge decisions, and revise its framework as new forms of risk appear.

Why it matters

A closed agreement among a few companies could improve coordination in the short term, but it could also reduce experimentation and make one commercial model appear to be the only safe model. AI governance must therefore answer two separate questions: how to limit dangerous capabilities, and how to prevent the definition of danger from being shaped primarily by companies that benefit from the rules.

The broader choice is between a safety regime that is transparent, contestable, and adaptable, and one that gives a small group of market leaders lasting authority over both the speed and direction of AI development.

Hacker News