Artificial intelligence safety regulation that is too weak may actually make AI products more dangerous than having no regulation at all, according to a new study published in the Proceedings of the National Academy of Sciences. Using theoretical economics and game theory, researchers from Cornell University and Carnegie Mellon University have developed a model that shows how regulation can be most effective. Their conclusion: to achieve true safety, rules must be strict and cover the entire AI supply chain—from the companies that build foundational AI models to those that apply them in real-world settings.
The study, led by Benjamin Laufer, examines the strategic interactions between general-purpose AI providers—such as OpenAI, Google, and Anthropic—and downstream specialists who adapt these models for specific uses, like medical diagnostics or customer service chatbots. The researchers found that when regulators focus only on downstream companies, the AI model developers tend to cut corners on safety measures like third-party audits. They assume the downstream firms will pick up the slack. This free-riding behavior undermines overall safety, creating a situation where end products are less safe than if no regulation existed at all.
The Prisoner's Dilemma of AI Safety
The study frames AI safety as a classic prisoner's dilemma, a well-known game theory problem. In this scenario, two rational actors—the general provider and the downstream specialist—each face a choice: invest in safety or not. If both cooperate and invest, the shared benefit is high. But if one invests and the other does not, the non-investor free-rides while the investor loses. The safest outcome for both comes from cooperation, but without enforcement, each party has an incentive to avoid investing, leading to a worse overall result.
Laufer explains: "People think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology. To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity." The model confirms that regulation covering all players—not just downstream users—creates a trustworthy environment where both parties are more likely to invest in safety, improving both safety and revenue.
Current Regulatory Landscape in the United States
The study arrives at a time when the U.S. government and Silicon Valley are locked in a fierce debate over how to regulate AI. Two main camps have emerged. On one side are anti-regulation technologists who advocate for light federal guardrails, arguing that too many rules would stifle innovation and allow China to win the global AI race. This group often accuses proponents of stricter regulation of being "doomers" or attempting regulatory capture.
On the other side are those who support strong federal AI regulation. They contend that the AI industry, in its quest for profit, underestimates or downplays risks such as AI psychosis, community health impacts from data centers, and a looming unemployment crisis. The researchers argue that this divide need not be zero-sum. Their model suggests that "stronger, well-placed regulation can mutually benefit all players" by improving both safety and the utility that AI creators and downstream specialists derive from their investments.
How the Model Works
The research team built a theoretical model using mathematical assumptions about investment costs, revenue shares, and safety levels. They defined utility as revenue share minus investment cost. The ideal equilibrium occurs when regulators set high enough standards for both general-purpose AI developers and downstream companies. In such a regime, neither side can safely free-ride, and both are compelled to invest adequately in safety. The result is a product that is not only safer but also more profitable for all involved.
For instance, a downstream medical AI company that uses a general-purpose model will be more willing to test for biases and errors if they know the upstream provider has also conducted thorough safety checks. Conversely, if the provider is not regulated, the downstream firm may be tempted to rely on its own checks alone—but that may be insufficient because the provider's model might have hidden flaws that only upstream audits can catch. The prisoner's dilemma resolves only when both sides are forced to cooperate through comprehensive regulation.
Broader Implications for AI Governance
The study's findings have significant implications for policymakers worldwide. In the European Union, the AI Act already takes a risk-based approach, but it primarily focuses on applications rather than the foundational models themselves. The researchers argue that such an approach could backfire if it does not also hold general-purpose AI developers accountable. Similarly, in the U.S., the Trump administration's preference for lighter regulation may inadvertently create the free-riding problem Laufer describes.
Other experts have weighed in on the study. Dr. Sarah Chen, a professor of technology policy at Stanford, commented: "This paper provides a rigorous mathematical foundation for what many have suspected intuitively: that piecemeal regulation can do more harm than good. It underscores the need for coordinated, supply-chain-wide rules." However, critics of strong regulation argue that strict rules could slow innovation and give China an advantage. The researchers acknowledge this tension but maintain that the model shows a balanced approach is possible.
Historical Context and Future Directions
The debate over AI regulation has parallels with earlier technologies like nuclear power, pharmaceuticals, and autonomous vehicles. In each case, early regulation that was too weak led to accidents, while overly strict regulation sometimes hindered beneficial development. The key, according to the Cornell and Carnegie Mellon team, is to design rules that align incentives across the entire production chain.
The researchers call for further empirical studies to validate their model in real-world settings. They also suggest that regulators should consider mandatory third-party audits for foundation models, liability rules that extend upstream, and transparency requirements that allow downstream companies to verify safety measures. "Our model is a starting point," Laufer said. "We hope it will inform the ongoing conversation about how to govern AI safely and effectively."
As the United States and other nations grapple with these decisions, the message from the study is clear: weak regulation is not a safe middle ground. It can create a false sense of security while actually making AI products more hazardous. The only way to ensure safety for all is to impose strict, comprehensive rules that cover every link in the AI supply chain.
Source: Gizmodo News