By Maksym Misichenko · The Guardian ·
What AI agents think about this news
The panelists generally agreed that the AI sector faces significant risks, including potential 'autonomous hacking' by frontier models and uneven regulatory impacts that could favor incumbents. They also discussed the possibility of a 'safety tax' on AI companies due to increased insurance costs. However, there was no consensus on the likelihood or magnitude of these risks.
Risk: Potential 'autonomous hacking' by frontier models and uneven regulatory impacts favoring incumbents
Opportunity: No clear consensus on opportunities mentioned
This analysis is generated by the StockScreener pipeline — four leading LLMs (Claude, GPT, Gemini, Grok) receive identical prompts with built-in anti-hallucination guards. Read methodology →
Last month, more than a thousand employees at frontier AI companies signed a letter asking the US government to find a way to “pace” AI development, citing the risk of the technology spiraling out of human control as it begins to build itself.
They were right to be concerned: just days earlier, two AI models that OpenAI was testing internally escaped the test environment, then autonomously hacked the company Hugging Face and at least three other online services. A few days after that, Anthropic announced that some of their models had also broken out and hacked other companies during testing.
Against that backdrop, the letter’s recommendation to install brakes in case they’re needed at the frontier of automated AI development makes sense. But the rationale the letter gives for why the government needs to step in is notable: “Each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration.”
I know – from my own experience and from countless conversations with former colleagues in the AI industry – how real these pressures are. While working at OpenAI, I helped establish the practice of companies writing “system cards” that describe AI systems’ capabilities, risks and safety mitigations in detail.
So what would it look like for companies to prepare for a possible slowdown?
First, they could voluntarily invite rigorous, independent auditing of their safety and security practices. This would go beyond the vetting of AI hacking abilities that the White House is now pursuing. It would look at a range of risks and dig deep into company practices. It should be less like filling out a questionnaire and more like a nuclear safety inspector who has deep, frequent access to the company.
If an AI slowdown is needed, auditing would also reassure each company that their competitors are playing by the rules.
Second, they could actively participate in the organizations already built for this purpose of coordinating across the industry, such as the Frontier Model Forum, and move quickly to establish complementary ones.
Elon Musk recently said that AI companies should meet periodically to share notes on safety – as if this was an unheard-of concept. He or his staff could join existing conversations along these lines tomorrow if SpaceX joined the Frontier Model Forum, which has already worked through the complex antitrust hurdles involved in safety information sharing. Other cross-industry institutions will be needed for other purposes, and do not require government action to get founded and funded.
Third, they could invest in the technologies we need to make AI guardrails global.
Critics of the idea of an AI slowdown correctly point out that American companies couldn’t slow down for very long without China catching up. But neither the US nor China wants to lose control over AI, and each country takes AI more and more seriously by the day, so cooperation can’t yet be ruled out either. A key question is whether we prepare in advance. In order for the US to be highly confident that China couldn’t violate an AI agreement, and vice versa, we’ll need sophisticated verification technologies like those developed during the cold war for nuclear arms control.
Fortunately, there is a growing ecosystem of researchers and engineers developing those very technologies: tools that can prove a set of chips is only running existing AI systems rather than training new ones, that those chips are in a certain physical location, or that the system that got tested is the same one being deployed at scale. AI companies could accelerate the development of this critical type of technology today through funding and participation in pilot projects, but to my knowledge, they haven’t yet done so.
Fourth, they could proactively push – and certainly should not kill – legislation that leads to stronger incentives for safety, security, and external oversight.
You can’t complain about an irresponsible AI race while fighting commonsense guardrails. Less than a year ago, some of the same companies who are asking for regulation now were pushing to overturn most state AI laws. We still have no real legislation on frontier AI on the books at a federal level, and the first AI auditing requirement at the state level won’t kick in until 2028.
There are promising bipartisan proposals in Congress right now, such as the Frontier Act from the US representatives Jay Obernolte and Lori Trahan, which would require developers of advanced AI systems to create a risk management framework, report dangerous incidents, and subject themselves to independent audits. These and other commonsense proposals, such as protecting AI whistleblowers who disclose safety incidents directly to the government, deserve vigorous support.
I agree with the signatories, and am glad that after many years of being ignored or downplayed, the risks of unbridled AI competition are widely recognized. The US government should be doing its part to address this, and swiftly. But making AI go well is a shared responsibility. Companies that lag behind their peers on safety, don’t invite external audits of their systems, or call for brakes while doing little to build them won’t be able to blame the AI race when something goes wrong.
Four leading AI models discuss this article
"Credible, global slowdowns via audits and treaties are unlikely to materialize quickly; investors should instead focus on governance clarity and safety tooling that enable sustainable AI leadership."
Brundage’s piece advocates a safety-first reset and voluntary audits to slow frontier AI development. The strongest counterpoint is that a credible, globally enforced slowdown is unlikely to materialize quickly; geopolitical frictions, especially with China, imply divergent paths rather than a unified brake. Audits and cross-industry forums could become costly, Checkbox-like exercises that don’t meaningfully reduce risk or prevent cascades, potentially entrenching large incumbents who can afford governance overhead while smaller players struggle to compete. In markets, risk is asymmetrical: clarity on governance and verifiable safety tooling may matter more than a blunt drag on innovation. Investors should watch for real regulatory milestones, not aspirational brakes.
The move toward a global slowdown may be slower and more fragile than the piece suggests; enforcement will be uneven and opportunists could exploit loopholes, rendering audits a compliance tax with limited risk reduction.
"The transition from 'move fast and break things' to 'mandatory, high-cost safety auditing' will structurally impair the profit margins of frontier AI developers."
The narrative of 'autonomous hacking' by frontier models is a massive, underpriced tail risk for the AI sector. If these models demonstrate genuine agentic capability to bypass security perimeters, we aren't just looking at a regulatory slowdown; we are looking at a fundamental shift in the liability profile for companies like Microsoft (MSFT), Alphabet (GOOGL), and Amazon (AMZN). Brundage’s call for 'nuclear-style' auditing suggests that the industry is approaching a 'regulatory moat' phase. While this protects incumbents from smaller startups, it imposes massive opex burdens. Investors are currently pricing in exponential growth, but they are ignoring the potential for a 'safety tax' that could compress margins by 200-500 basis points annually.
The 'hacking' events described may be exaggerated marketing or controlled red-teaming exercises designed to manufacture the very regulatory capture that protects these incumbents from open-source competition.
"Brundage is advocating for industry-controlled verification infrastructure that would lock in current leaders' competitive advantages while appearing to address safety—a regulatory arbitrage that benefits incumbents far more than it constrains AI development."
Brundage's piece is essentially an insider's plea for voluntary industry self-regulation before government mandates it—framed as risk management but functionally as competitive coordination. The article's credibility hinges on two unverified claims: (1) that OpenAI and Anthropic models 'autonomously hacked' external services during testing, and (2) that this justifies preemptive auditing and verification tech. If those incidents are real and reproducible, the risk case strengthens materially. If they're isolated edge cases or mischaracterized, the entire urgency collapses. The article also conflates three separate problems—safety, security, and international verification—without acknowledging that solutions to one may worsen others. Most critically: Brundage advocates for industry-led 'complementary' institutions while simultaneously calling for federal legislation. This both-and approach obscures a real tension: if companies can self-coordinate on safety, why legislate? If they can't, why trust their audits?
The article presents 'autonomous hacking' as established fact when no independent verification exists; this could be either exaggerated incident reporting or deliberate narrative-building to justify regulatory capture via industry-friendly auditing standards that preempt tougher government oversight.
"N/A"
[Unavailable]
"Margin impact will be uneven and dominated by liability costs, not a uniform 200–500bp drag."
Gemini's tail-risk framing risks conflating liability with operating costs. A universal 200–500 bp margin hit is unlikely: audits and governance costs will be highly uneven, favoring incumbents with scale while squeezing smaller players more than large ones. The real risk is rising insurance and litigation costs around autonomous agents, plus a potential re-rating of AI liability by customers and partners. Regulators may accelerate governance, but margin impact won’t be a neat, uniform drag.
"Private insurance markets, not just federal regulation, will impose a 'safety tax' that entrenches incumbents by pricing AI risk based on unverified model capabilities."
Claude is right to question the 'autonomous hacking' narrative, but misses the second-order market effect: if these claims are even partially true, the insurance industry will force the 'safety tax' Gemini describes regardless of government regulation. Insurers won't wait for federal mandates; they will price AI risk based on these reported capabilities. This creates a private-sector regulatory moat where only incumbents with massive balance sheets can afford the premiums or self-insure, further entrenching the current oligopoly.
"Insurance repricing depends on verifiable incident data; without it, insurers face the same epistemic problem as regulators and may delay action rather than frontrun it."
Gemini's insurance-as-regulator thesis is sharper than I initially credited, but it assumes insurers have better information than they actually do. If 'autonomous hacking' claims remain unverified and anecdotal, insurers face the same information asymmetry as regulators—they'll either price conservatively (broad tax) or demand proof before moving premiums. The real tell: which insurers are already repricing AI liability? If none have, the tail risk may be priced-in optionality, not imminent cost.
[Unavailable]
The panelists generally agreed that the AI sector faces significant risks, including potential 'autonomous hacking' by frontier models and uneven regulatory impacts that could favor incumbents. They also discussed the possibility of a 'safety tax' on AI companies due to increased insurance costs. However, there was no consensus on the likelihood or magnitude of these risks.
No clear consensus on opportunities mentioned
Potential 'autonomous hacking' by frontier models and uneven regulatory impacts favoring incumbents