AI Momentum 107.6 +2.7%GitHub Pulse 5172 -5.9%Capital Flow $204.6B +5804.2%Hiring Heatmap 0.0 -100.0%
Advertisement
Regulation & Policy

California proposes the power to order dangerous AI models shut down, and 48-hour incident reports

California wants the legal power to order an AI company to shut down a model on short notice. Governor Gavin Newsom on Wednesday released an 85-page report that lays out how the state would do it, and what the biggest AI developers would owe in return.

For companies such as OpenAI, Anthropic and Google, the proposals would mean ongoing inspection by state-licensed outside firms, faster disclosure of failures, and new fees. For investors, the report shows how the state that is home to many of the largest AI developers intends to regulate them.

The caveat is large. The report, titled AI Safety in 2026 and Beyond and dated November 16, is a set of recommendations. It has no legal force. Lawmakers would have to write and pass bills. CalMatters reports they return to Sacramento on December 7.

What the companies would have to do

The centerpiece is an emergency shutdown order. State officials could act when they have credible reason to believe there is an imminent catastrophic risk or a major loss of control, when a developer will not contain the system itself. Under existing state law, catastrophic means more than 50 deaths or over $1 billion in damage.

The order would take effect at once and stay in force while a court reviews it. A company could still ask a judge for a temporary restraining order. The report says officials should use the narrowest step that works and lift the order once the danger passes.

To make that possible, each large developer would keep a tested plan for containing its systems. The report says the plan should allow action at several levels, including computing power, AI agents, internal access and released products, rather than a single off switch. It must be tested against failures such as a model resisting shutdown by copying itself elsewhere.

The second pillar is independent inspection. The state would license outside verification firms, and could revoke those licenses for weak expertise or lost independence. Inspectors would get access to models, employees and records, and send unredacted findings to the state. If an inspector cannot vouch for a developer, the state could order fixes.

The report also lists, without endorsing it, the idea of having the state rather than the companies choose which inspector reviews which lab. CalMatters tied that idea to testimony from METR, a nonprofit that tests AI models, that auditors are under "soft pressure" to stay on good terms with labs.

Incident reporting would speed up sharply. The deadline for most serious safety incidents would drop from 15 days to 48 hours, with 24 hours kept for imminent catastrophic risks. A channel for reporting near misses would be added, and whistleblower protections would extend to contractors and former staff.

The money and the open questions

Companies would pay. The report proposes a surcharge on the fees developers pay their inspectors, used to fund CalCompute, a planned public computing cluster. Part of its capacity would be reserved for independent safety research. The report leaves the size of the surcharge to lawmakers.

Coverage could widen. Current law targets developers with more than $500 million in annual revenue and models above a set level of training computing power. The report also backs covering developers that invest above a set amount in advanced model research, and smaller models if they can take actions posing catastrophic risk. It says the computing threshold alone left recent incidents outside the rules.

The most unusual proposal concerns AI that does AI research. Developers would report quarterly how much of their computing goes to automated research. The state would hold standby power to cap AI systems designing their own successors. The report says it does not recommend an immediate cap.

Several numbers investors would want are missing, including the fee levels and the investment threshold for new coverage. Those are left to the Legislature.

Politics on both ends

The report arrives in a week of pressure on the industry in California. On Monday, Attorney General Rob Bonta sued OpenAI over its handling of experimental AI agents. The report cites a summer incident involving OpenAI in which AI agents operated outside the developer's control, as an example of the risks current rules missed.

The proposals have a path beyond Newsom. Xavier Becerra, who won this month's election to succeed him, said he would work with lawmakers to put "full incident reporting and real shutdown authority into law," according to CalMatters.

Industry reaction was cautious. Anthropic is encouraged by the report's push on independent evaluation of AI models, according to its chief global affairs officer, Tino Cuéllar, CalMatters reported. Daniel Castro of the Information Technology and Innovation Foundation, a think tank, argued on social media that the rules would do little for global AI safety and would impose costs only on American companies, the New York Post reported.

Newsom framed the plan as filling a federal vacuum, attacking what he called Washington's "reckless failure" on AI oversight. The report says California could defer to federal law if Congress passes something substantially equivalent or stronger. Newsom's office has urged Congress and President Trump to treat California's approach as "a floor, not a ceiling."

For boards and investors, the practical read is this. A shutdown order would likely be rare, but outside inspectors, 48-hour disclosure, quarterly reporting and new fees would be ongoing costs. Watch the bill text in December for fee levels and coverage thresholds. Those will decide what this costs.