Anthropic’s AI Safety Bill Defines Catastrophe as More Than 50 Casualties or $1 Billion in Damage
How many people must face death or serious injury before artificial intelligence becomes a catastrophic threat? More than 50, according to legislation drafted by one of America’s leading AI companies.
Even then, federal authorities would face substantial legal hurdles before stopping the dangerous technology.
The proposal comes from Anthropic, developer of the Claude AI system. It would establish national AI safety standards while giving developers considerable influence over how regulators assess their products and when they can intervene.
Defining Catastrophe
Last Friday, Politico reported that Anthropic had quietly circulated draft legislation among congressional offices, making its reporting and details of the proposal available to subscribers.
On Monday, the Daily Caller published the full 37-page draft, dated September 29. Its tentative title is the Artificial Intelligence Evaluation, Governance, and Incident Safety (AEGIS) Act.
Section 3 provides,
The term “catastrophic risk” means a foreseeable and material risk that a frontier developer’s development, storage, use, or deployment of a frontier model will materially contribute to the death of, or serious injury to, more than 50 people, or to more than $1,000,000,000 in damage to, or more than $1,000,000,000 in loss of, property, arising from a single incident[.]
The definition covers assistance in developing chemical, biological, radiological, cyber, or nuclear weapons. It also includes certain autonomous criminal acts and AI systems escaping human control.
The bill separately defines “enumerated risk,” a narrower category relevant to emergency intervention. It covers biological weapons, offensive cyber operations, loss of control, and automated research that amplifies those threats.
The financial threshold offers an alternative to the casualty requirement. Authorities would not have to establish both, nor would they have to wait for actual deaths.
Still, the numerical thresholds are staggering. An AI model threatening 50 human lives would fall below the casualty threshold. So would one threatening $900 million in property damage.
But what makes the 51st potential victim, or the next $100 million in losses, the dividing line between a serious danger and a catastrophe?
A High Bar to Stop Dangerous AI
Under Section 12, the Commerce Department could seek a federal court order suspending a dangerous AI model’s development, deployment, or internal use.
But the danger must qualify as “imminent,” which Section 3 defines simply as “present or impending.” That adds another hurdle: Authorities must establish not only the potential scale of a catastrophe, but also that the threat is sufficiently immediate to justify intervention.
The commerce secretary would first have to submit a detailed determination identifying the expected harm, the model’s dangerous capabilities, and why existing safeguards would fail.
The department would also have to explain
why no less restrictive measure, including any measure the frontier developer proposed, will adequately avert or mitigate the risk, and each less restrictive measure the Agency considered[.]
A qualified independent evaluator would also have to assess the danger.
The U.S. District Court for the District of Columbia could authorize suspension only after finding sufficient evidence of the imminent threat and determining that no less restrictive intervention would work.
The court must also consider the harm that suspension could cause the developer, users, and the public. Judges would review the government’s case independently, without deferring to the Commerce Department’s conclusions.
Ordinary suspension orders would expire after 45 days unless renewed. Emergency orders issued before hearing the developer’s arguments would expire within seven days.
The draft also prohibits courts from ordering the destruction of model weights, training data, or source code under these provisions.
Such safeguards could prevent arbitrary government intervention. But by the time independent evaluators, federal bureaucrats, and judges finish determining whether a catastrophe is sufficiently imminent, the AI system in question might have already inflicted considerable harm, perhaps even in convenient batches of 50 casualties or fewer.
An Exemption for Government Activity
Section 3 contains another consequential provision. It excludes certain risks from the definitions of catastrophic and enumerated harm, including those arising from “lawful activity of the Federal Government.”
The exemption applies even when AI could contribute to consequences otherwise meeting the bill’s catastrophic-risk threshold.
It does not legalize otherwise unlawful conduct or provide blanket immunity. But it creates a distinction between risks arising from private actors and those associated with lawful federal operations.
But consider the vast scope of activities Washington already deems lawful: mass surveillance, warrantless intelligence gathering, extensive military operations abroad, increasing reliance on the military for domestic policing, and the collection and sharing of Americans’ personal data across federal agencies.
Then there is the sprawling federal machinery overseeing education, healthcare, environment, and much of law enforcement, responsibilities traditionally exercised primarily by states and local communities. In these cases, Washington’s ever-expanding role rests on broad interpretations of federal authority, some of which constitutional federalists have long challenged.
At the same time, Washington increasingly incorporates AI into national security, intelligence, and civilian administration. Therefore, the exemption raises an obvious question: Why should potentially catastrophic harm receive different treatment simply because it arises from an activity over which the federal government has usurped authority?
Limiting State Authority
The draft would also preempt certain state regulations governing AI safety disclosures, independent evaluations, and incident reporting. It also addresses state suspension orders, although the draft’s operative provisions require careful interpretation.
The legislation preserves other state authorities, including specified child-protection and privacy laws and common-law injury claims.
Nevertheless, federal preemption could prevent states from imposing their own requirements in areas covered by the national framework.
That raises a separate constitutional concern: Washington would consolidate authority over AI safety while limiting both state independence and certain forms of federal intervention.
Anthropic Defends Its Proposal
Anthony Cimino, Anthropic’s head of federal affairs, defended the proposal on Sunday.
He explained that its casualty and financial thresholds came from existing California legislation:
The threshold in our draft language pulled directly from existing standards in California law.
He did not explain why exactly it should become the national standard.
Cimino stressed that the bill would not eliminate corporate liability or other government authorities. He also acknowledged concerns about the high bar for emergency intervention:
But the bar can’t be so high that the power never gets used when it’s needed.
He further conceded:
It’s still probably time to re-evaluate thresholds to make sure the national standard is as strong as it can be.
Cimino described the legislation as a discussion draft, not a finished proposal. As of Tuesday, no lawmaker had formally introduced it.
The stakes extend far beyond Anthropic. The debate presents two dangers: granting Washington excessive power over emerging technologies and allowing the corporations developing them to determine the limits of their own oversight.
Neither deserves blind trust. Government agencies notoriously expand their authority, while corporations have powerful incentives to protect their commercial interests. Concentrating regulatory power in Washington, particularly under rules shaped by the industry itself, risks entrenching both.
The alternative is not to abandon public safety, but to preserve meaningful accountability through state and local authority, judicial remedies, and the rights of individuals harmed by dangerous technologies.
Ultimately, decisions affecting human life and liberty should remain as close to the people as possible, rather than resting with distant federal bureaucracies or the corporate giants they are supposed to oversee.
Related Articles:
OpenAI Backs Bill Shielding AI Firms From Liability in Mass-harm Scenarios
AI Giants Prepare for a “Catastrophic AI Event” and the Revolt That Could Follow
The “Rogue AI” Scare: Threat, Hype, or New Globalist Push?
Trump, Blackburn Push to Federalize AI Control
Anthropic Sues Trump Administration Over Pentagon Blacklist
