How Existential Fears Are Shaping the Debate Over AI

Why Is the Fear of Extinction Driving AI Policy?

In the spring of 2023, the vocabulary of AI policy changed almost overnight. In March, the Future of Life Institute published an open letter calling for a six-month pause on training systems more powerful than GPT-4. In May, the Center for AI Safety released a single-sentence statement: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." Within months, heads of state were meeting at Bletchley Park, and the Bletchley Declaration of November 1, 2023 named the potential for "serious, even catastrophic, harm" from frontier models.

I have come to think that the most important question about that moment is not whether the fears were sincere. Many of the people who signed those statements clearly meant them. The question is what happens to a policy debate when its loudest register is extinction, and who ends up holding the pen when the drafting starts.

That is the ground Sarah Myers West covers in her piece for the AI Now Institute. According to AI Now's summary, as panic about AI's capabilities turns into proposals to regulate the companies building it, she argues for clarity: break down what the actual risks are so they can be properly mitigated. I have worked from that summary rather than the full text, so I limit what I attribute to West to what the summary states. I want to take that argument seriously here, add some context about how the debate developed, and talk about what it means for anyone running an organization that has to make real decisions about AI while the policy conversation is still unsettled.

What Does "Existential Risk" Actually Mean in the AI Debate?

The phrase gets used as if it were a single claim, and it is really at least three.

The first is a claim about future systems: that sufficiently capable AI could act against human interests in ways we cannot control, up to and including human extinction or permanent loss of control. The second is a claim about misuse: that capable models could help bad actors build weapons, run large-scale cyberattacks, or manipulate populations. The third, which gets less airtime, is a claim about the present: that AI systems already deployed are producing harms in hiring, lending, policing, housing, health care, and information environments, and that these harms fall unevenly.

The trouble is that all three travel under one banner. When a legislator says "AI safety," the audience often hears the first claim, while much of the documented evidence to date, such as incident reports and audits, concerns the third. Conflating them makes it hard to ask the plain question any policy needs to answer: what specific harm are we trying to prevent, how likely is it, and what would actually reduce it?

This is my reading of the distinction that AI Now's summary of West's argument points toward. Clarity is not a request to be less worried. It is a request to say which worry we are talking about, because the remedies differ so much.

How Have Existential Fears Shaped Regulation So Far?

The fingerprints of the extinction framing are visible in the concrete choices governments have made, and the clearest evidence is in the thresholds they picked.

Three examples are easy to check:

• Executive Order 14110 (October 30, 2023) required developers of dual-use foundation models to report on training and safety testing once a model was trained using more than 10^26 integer or floating-point operations. That order was rescinded on January 20, 2025, by Executive Order 14148, but it showed how quickly compute became the unit of regulatory concern.
• California SB 1047, vetoed by Governor Gavin Newsom on September 29, 2024, would have applied to "covered models" as defined in the enrolled bill text (Cal. SB 1047, 2023–24 session): those trained with more than 10^26 operations at a compute cost above $100 million. Its supporters framed it around catastrophic harm.
• The EU AI Act (Regulation (EU) 2024/1689) presumes that a general-purpose AI model has systemic risk when the cumulative compute used for training exceeds 10^25 floating-point operations (Article 51(2)). The Commission can amend that threshold through delegated acts (Article 51(3)), which is one reason thresholds may keep moving.

These are different instruments from different jurisdictions, but they share a design instinct: draw a line by the size of the model, and attach heavier duties above it. That instinct comes naturally if your central fear is a runaway frontier system. It fits much less naturally if your central concern is a mid-sized hiring tool trained on a fraction of that compute that quietly screens out qualified applicants.

Here is how the two framings compare when you look at what they tend to produce.

Neither column is wrong to exist. The problem arises when the left column absorbs nearly all the political oxygen and the right column is treated as a footnote, or as a distraction from the "real" issue.

Who Benefits When the Debate Centers on Extinction?

This is a delicate question, and I want to handle it carefully, because it is easy to slide into cynicism about people's motives.

Some researchers who warn about extinction risk have spent years on the problem and left comfortable careers to do it. Others who signed the 2023 statement lead companies that continue to build the most capable systems in the world. Both facts can be true at once, and they do not cancel each other out. What I think we can say, without accusing anyone of bad faith, is that a framing has consequences that are independent of its authors' intentions.

Consider what an extinction-centered debate does to the structure of the conversation. If the danger is a future system of enormous power, then the natural experts are the people who build such systems. The natural regulatory conversation is a negotiation with a small number of companies. And the natural time horizon is the future, where claims are harder to check against evidence. Meanwhile, the questions that would demand present-tense accountability, such as whether a deployed system is accurate, who is harmed when it is wrong, and who pays, recede.

Urgency combined with vagueness can be powerful. When a risk is both enormous and undefined, it becomes easier to defend almost any proposal in its name and to dismiss objections as naive. That is a lot of discretion to hand to whoever defines the terms.

There is a second, quieter effect. Rules designed around frontier developers can be expensive to comply with, and compliance costs can favor larger firms with bigger legal budgets. The dynamic of who shapes the rules and who ends up advantaged is the subject of my piece on regulatory capture in AI governance. I am not claiming that any specific law was captured. I am saying that a debate organized around a very small number of very large actors may be more easily shaped by them.

What Is West's Argument for Clarity?

AI Now's summary describes the central move as asking what the actual risks are and breaking them down so they can be mitigated. I find that a useful discipline, and I would put it in the form of a few questions anyone can ask of a proposal or a headline.

First, what is the specific harm? "AI could be dangerous" is a mood. "A model could provide meaningful uplift to someone synthesizing a pathogen" is a claim you can test, argue about, and design a control for.

Second, what is the evidence, and what kind of evidence is it? A documented incident, a controlled evaluation, an expert forecast, and a thought experiment are all legitimate inputs, but they deserve different weights, and a good policy conversation says which one it is using.

Third, what would the proposed remedy actually change? A licensing regime for frontier models might be a reasonable idea, but it will not fix a biased tenant-screening algorithm sold by a small vendor. If a proposal cannot say which harm it addresses, it is probably addressing a feeling.

Fourth, who is in the room? If the people affected by a harm have no seat where the risk is being defined, the definition will tend to miss them.

None of these questions requires you to dismiss catastrophic risk. They only require that catastrophic risk be argued for in the same way as any other risk: specifically, with evidence, and with a remedy that matches the problem.

Why Does This Matter for Organizations Using AI Today?

If you run a company, a hospital, a school district, a law firm, or a public agency, you are probably not building a frontier model. You are buying tools, embedding them in workflows, and trusting them with decisions that affect real people. So the practical question is how the policy debate touches you.

I think it touches you in three ways.

The rules you will actually face are about deployment

While headlines focus on frontier labs, the obligations that land on ordinary organizations tend to concern how AI is used. The EU AI Act is the clearest example: Article 5 lists prohibited practices, and the Act's high-risk regime is organized around use cases such as employment, education, credit, and essential services, rather than around model size. In the United States, the picture is more fragmented, with a patchwork of state laws and sector-specific rules layered on existing anti-discrimination and consumer protection law. In other words, the existential debate may decide what happens to a few labs, but your exposure comes from what you do with the tools.

I covered how quickly the American landscape has been shifting in my week-in-review on Washington rewriting the AI rules. The lesson holds: policy is moving, sometimes reversing itself within a single administration. An organization that built its whole approach on one specific rule can find that rule gone within a year. The 2023 executive order is the plainest example.

Governance built on fear ages badly

A governance program built around dramatic scenarios tends to produce dramatic documents: principles, statements, committees that meet quarterly. A governance program built around specific uses produces less exciting but more useful things: an inventory of where AI is used, a named person responsible for each use, a way for someone affected to contest a decision, and a record of what happened when something went wrong.

The NIST AI Risk Management Framework 1.0, released in January 2023, is organized around four functions (Govern, Map, Measure, and Manage), and its emphasis is on context: you map a system's purpose, its setting, and its likely impacts before deciding what to measure. It is voluntary, and it is not the only approach, but the shape of it illustrates the point. Clear risk analysis starts with the specific system in its specific setting.

For generative systems, NIST also published the Generative AI Profile (NIST AI 600-1) in July 2024. It lists risks specific to generative AI, such as confabulation, information security, and chemical, biological, radiological, and nuclear information, and suggests actions under the same four functions. That makes it more relevant to the frontier-versus-deployment question, because it treats model-level and deployment-level risks together.

A concrete starting checklist:

• Inventory every AI use, including tools embedded in vendor software.
• If you operate in or serve the EU, map each use against Article 5 prohibited practices and the Annex III high-risk categories (employment, education, access to essential services such as credit, and others).
• Name an accountable owner for each use.
• Ask vendors for intended-use documentation, evaluation results, and known limitations.
• Create a route for affected people to contest a decision.
• Log incidents and review them on a fixed schedule.

Vague risk language is a trap for your own communications

Organizations sometimes borrow the extinction-era vocabulary in their own public statements, because it sounds serious and responsible. I would be cautious about that. A statement that says you take "AI safety" seriously, without saying what you have done, invites suspicion that it is a performance rather than a commitment. Readers, customers, and regulators may well notice the difference.

Are Existential Risks Real, or Is This All Hype?

I do not think that is the right way to divide the question. As AI Now summarizes it, West's argument is not that catastrophic risks are imaginary but that we owe ourselves an honest accounting of which risks are which.

Some things I feel reasonably confident about. Serious people disagree about how likely catastrophic outcomes are, and the disagreement is not resolved by counting signatures on open letters. Uncertainty cuts in both directions: it is a reason to take low-probability, high-severity outcomes seriously, and it is also a reason to be skeptical of confident timelines from people who profit from the technology. And documented harms from deployed systems are not hypothetical, so a policy conversation that treats them as secondary has, in my view, its priorities backwards.

What I resist is the idea that we must pick a side. A society can fund evaluations of frontier models and also require impact assessments for hiring algorithms. It can care about loss of control and also about who gets denied a loan. The scarce resource is not concern. It is attention, and attention is exactly what a single overwhelming frame consumes.

What Should Readers Watch for Next?

A few signals will show which direction the debate is heading.

Whether thresholds keep shifting. Compute thresholds were an early proxy for danger, and they are already being questioned, because some observers argue that smaller models are becoming more capable, which would weaken compute as a proxy. If regulation stays anchored to size while capability moves elsewhere, the rules will chase the wrong target.

Whether deployment rules get stronger or weaker. The most consequential fights for ordinary organizations are over state-level laws, sector regulators, and existing civil rights and consumer protection statutes. Watch how those are applied to AI, because that is where your obligations will come from.

Whether affected communities gain a seat. Ask of any new advisory body, summit, or commission who is on it. If the answer is almost entirely developers and safety researchers, you know which risks will be defined.

Whether evidence requirements rise. A healthy sign would be policy proposals that name the harm, cite the kind of evidence behind it, and say how success would be measured. An unhealthy sign would be more statements, more pledges, and fewer numbers.

I also think it is worth reading debates like this one with some care about your own reactions. Fear is contagious, and so is cynicism. Both can substitute for thinking. The short version is that you do not have to outsource your assessment of risk to either the alarmed or the dismissive.

A Practical Way to Hold the Debate

If I had to give a reader one habit from all of this, it would be to translate. When you hear "existential risk," ask what specific thing is being described. When you hear "AI safety," ask whose safety and from what. When you hear a promise to regulate, ask what the rule would require of whom, and by when.

Translation is slow, but the clearer the risk, the easier it is to build a real control against it.

The argument AI Now summarizes is, as I read it, an invitation to trade a little awe for a little precision. Organizations that make that trade early may be better prepared for whatever the rules turn out to be.

Frequently Asked Questions

What are existential fears about AI?

They are concerns that advanced AI systems could cause human extinction or permanent loss of human control. The best-known statement is the Center for AI Safety's May 2023 sentence calling mitigation of extinction risk from AI a global priority alongside pandemics and nuclear war.

How do existential fears affect AI regulation?

They push regulators toward rules aimed at the largest frontier models, often defined by training compute. Examples include the 10^26 operations threshold in Executive Order 14110 and California's SB 1047, and the 10^25 floating-point operations presumption of systemic risk in Article 51(2) of the EU AI Act.

What is Sarah Myers West's argument in the AI Now piece?

According to AI Now's summary, as panic about AI's capabilities turns to regulating AI companies, West argues for clarity: breaking down what the actual risks are so that they can be properly mitigated.

Should organizations that only use AI tools care about this debate?

Yes, because most obligations for ordinary organizations come from how AI is deployed, such as in hiring, credit, or education, and not from frontier model rules. The framing of the debate affects which harms regulators prioritize and which rules you will eventually face.

Does focusing on present harms mean ignoring catastrophic risk?

No. A society can evaluate frontier models for catastrophic misuse and also require impact assessments and accountability for deployed systems. The point is to name each risk specifically, so the remedy fits the problem.