Administrative Law and AI's Overconfidence Problem

There's a particular kind of danger that comes not from a machine that fails, but from a machine that sounds absolutely certain — even when it's wrong. This is the problem at the heart of a growing debate in administrative law, and it's one that regulators, agency officials, and anyone building AI-assisted decision systems urgently need to understand.

In March 2026, legal scholar Cary Coglianese published a pointed analysis in The Regulatory Review arguing that while AI can genuinely support many government tasks, officials must remain alert to the hazard of "overly confident results." The piece wasn't a broadside against AI adoption. It was something more nuanced: a warning about a specific failure mode that administrative law is structurally ill-equipped to handle — and that agencies may be walking into without fully appreciating the risks.

This article unpacks that argument, extends it, and explores what it means practically for every institution — public or private — deploying AI in high-stakes decision-making.

What Is AI Overconfidence, and Why Does It Matter in Regulatory Contexts?

AI overconfidence refers to the tendency of machine learning systems — particularly large language models and probabilistic classifiers — to express high certainty in outputs that are, in fact, unreliable. The system doesn't say "I'm not sure." It gives you a clean answer, a confident recommendation, or a polished summary, even when the underlying reasoning is flawed, the training data was insufficient, or the question fell outside the model's reliable domain.

This is not a fringe edge case. Research from MIT and Stanford has consistently shown that large language models miscalibrate confidence, presenting incorrect answers with the same fluency and assertiveness as correct ones. A 2023 study in Nature found that GPT-4 exhibited overconfident outputs in approximately 30% of tested medical diagnostic scenarios — a domain structurally similar in risk profile to regulatory adjudication.

In ordinary commercial applications, overconfidence is an inconvenience. A chatbot gives a wrong product recommendation; a user notices and corrects it. But in administrative law — in the world of permit decisions, enforcement actions, benefit determinations, and rulemaking — the consequences of unchecked AI confidence are categorically different.

The Administrative Law Problem: Procedural Legitimacy Meets Probabilistic Output

Administrative law is built on a set of foundational requirements: reasoned decision-making, transparency, notice, and the opportunity to challenge. An agency cannot simply issue a decision. It must explain its reasoning in a way that is reviewable by courts, challengeable by affected parties, and traceable to evidence in the record.

This architecture was designed for human deliberation. It assumes that the decision-maker can articulate why they reached a conclusion, what evidence they weighed, and where uncertainty existed.

AI systems — particularly modern deep learning models — operate differently. They produce outputs from patterns across billions of parameters, and those outputs often cannot be traced back to a legible chain of reasoning. When an AI system tells a regulator "this application should be denied" with 94% confidence, that number is not the same as a human expert saying "I'm 94% sure." It is a statistical artifact of model architecture, training distribution, and calibration choices that most agency officials are not equipped to interrogate.

Coglianese's argument, as I read it, points to a structural mismatch: the confidence register of AI output is incompatible with the epistemic humility that legitimate administrative decision-making requires.

Three Ways Overconfidence Corrupts Administrative Process

1. It Short-Circuits the Duty to Explain

When an AI system delivers a high-confidence output, it creates a cognitive shortcut for the human official reviewing it. The psychological literature on automation bias — the tendency of humans to defer to automated systems, particularly when under time pressure — is extensive and troubling.

A landmark 2014 study by Parasuraman and Manzey found that automation bias leads trained professionals to miss critical errors at rates exceeding 70% when the automated system was presenting confident outputs. In agency settings, this means the AI's overconfidence can effectively become the agency's reasoning — without anyone consciously choosing to make it so.

The legal risk is significant. A court reviewing an agency decision under arbitrary and capricious review doesn't care that an AI said so. It wants to know that a human official, exercising genuine deliberation, made a reasoned choice. If the record shows only that the agency adopted an AI recommendation without meaningful scrutiny, that decision is vulnerable.

2. It Creates Invisible Discrimination Risks

Overconfident AI systems frequently present their outputs uniformly across population groups, even when their accuracy rates are not uniform. A model might be 92% accurate overall — but 78% accurate for one demographic subgroup and 97% accurate for another. The average confidence figure masks the disparity.

According to a 2022 audit by the Algorithmic Justice League, facial recognition systems used in government contexts showed error rates up to 34% higher for darker-skinned individuals compared to lighter-skinned individuals — despite vendor claims of high overall accuracy. When agencies rely on such systems without disaggregated accuracy analysis, they aren't just making errors. They are systematically encoding inequity into government decision-making, often with no awareness of it.

3. It Undermines the Notice-and-Comment Process

Rulemaking under the Administrative Procedure Act requires agencies to disclose the basis for proposed rules and invite public comment. If an agency uses AI to analyze public comments, summarize research, or model policy impacts — and that AI system produces overconfident outputs that go unchallenged — the rulemaking record becomes corrupted in ways that participants cannot detect or contest.

You can challenge a human expert's methodology. You can cross-examine their assumptions. But if an agency's AI summarization tool confidently misrepresented a set of public comments, and no one knew the tool was doing so, affected parties may have had no meaningful opportunity to correct the record.

Comparing AI Confidence to Legal Standards of Certainty

This table illustrates the misalignment between how AI systems express certainty and the legal standards that administrative law actually applies:

The column that matters most is the last one. In nearly every dimension that administrative law cares about, the surface-level confidence expressed by AI systems is either misleading or legally meaningless.

What Coglianese Gets Right — and What the Debate Still Needs

Coglianese's intervention is valuable precisely because it doesn't argue that AI has no place in administrative decision-making. That argument, made by AI skeptics, is increasingly untenable. Agencies are under-resourced. Rulemaking dockets have exploded. AI tools that can process large volumes of text, flag patterns, or model regulatory scenarios represent genuine value.

The more important argument — the one Coglianese is making, and the one I think deserves wider traction — is about epistemic position. The question isn't whether to use AI. It's whether officials are maintaining the right posture toward AI output: treating it as input to judgment, not as a substitute for it.

The most dangerous moment in AI-assisted governance is not when the system fails visibly. It is when it succeeds confidently enough, often enough, that officials stop asking questions.

What the debate still needs, and what neither Coglianese's piece nor the broader administrative law literature has fully grappled with, is a taxonomy of AI use cases by overconfidence risk. Not all agency AI applications carry equal danger.

A Framework for Thinking About Overconfidence Risk in Agency AI Use

Here's a rough framework I'd suggest for categorizing AI use in administrative contexts:

Low overconfidence risk: - Document retrieval and indexing - Translation of routine communications - Scheduling and workflow automation - Flagging submissions for human review

Moderate overconfidence risk: - Summarizing large bodies of public comments - Drafting initial versions of guidance documents - Pattern detection in enforcement data - Modeling regulatory impacts using historical data

High overconfidence risk: - Adjudicating individual benefit or permit applications - Generating explanations for agency decisions - Predicting recidivism, risk, or compliance likelihood for individuals - Classifying regulated entities for enforcement targeting

The distinction that matters across these categories is not just accuracy — it's the consequence of error combined with the detectability of that error. High-risk applications share two features: the error can cause serious harm to an identifiable person or entity, and the AI's overconfidence makes the error hard to catch before that harm occurs.

What This Means for Institutions Beyond Government

Although Coglianese's argument is framed in administrative law, the logic generalizes broadly. Any institution using AI in high-stakes decisions where the decision must be:

1. Legally defensible
2. Non-discriminatory
3. Traceable to evidence
4. Challengeable by affected parties

...is operating in an environment where AI overconfidence is a first-order problem, not a technical footnote.

This includes hospitals making clinical decisions, financial institutions making credit determinations, universities making admissions decisions, and private employers making hiring decisions. In each case, regulatory frameworks — whether civil rights law, financial regulation, or sector-specific compliance regimes — impose requirements that AI overconfidence can silently undermine.

The practical upshot is this: institutions that deploy AI without building explicit mechanisms to interrogate and challenge model confidence are not just taking a technical risk — they are taking a legal and reputational one.

What Good AI Governance Looks Like in This Context

Rather than concluding with abstract principles, let me be specific about what institutions need to do differently:

Require confidence calibration reporting. Before deploying any AI system in a consequential decision context, demand from vendors not just overall accuracy metrics, but calibration curves — evidence that the model's stated confidence levels actually correspond to its accuracy rates across the relevant population.

Build adversarial review into the workflow. Designate someone whose explicit job is to challenge the AI's output before it informs a decision. This is not bureaucratic overhead. It is the minimum necessary to maintain the reasoned decision-making standard that law requires.

Document uncertainty, not just conclusions. When AI is used to inform a decision, the record should reflect the system's confidence level, the conditions under which that confidence was generated, and the human official's independent assessment. Hiding the AI's role doesn't make the legal problem go away — it makes it worse.

Invest in disaggregated accuracy analysis. Aggregate accuracy figures are insufficient for any use case touching individual rights or opportunities. Require accuracy data broken out by the demographic variables that matter legally and ethically.

Treat high confidence as a red flag, not a green light. This is counterintuitive but important. When an AI system is expressing very high confidence in a novel, complex, or contested question, that should prompt more scrutiny, not less. Overconfidence is most dangerous precisely when it feels most reassuring.

The Deeper Institutional Challenge

There's a reason I find Coglianese's argument important beyond its immediate legal context. It touches on something fundamental about how institutions are going to relate to AI over the next decade.

We are in a period where AI systems are being presented — and are presenting themselves — as authoritative. The UX of these tools is designed to inspire confidence. They don't hedge. They don't equivocate. They give you an answer. That design choice is commercially rational. But it is institutionally dangerous when the institution's legitimacy depends on maintaining genuine human deliberation and accountability.

The institutions that navigate this well will be the ones that build cultures of intelligent skepticism toward AI output — not resistance to AI, but structured, proceduralized doubt. They will treat AI the way good lawyers treat expert witnesses: as valuable input that must be interrogated, qualified, and placed in context before it can inform a consequential judgment.

The institutions that navigate this poorly will be the ones that let AI confidence collapse the space between analysis and decision — and find out, in court or in public, that their delegated judgment was never really theirs.

For more on how AI is reshaping decision-making across institutions, explore related analysis at prepareforai.org.

Last updated: 2026-03-29

Source referenced: Coglianese, C. (2026, March 23). Administrative Law and AI's Overconfidence. The Regulatory Review. https://www.theregreview.org/2026/03/23/coglianese-administrative-law-and-ais-overconfidence/