AI can write proofs, but only people can stand behind choices

Mathematicians and software engineers are, understandably, anxious about AI. Each is a way of life as much as a body of knowledge, and threats to ways of life unsettle us, especially when we’ve been formed, in whole or in part, within and by them.

However, three separable things are at stake here: (1) a livelihood, (2) a professional status, and (3) a vocation. The first two are the weakest grounds for alarm, since mathematics was operating at a large global deficit before AI arrived, with far too few people who can think mathematically and far too few areas of life formalized that could be, and cheaper output in a field that short of output is likelier to shrink the deficit than to produce a glut. In fact suddenly there’s a flood of new results that need to be learned, explained, and internalized, so in some sense the shortage of workers relative to the work has grown with AI’s arrival. Software engineers are in much the same position.

Vocation is another matter. The years spent learning to find things, the pleasure of having found one, the sense that a life went into a result. That loss is real and it cannot be easily argued away, nor should we really try to do so.

But the public argument keeps running it together with an epistemic claim, that a result found by a machine is somehow a lesser kind of mathematical knowledge than one found by a person, and that epistemic claim doesn’t hold up.

A proof carries its own warrant

The distinction between results derived by people and results derived by AI is idle in mathematics, not because AI and people are the same kind of thing. I happen to believe they are the same kind of thing more so than most, but that belief is stronger than my argument here needs. AI is wholly a human product, built from our algorithms and trained on our cultural output, but telescopes and printed books are human products too, and nobody thinks they’re like us.

The narrower point is key.

Even if AI does differ in kind from us, those differences have no bearing on whether a mathematical result is true, because a proof’s warrant is ultimately in the proof, not in the prover. A correct derivation is correct whoever or whatever wrote it. Math and philosophy and science, generally, have kept the prover’s identity out of a proof’s validity for as long as they’ve been rigorous disciplines, maybe longer.

It also helps to notice how little of mathematics any mathematician produces. Nearly every result a working mathematician knows came from somewhere else: a paper, a lecture, a colleague, a textbook or proof or note written by someone long dead. Their relation to most of their own field is receptive. They take in results, check what they need to check, and build on the rest. That receptivity is the same whether the result arrives from a person or from a person’s machine. A mathematician reading GPT (and its machine-generated proof) is doing exactly what they did when they read Gauss or Gowers.

Results nobody understands aren’t new either

A second worry is that AI will produce results that are true and that people don’t understand. The problem is older than AI. The four colour theorem was proved in 1976 with a computer check of nearly two thousand configurations that no person could do by hand. The classification of finite simple groups runs to something like ten thousand pages across hundreds of papers, and it’s doubtful any single person has absorbed all of it.

Mathematicians already live with results that are verified but not understood fully by any one of them. Even assuming a difference between don’t understand, in fact, and can’t understand, in principle, it’s not fully clear how to classify a 10,000 page proof.

The worry is sometimes put as a claim about the limits of human reason, though the claim is rarely stated precisely. It’s usually something smaller: a claim about the limits of human time and attention. That is, the result can be understood in principle and isn’t understood in practice. But that, again, describes most of regular mathematics for any one mathematician.

William Thurston argued (“On Proof and Progress in Mathematics”, in 1994) that the product of mathematics is human understanding, not a body of theorems. If he’s right, and he is for some mathematicians, machine-generated results don’t remove the mathematician. What they do is make the mathematician’s real job explicit: turning verified results into understanding, deciding which questions are worth asking, and seeing which of a thousand true statements matters.

AI does move mathematicians closer to everyone else with respect to the results they might have found themselves, since they now receive some of those results as non-mathematicians have always received all of them. That is, with respect to something like “mathematical reality” or as many believe “reality as such”, AI makes mathematicians a bit more ordinary, a bit less like a priesthood of those in special relation with “things as they really are”, etc. But in fact, with respect to the rest of mathematics, that is, the math they weren’t working on at or near the frontier, they were always already in that position. AI changes little in this regard.

Mathematics is the clearest case, and that’s the lesson

Mathematics proves there’s at least one field where a machine’s work can be checked without trusting where it came from. Checking a proof is cheap compared with finding it, and with a proof assistant like Lean the check is close to mechanical. The four colour theorem itself was formalised in Coq in 2005, which turned a proof people had to take partly on trust into one a small, well-audited program confirms. Where checking is that cheap and that independent of the source, the origin of a result really is irrelevant, at least epistemically, if not always vocationally.

But most knowledge work isn’t like this. A legal opinion, a credit decision, a reconciliation of accounts, or a medical diagnosis is accepted largely on the testimony of expertise, because the person receiving it can’t re-derive it at reasonable cost. In those fields origin isn’t irrelevant. Who produced the work stands in for how reliable it is, because there’s no cheaper way to tell.

So far this looks like a story about verification: where checking is cheap, origin stops mattering, and where it’s expensive, origin matters. That’s true, but it’s not the whole story, and for the rest of us, directly, mathematics is what shows the rest of it.

Checking a claim isn’t like accountability or answering for a choice

Mathematics mostly produces claims. A false theorem gets found, retracted, and absorbed by the field. Nobody’s mortgage really depended on it.

Most knowledge work produces choices, or produces claims that choices are made on: a loan is approved, a payment goes out, a patient is treated, an invoice is posted to the ledger. Accountability attaches to decisions. It’s the practice of asking someone to give reasons, holding them to account when the reasons were bad, and imposing a cost they actually bear.

The difference in kind set aside earlier, as irrelevant to whether a proof is valid, is decisive for who answers for a choice. As P.F. Strawson argued in “Freedom and Resentment” (1962), responsibility is constituted by reactive attitudes like resentment and blame. And those attitudes need a target who could have cared about what they did. An agent isn’t that kind of target. Nobody resents a model in any sense that changes its conduct or costs it anything. No theory of AI’s future development proposes that we will.

The nearest thing to an exception is the European Parliament’s 2017 suggestion of a legal status of “electronic persons” for autonomous robots, and the comparison it relied on, the corporation, shows the limit. A corporation can be held accountable only because the accountability eventually lands on people who can be sanctioned and on assets that can be taken. An agent given legal personhood would work the same way, through its owner or its insurer. That’s a way of assigning accountability to someone else, not accountability of the agent per se.

This is where verification and accountability come apart. Verification tells you whether a claim is true. Accountability settles who answers for a choice. A perfectly checked output, acted on, still leaves the question of who decided to act on it, and no amount of checking answers that question.

Two axes, not one

Once the two are separated, knowledge fields sort by two axes rather than one: how expensive their work is to verify, and how much accountability the choices made require.

In the low-accountability column agents can take most of the work. In mathematics the human role becomes understanding and taste. In drafting and exploration errors are cheap and someone downstream filters them. The expensive, high-accountability corner is the hardest case, and it’s where people stay most clearly in the loop.

The table’s cell that matters most is the one a verification-only account gets wrong: cheap verification with high accountability. A reconciliation can be checked mechanically, line by line, against the bank statement and the ledger. Even so, someone still has to sign for it. Cheap checking makes the signature easier to give but doesn’t make it unnecessary.

The rows aren’t fixed. Formalizing a domain is how it moves from the expensive row to the cheap one, and this is where the mathematical deficit stops being a matter for mathematicians alone. Every area of life that gets formalized becomes one where an agent’s work can be checked without trusting it, which lets agents act there more widely, which in turn creates more choices that someone has to answer for. Mathematicians aren’t only the example in this argument. But they’re among the people who decide how fast the cheap row grows.

This bears directly on WunderOS, an operating system for back-office agents. PLRN-000 argued that a system’s speedup is capped by the share of the work that still needs a human, and that at the far reach of autonomy nobody can verify the agent or answer for it. The mathematics case breaks the first half of that second claim, since verification cost belongs to the artifact rather than to the agent’s expertise. The accountability argument narrows the second half. The void where nobody answers isn’t a property of autonomy. It’s what happens when nobody records who decided to stop checking, and that decision is itself an accountable choice. I’ve revised the note to say so; the details are in its revision record.

What’s left for knowledge workers

The change mathematicians are living through is happening to other knowledge workers, too. It seems that, generally, experts will relate to their own field less as producers and more as informed receivers. Expertise becomes the ability to judge, select, and integrate work they didn’t do. Most knowledge work already ran on receiving other people’s output. What changes is the proportion.

What’s left for people isn’t, primarily, the work agents can’t do. It’s the role agents can’t occupy, and that role grows as agents take on more of the work. The more choices an organization makes on agent outputs, the more it needs clear lines from each choice to a person who answers for it. Mathematicians feel this less than most, because their field makes so few choices of that kind.

That role of accountability-in-chief has a condition attached.

Accountability without judgement is only a signature: a controller who signs a reconciliation they can’t evaluate isn’t answering for anything. So the role agents can’t occupy needs people who know the work well enough to tell when it’s wrong. Knowledge of that type has mostly been learned by doing the work, through years of producing less than ideal output, having it reviewed, and slowly becoming the person who sees the error at a glance.

The gap between someone who once had an agent build a thing and someone who has done the work for decades is real, and today’s experts are on the right side of it. The open question is how the next generation crosses it once agents do most of the production that used to train them.

Outside mathematics, the useful questions aren’t who produced the work, but who can check it, at what cost, and who answers when the choice made on it is wrong. Mathematics lets you ignore the last question because nobody acts on a theorem in the same way a firm acts on a posted invoice. Most fields can’t ignore it, and the infrastructure around agents has to answer it: provenance for every output, verification where it’s cheap enough to run, and a named person behind every choice that isn’t.