Sam Freedman asked a question in his newsletter this week that a lot of people in and around government are quietly asking themselves. Large parts of the state are very worried about being overwhelmed by claims, applications, complaints, appeals and objections all written by AI.
I wrote in February about AI and the end of friction as a policy lever. This is that, arriving on schedule. It’s sometimes called agentic flooding, although it doesn’t require mass adoption of AI agents to be a big problem.
Since writing that piece I’ve talked to dozens of public servants about it. Most recognise the problem. Most are reaching for tactical fixes such as increasing numbers of staff, or introducing more friction into the service design. Most know this won’t be enough.
A more fundamental rethink is required.
Most public services in Britain run on what you might call a declarative model. You make a claim. You fill in the form, set out your circumstances, attach your evidence, and the state decides whether to believe you. The whole thing quietly assumes that writing the claim takes effort, and that the effort is a rough proxy for sincerity and need. That quiet assumption is often entirely false.
AI removes the effort. Once a convincing claim, objection or appeal costs nothing to produce, a declarative system has little way of telling who to believe. So it drowns.
There may be a bolder alternative: shift the default service design from declarative claim to proactive confirmation.
What if government knows enough about your circumstances, with your consent, to make you an offer rather than demand you make a claim? “We think you’re entitled to this. Tell us if we’ve got it wrong.” Tax returns are already heading this way. The state pension is mostly already there. A lot more may have to follow.
None of this is free, and we should be clear-eyed about what gets lost, and the risks this introduces.
A proactive system needs rules a machine can apply. So any remaining policy grey areas, the space left for a human to weigh an individual’s particular circumstance, may get codified away. That is not a good thing. People’s lives are complex and the rules never quite fit. I fear it is probably inevitable all the same.
The right to appeal will get severely curtailed, because AI will appeal by default. Why wouldn’t it? The appeals that survive will come with a price tag or some other form of friction: a fee, a lawyer, some other hoop to jump through. That will be bad, and it will cause real suffering for people who had a case.
And no, putting “humans back in the loop” won’t rescue us, or not for more than a few months. AI is constrained to online. AI-generated conversational phone and video calls are already a thing and will only get better. And who’s to say a disabled person using one to deal with the state isn’t making an entirely reasonable adjustment? It’ll be tricky to write a policy that bans them.
So I fear it is arms races against AI all the way down, for as long as we cling to a declarative model of public service delivery. As a journalist from Les Echos put it to me with ill-disguised gallic relish: “The UK will suffer first because you govern in prose.”
Our forms, our guidance, our decisions and our appeals are mostly words. Words are the thing large language models are best at.
Watermarking won’t save us either. The EU’s AI law requires AI companies to watermark the text their models generate. OpenAI, very begrudgingly, agreed to comply last week. The labs all hate it, probably mainly for liability reasons. Had we still been in the EU it would at least have given us some options to explore. That said, locally-hosted open LLMs like qwen won’t carry a watermark, or will easily be persuaded not to.
(BTW your competent organised fraud gang is not typing prompts into ChatGPT. It is likely building itself a rack or two of servers running a big open model with the guardrails turned off.)
There is a more uncomfortable point underneath all of this. “Making people answer lots of detailed question to constrain demand” has always been horribly iniquitous. It has also been a politically convenient lever to pull, which is why it has lasted so long.
Look at what the existing prose-related friction has actually been doing. The people most in need of public services are the people most likely to be excluded by the linguistic hoops that individual services put in the way to “manage demand”. Those people and their unmet needs did not disappear. They turned up somewhere else in the system, later and more expensively. Displacement demand, redux.
AI slop hasn’t created that dystopia. It has just made it impossible to ignore.
Resetting the trust relationship between citizen and state has been essential for years. It is also the precondition for services that make offers rather than demand claims. Maybe this is the crisis that finally catalyses the fundamental public sector reform that’s been essential for so long?
Leave a Reply