AI Agents: More Medicine, Less Software?

Agents on Prescription?

In my role at GDS, I have been exploring how artificial intelligence, and especially agentic AI, might change the way people interact with public services. As we dig deeper into the the capabilities and limitations of what happens when this technology meets policy and public services, we find ourselves facing this challenge:

How do we safely introduce probabilistic technologies into a system that fundamentally depends on deterministic outcomes?

Government cannot “probably” award someone a benefit or “approximately” determine their status. The final outcome must remain deterministic.

This question is about trust, legitimacy (the public’s justified belief that the state’s decisions are fair, rule governed, and authoritative), and the nature of government itself. As we’re working through it, I find myself increasingly reaching for an unexpected analogy - one that sits outside the traditional world of digital services.

Government depends on rules; AI depends on probabilities

Much of the state’s legitimacy comes from its ability to apply rules consistently. Whether someone is eligible for a benefit, entitled to a licence, or permitted to take a specific action, government systems must provide clear, repeatable, legally grounded answers.

Even when the underlying law is complex, the outcome must be:

• consistent
  explainable
  fair
  contestable
  ultimately binary: eligible or not, permitted or not.

By contrast, AI agents - particularly those built on large language models - operate probabilistically. They infer, predict, and approximate. That makes them incredibly powerful at:

• explaining information
  navigating complex policy
  interpreting citizen queries
  helping people understand their options

But it also means they cannot reliably act as the final arbiter of legally consequential decisions.
In other words:

• AI Agents will help citizens understand the rules.
  AI Agents might help citizens act on the rules.
  AI Agents must not determine the rules.

We are exploring whether and how these thresholds help us design AI Agents that are useful and valuable - while remaining safe to use.

Not every AI action carries the same risk

Through our explorations, another core insight has emerged. The risk of an AI agent in government is not defined by the capability of the model - it is defined by the context and impact of the task it is set to work on.

AI looks like software but behaves more like medicine

At first glance, AI systems appear to sit comfortably within the traditional digital and technology domain. They use data. They run on cloud infrastructure. They are delivered through familiar front-end interfaces.

But in practice, their behaviour is markedly different from rule-based systems. AI agents are probabilistic. They will behave differently for different people in different contexts. The efficacy of the model can drift over time and requires ongoing monitoring. And most crucially, in certain situations, might cause unintended yet significant harm.

We, socially, already have a trusted mechanism for releasing technologies like these into the public domain - but they aren’t digital technologies - they are pharmaceutical technologies. When viewed through this lens, the closest analogy for AI agents is not a new piece of software. It is a drug.

Pharmaceuticals are powerful, sometimes unpredictable, potentially risky technologies that society has learned to introduce and manage responsibly. They go through:

• controlled trials
  safety and efficacy assessments
  phased deployment
  licensed prescribers
  post-market monitoring
  strict rules about who can use what, and under what conditions

This domain offers something digital government has never had to develop before:
a mature governance model for probabilistic technologies with non-zero risk.
I am wondering whether the AI ecosystem in government may need something similar?

Anyway...