Davies' recent warning that agentic AI is generating "a new generation of conduct risk" in insurance deserves more attention from boards than it has had so far, and not because the technology itself is especially new. Insurers have been using machine learning in underwriting and claims triage for years. What's different about agentic AI is the word agentic: these systems don't just score a risk or flag a recommendation for a human to act on, they act. They authorise, they decline, they negotiate, they communicate with the customer directly, and they do it continuously rather than at the single, reviewable moment a traditional model output would have been checked.
Three problems worth sitting with
Davies identifies three specific problems with that shift, and I think all three are worth sitting with rather than skimming past. The first is opacity: as these agents chain decisions together autonomously, it becomes genuinely harder to trace a given outcome back to the specific input or piece of logic that produced it. The second is reinforcement bias: an agent that learns and adapts as it operates can intensify a problematic pattern over time rather than repeat a fixed, auditable error, which means a small, defensible bias at launch can become a large, indefensible one eighteen months later without anyone having changed anything on purpose. The third is what Davies calls autonomous deviation: an agent operating with real independence can simply drift from the guidelines it was set up to follow, in ways that may not be visible until the drift has already produced a customer outcome someone has to explain.
Why this is a Consumer Duty problem, not just an AI one
Here is why I think this matters as much for Consumer Duty as it does for AI governance, and why I'm increasingly convinced those two things cannot sensibly be treated as separate committees reviewing separate risks. An agent that drifts from its guidelines and starts declining claims it shouldn't, or pricing a cohort of customers unfairly, or missing a vulnerability indicator it was supposed to catch, is not producing a technology incident that happens to have a customer consequence. It is producing a conduct incident and a governance incident at the same moment, from the same root cause, discoverable through the same audit trail if anyone had built one.
Firms that review AI governance in a technology risk forum and Consumer Duty outcomes in a separate customer outcomes forum, on different cycles, reporting to different people, are structurally set up to catch each half of that story late — if they catch it at all.
Continual compliance is the right instinct — but not far enough
Davies' proposed answer is what they call continual compliance: treating oversight of an agentic system as an ongoing, live process rather than something signed off once at deployment and revisited on a fixed annual or biannual cycle. I think that's the right instinct, but I'd push it one step further than a compliance function's monitoring cadence. The people who need to see agentic AI outputs on a continual basis aren't only the ones checking whether the model still works. They're the ones who own the customer outcome the Consumer Duty makes them accountable for. If an agent is making live decisions that affect price, eligibility, or claims outcomes, the audit trail Davies calls for needs to be legible to a Consumer Duty champion, not only to a model risk officer.
The governance question every insurance board should ask this quarter
This is, I'd argue, the most practical governance question an insurance board can ask this quarter about any agentic AI system already in customer-facing use: who reviews its outputs, on what cycle, and do both the AI governance forum and the Consumer Duty owner see the same evidence at the same time, or only after something has already gone wrong? If the answer involves two different teams looking at two different dashboards on two different schedules, that's not a governance structure. That's a gap waiting for an agent to find it first.