Two things landed in the same week that, read together, should worry every board that has quietly told itself AI is under control because a person can technically step in.
The first is a report from Parliament's Joint Committee on Human Rights, published on 14 September, calling for a comprehensive new AI Bill: a risk-based framework, a single independent regulator with real enforcement powers, and mandatory transparency across the AI lifecycle. Buried in that report is a line I think matters more than the headline recommendation. The Committee states plainly that "the mere presence of a 'human in the loop' is not sufficient to constitute meaningful human involvement." Existing data protection safeguards on automated decision-making, it says, need strengthening precisely because they let firms treat the existence of a review step as proof of control, rather than asking whether that review does anything at all.
The second is research published by the Payments Association on 23 September, and it is the empirical version of the same warning. Ninety-three per cent of UK financial services firms told researchers that AI has improved outcomes for vulnerable customers. Firms rated their own AI inclusion readiness at 4.1 out of 5. And yet 35 per cent of those same firms do not separately monitor vulnerable-customer outcomes at all. Set that alongside the customer-side finding: 17 per cent of vulnerable consumers said they could not complete a banking task because an AI or digital tool failed them, against 5 per cent of non-vulnerable consumers. Only a third of firms report specific provisions for customers going through an adverse life event, compared with two-thirds for older customers. As the Payments Association's chief executive, Emma Banymandhub, put it, the industry is confident AI is working for vulnerable customers while a third of it isn't checking.
The same failure mode from two directions
Both are describing the same failure mode from different directions. Parliament is saying: don't let the presence of a control stand in for evidence the control works. The market research is showing, in numbers, exactly how that substitution happens in practice.
Put the two stories next to each other and the pattern is obvious. Both are describing the same failure mode from different directions. Parliament is saying: don't let the presence of a control stand in for evidence the control works. The market research is showing, in numbers, exactly how that substitution happens in practice — firms have a reviewer, a fallback, an escalation path, and they treat those as sufficient without ever measuring whether vulnerable customers are actually being served by them.
Why this is a board reporting failure, not a compliance shortfall
I want to be specific about why this matters for boards rather than compliance teams. Consumer Duty already asks firms to evidence outcomes, not intentions. The natural reading of that obligation, for AI-assisted journeys, is that "we have a human who can intervene" was never actually the bar. It was always meant to be "we can show the outcome was good, and where it wasn't, we can show why and what we did about it." What's changed is that the gap between those two standards is now being measured from outside the firm, not just asserted from inside it. When 35 per cent of the market cannot separately report vulnerable-customer outcomes for AI-touched journeys, that isn't a technical shortfall. It is a board reporting failure, because it means the board itself cannot see the thing it is being asked to be accountable for.
The question to put to your board this quarter
There is also a harder question sitting underneath this, and it's the one I'd put to any board this quarter: what would you actually show a regulator, today, as proof that a human reviewing an AI-assisted decision exercised real judgment rather than rubber-stamping an output? Not the policy that says a human reviews it. Not the audit trail that shows a person clicked approve. The evidence that the review changed something, caught something, or could have. If the honest answer is that you don't have that evidence readily to hand, you are not alone, and you are also not as protected as your governance paperwork suggests.
None of this is an argument against using AI in customer-facing journeys. It's an argument against confusing the existence of oversight with the fact of it. The Committee's report is a legislative signal that this distinction is about to be tested externally rather than assumed internally. The Payments Association's numbers are the current state of the gap it will be tested against. Firms that close that gap on their own terms, before it's forced on them, will find it a great deal cheaper than firms that wait to be asked.