You Don't Have to Understand the Model. You Do Have to Explain It

Explainability is not a technical property of a model — it is an operating capability: the firm's ability to describe, on demand, why an AI-influenced decision came out the way it did.

A regulator will not ask you to explain the mathematics of your model. They will ask why this customer got that outcome — and “the system decided” is the one answer guaranteed to turn a question into an investigation. Explainability is the difference between the two.

This series has moved from funding decisions to infrastructure, data, training, audit, value, accuracy, agents and — across this final run — incidents, outsourcing, financial crime, the front office and workforce competence. It closes where regulated AI is ultimately tested: the moment someone with authority asks a firm to account for a decision a model helped make.

The JFSC's July 2026 guidance is notably practical on this point. It states that firms do not need to understand every technical detail of their AI, but must be able to describe the main factors driving an outcome. That single sentence reframes explainability from a data-science problem into a governance capability — and this piece is about building the capability, not decoding the model.

Explainability is where the abstract promise of “responsible AI” meets a concrete, adversarial test: someone with authority, after the fact, asking a firm to justify a specific outcome. Everything else in a governance framework can look complete on paper and still fail this moment, because this moment does not reward documentation — it rewards the ability to give a clear, true, proportionate account on demand.

Explainability is a capability, not a property of the model

Firms often treat explainability as something a model either has or lacks — a feature to be procured, a diagram to be filed. That framing fails the moment it is tested, because the question a regulator asks is never “is your model explainable?” It is “explain this outcome.” Those are different demands. The first is about the model; the second is about the firm.

Explainability is not a box the vendor ticks. It is a sentence the firm can say. The guidance's standard — describe the main factors driving an outcome — is a test of the firm's operating capability, not the model's internal transparency. A perfectly interpretable model is useless if no one in the firm can retrieve, on demand, why it produced a particular result for a particular customer.

The register is where explainability is won or lost

The guidance ties explainability directly to an up-to-date AI use-case register, and specifies the fields that register must hold. This is the most concrete, most ignored instruction in the entire document — concrete because it lists exactly what to record, ignored because a register feels like administration until the day it is the only thing standing between the firm and an unanswerable question.

Table 1 sets out the register fields the guidance names. The discipline is not in creating the register once; it is in keeping it true as models, data and owners change — which is why the register belongs to governance, not to a project that ends.

Table 1 — The AI register as an explainability instrument: the fields the guidance requires, and the question each one answers when a regulator calls.

The third field is the one that does the real work. “Main data inputs” is the guidance's own language for the main factors driving an outcome — which means a firm that keeps this field accurate has already, in effect, written the first line of every explanation it will ever be asked to give. A firm that leaves it vague has guaranteed that every explanation starts from a blank page under time pressure.

Describe the factors, not the mathematics

The most helpful sentence in the guidance is the one that says firms need not understand every technical detail. It means the goal is not to make a data scientist of the MLRO or the board. It is to ensure that for any material decision, someone can name the handful of factors that drove it in language a customer, an auditor and a regulator would all accept.

Explainability is not the model showing its workings. It is the firm owning its reasons. In practice this is a sentence of the form: “this outcome was driven mainly by these inputs, it was classified at this materiality tier, this named person oversees it, and here is the record of the human who reviewed it.” A firm that can produce that sentence for a material use case has met the standard. A firm that reaches for the vendor's documentation instead is not yet meeting the standard.

Explainability scales with materiality, like everything else

A firm cannot explain every AI-influenced output to the same depth, and the guidance does not ask it to. Explainability, like every other requirement in the document, is risk-based: the depth of explanation a firm must be able to produce should track the materiality tier of the use case. A low-impact tool that reformats internal text needs a register entry and little more. A higher-impact model that shapes a customer outcome or a regulatory report needs an explanation a firm can defend line by line.

This is what stops explainability becoming an impossible universal burden. For low-impact use cases, the register is the explanation. For material ones, the register is the starting point and the firm must be able to add the specific reasoning: which inputs drove this particular outcome, which human reviewed it, and on what basis they let it stand. Grading the explainability obligation to materiality is not a way of doing less — it is how a firm concentrates its finite explanatory effort on the decisions where being unable to explain would actually cost it. A firm that tries to explain everything equally will explain the trivial thoroughly and the material inadequately, which is exactly the wrong way round.

The last mile of a jurisdiction built on trust

Everything this series has argued converges here. Incident response gives you the record of when the model failed; outsourcing oversight gives you visibility into the models you did not build; financial-crime and front-office governance put a named human on the decisions that matter; competence ensures that human can actually do the job. Explainability is what turns all of it into an answer — the capability to stand behind an AI-influenced decision when it counts.

Jersey's advantage was never going to be the cleverest models; larger centres will always outspend the island on technology. The advantage is the one the Playbook keeps pointing to — responsible adoption as ethical differentiation in a jurisdiction that sells trust. A firm that can explain its AI, proportionately and on demand, is offering exactly what an international finance centre exists to provide: not the most automation, but the most accountable automation. That is the last mile, and it is the one worth winning.

If this series has one practical instruction to leave behind, it is this: keep the register true. Not as an administrative chore, but as the single artefact that turns every other piece of AI governance into an answer a firm can give under pressure. The register is where accountability, materiality, oversight and explanation all meet on one page — and a firm that maintains it honestly has already done most of the work the guidance asks, because it can say, of any material decision, what the AI did, why, and who stood behind it.

If a regulator asked you today to describe the main factors behind a single material AI-influenced decision, could you answer from your own register in a sentence — or would you have to go and ask the model's vendor?

This concludes the series. The argument has not changed since Part 1: in regulated financial services, AI is a leadership and operating-model decision first — and the firms that treat it that way are the ones that will still be trusted with it.

Sources: JFSC, Guidance on the use of AI in Jersey's financial services sector (July 2026) — Principle 5 (transparency and explainability, the “main factors” standard, and the AI register fields), supported by Principle 1 (accountability) and the materiality classification; Government of Jersey et al., Jersey AI Playbook (2026) on ethical differentiation.

Professional disclaimer: The views expressed are my own and are shared for general informational purposes. This content is intentionally non-confidential and does not include proprietary, client-specific, or employer-sensitive information. It is not legal, regulatory, investment, or professional advice; readers should seek appropriate advice for their circumstances.

Financial services leader (20+ years) | Lean Six Sigma Green Belt | London School of Economics (LSE) AI Leadership Accelerator

A version of this article was first published on LinkedIn as part of the Sequence body of work. This is the canonical version.

The views expressed are the author's own and shared for general information. This content is intentionally non-confidential and does not include proprietary, client-specific or employer-sensitive information. It is not legal, regulatory, investment or professional advice. © 2026 Paul Soares. Sequence is the trading name of Sequence Advisory Limited (in formation) · hello@sequenceadvisory.com