The RFP Questions That Only Appear After Go-Live
Every feature you evaluated works in the demo. The questions that decide whether you regret the purchase are the ones the demo is designed not to raise.
Every feature you evaluated works in the demo. The questions that decide whether you regret the purchase are the ones the demo is designed not to raise.
The demo was flawless. Every feature you asked to see, you saw. Eighteen months later the problems are all things you never thought to ask about — because the demo only ever shows you the happy path.
Feature-led procurement measures the wrong thing. A request for proposal built around a checklist of capabilities tells you what a system can do on its best day, in a controlled demonstration, with clean data and a friendly presenter. It tells you almost nothing about how the system behaves when a client is unusual, when a process fails, when someone needs to override it, or when the firm eventually wants to leave. Those are the questions that determine whether the purchase was a good one — and they only appear after go-live, when it is too late to have asked them.
The fix is to build the failure-state and control-state questions into the RFP itself, so that the evaluation tests dependability rather than functionality. The strongest technology evaluations I have been part of spend as much time probing what a system does when things go wrong as what it does when things go right.
The demo shows the happy path by design
A vendor demonstration is a piece of theatre, and there is nothing dishonest about that — it is the vendor showing its product at its best. The problem is that the buyer mistakes the performance for the product. What the demo cannot show, and is not built to show, is the behaviour that matters most in a regulated firm: what happens with the awkward client structure, the multi-currency arrangement, the restriction that should have stopped a transaction, the outage that takes the system offline mid-process.
The pattern I return to is that the highest-value knowledge about any system sits in its failure modes and its constraints, not its features. Two systems can have an identical feature list and be completely different to operate, because one fails gracefully with clear warnings and a manual fallback, and the other fails silently and leaves you reconstructing what happened. A feature list cannot distinguish between them. Only failure-state questions can.
A good RFP does not ask what the system can do. It asks what the system does when it cannot.
The questions that surface after go-live
When you evaluate a custody platform, a portfolio management system, or any core operational tool, a specific set of questions reliably predicts post-implementation regret. They cluster around control, failure and exit — the three areas a feature demo is structured to avoid. Table 1 sets them out as a reusable due-diligence instrument.
Table 1 — The post-go-live RFP questions: a control-and-failure due-diligence checklist
| Area | The question to ask before signing | Why it only surfaces later otherwise |
|---|---|---|
| Restrictions | Do restrictions create hard stops, or only warnings? | A warning that can be clicked through is not a control. |
| Override | Who can override the system, and is the override logged? | Unlogged overrides destroy your audit trail invisibly. |
| Audit history | What history is retained, for how long, and can you extract it? | You discover the gap the first time a regulator asks. |
| Failure behaviour | What happens when the system is unavailable mid-process? | Silent failure with no fallback halts operations. |
| Change control | What can change without notifying us? | Silent vendor changes alter your control environment. |
| Unusual clients | How does it handle multi-currency, complex structures, edge cases? | The demo used a clean, typical client; your book is not. |
| Data exit | Can we extract our data, complete and linked, if we leave? | Exit readiness is invisible until you try to leave. |
| Operational scope | Which functions remain manual rather than automated? | Assumed-automated steps become surprise manual work. |
Control-state and failure-state are the real evaluation
The questions in that table share a structure. They do not ask what the system does — they ask what the system permits, what it prevents, what it records, and what it does at its limits. That is the difference between evaluating features and evaluating controls, and in a regulated firm the controls are the point. A system that can do everything but cannot enforce a hard stop, cannot log an override, or cannot tell you what changed is not a capable system. It is an ungoverned one.
The same applies to the boundary between the investment side of a platform and its operational reality. Demonstrations naturally showcase the impressive, front-of-house capabilities. The questions that determine the operational load are duller and more important: how orders are staged, which functions remain manual, where the hand-offs sit, and what the firm will actually have to do by hand every day. Those answers rarely appear in a demo because they are not what a demo is for.
The most reliable way to force these answers out is to change what you ask the vendor to demonstrate. Instead of accepting a scripted walk-through, hand the vendor a deliberately awkward scenario drawn from your own book — the multi-currency client, the account with a restriction that must not be breached, the mid-process interruption — and ask them to show you the system handling it, including what it does when it cannot. A vendor confident in their failure states will welcome this. A vendor who deflects, reschedules, or answers a different question has told you something the feature list never would. The willingness to be tested on the unhappy path is itself a due-diligence signal, and it costs nothing to ask for.
The same framework governs AI due diligence
This procurement instrument transfers almost unchanged to AI and agentic tools, where the failure-state questions matter even more because the technology is less transparent. The questions become: what can the model or agent do; what can it never do; who can override it; what gets logged; and what happens when it is unavailable or wrong. That is the same control-and-failure structure applied to a system whose behaviour is probabilistic rather than deterministic — which raises, rather than lowers, the importance of asking.
This is consistent with how Jersey's regulatory guidance frames AI adoption. The JFSC's Materiality Ladder ties governance depth to potential impact, and its treatment of agentic AI — systems that act without real-time human approval — calls for the strongest controls: explicit boundaries, human oversight of material decisions, and defined fallbacks. An RFP that already asks what a system can never do, who can override it, and what happens when it fails is an RFP that produces the evidence the guidance expects. Feature-led procurement produces none of it. Turn this checklist into your standard scorecard, and regulated-technology due diligence stops being something you retrofit and starts being something you buy on.
For the last core system your firm bought, could the RFP tell you what it does when it fails, who can override it, and whether you can get your data back — or did you only ever evaluate what it does when everything works?
Next: why management information is an operating system, not a monthly pack — MI Is Not Reporting. It Is the Management Operating System.
Sources: JFSC, Guidance on the use of AI in Jersey's financial services sector (2026) — the Materiality Ladder, Principle 1 (human oversight, agentic AI controls) and Principle 2 (auditability, third-party resilience); AI Strategy in Jersey FS, Series 2 Part 5 (agentic AI sourcing and controls).
Professional disclaimer: The views expressed are my own and are shared for general informational purposes. This content is intentionally non-confidential and does not include proprietary, client-specific, or employer-sensitive information. It is not legal, regulatory, investment, or professional advice; readers should seek appropriate advice for their circumstances.
Financial services leader (20+ years) | Lean Six Sigma Green Belt | London School of Economics (LSE) AI Leadership Accelerator
See where your firm stands
The free diagnostic scores your firm across the twelve operating-model instruments in about eight minutes.
Take the diagnostic →A version of this article was first published on LinkedIn as part of the Sequence body of work. This is the canonical version.
The views expressed are the author's own and shared for general information. This content is intentionally non-confidential and does not include proprietary, client-specific or employer-sensitive information. It is not legal, regulatory, investment or professional advice. © 2026 Paul Soares. Sequence is the trading name of Sequence Advisory Limited (in formation) · hello@sequenceadvisory.com