Summary

A fintech lender had real AI ambition and a growing set of AI initiatives, but no common discipline underneath them. Teams were selecting their own tools, customer data was reaching model calls without a masking layer, and the highest-capability models were being used for trivial tasks because nothing stopped it. When something went wrong, nobody could answer a basic question: what did the model do, for whom, and at what cost.

We ran a structured assessment across their live and planned use cases, then closed three design decisions before a single production agent was built: mask personally identifiable information upstream of the model, define the agent handoff and audit log first, and instrument measurement across agentic execution rather than just tokens and spend.

The result is one platform where technical and functional metrics sit together, and where the CTO, the CFO and product owners each see the part they are accountable for. The client now upgrades existing agents and commissions new pipelines without a security or cost review becoming the bottleneck.

Business Challenge

Summary: AI adoption had outrun AI governance. Customer data was flowing into model calls unmasked, model spend was uncontrolled, and no one could reconstruct what any agent had actually done.

The organization was not short of AI ambition. It was short of the architecture required to hold that ambition at enterprise scale. The assessment surfaced six intervention areas.

Governance running in silos. Teams independently shortlisted opportunities and tools. Each choice was defensible on its own. Together they produced a landscape nobody owned, with no shared standard for what qualified as ready for production.

Customer data reaching models unmasked. There was no masking layer between the source systems and the model calls. Personally identifiable information was flowing through in the clear, and because there was no detection in place, the exposure was invisible rather than accepted.

No infosec oversight of tool usage. No one could say which AI tools employees were using or what data they were routing through them. The risk here is not the tools. It is the absence of the question.

No cost governance. Teams were calling the newest and highest-capability model for low-level tasks, because that was the default and nothing challenged it. Spend grew in a way that could not be attributed back to any particular decision.

No observability. The core failure. The organization could not answer what the model did, for whom, and at what cost. Without that, every incident became an argument rather than an investigation.

Fragmented context producing misdiagnosis. Specialized agents were losing context on handoff. Users experienced the resulting partial answers as hallucination. Engineering then treated them as a model quality problem and tuned prompts, which fixed nothing, because the fault was in the handoff and not in the model. Wrong diagnosis, wrong fix, repeatedly.

Left unresolved, the trajectory was predictable. More agents, more surface area, more spend, and a first serious incident with no forensic trail to work from.

Solution Approach

Summary: We assessed the estate across four dimensions, crystallized three design decisions, and closed all three before building agents. Masking, audit and measurement came first, deliberately.

Step 1. Assessment

Structured workshops with the teams already running or planning AI work. Every use case was enlisted and scored across four dimensions.

Dimension

What we assessed

Use case and business priority

Which use cases would return most, and in what order

Readiness of existing architecture

Cloud stack, configurability, retrieval pipelines

Governance

Role-based access to platforms, tools and models

Observability canvas

Existing log stack, reporting, monitoring KPIs

The scoring did something the workshops alone would not have. It converted a list of enthusiasms into a priority and risk matrix, which is what made sequencing possible.

Step 2. Three decisions, closed before any agent shipped

Decision one: mask upstream, not downstream. Masking rules were written for each cohort of sensitive data, covering identity documentation, demographic attributes and address information. The anonymization layer sits between the raw customer data and the model, so what reaches the model is a runtime-generated key. The model does not process personally identifiable information and does not store it. This was implemented ahead of the data protection regulations landing in the client's market, not in response to them.

The distinction that matters technically: this is structured extraction at ingress, not redaction after the fact. Post-hoc redaction assumes the data already reached the model, which is the thing you were trying to prevent.

Decision two: define the audit trail before the agents. Because context loss between agents was already surfacing to users as apparent hallucination, we defined the handoff log before designing the agent suite. Four things are specified for every handoff: what each agent passes forward, what is recorded, what is written to long-term memory, and where a human acts in the workflow.

This is the decision teams usually defer, and deferring it is why agentic systems become undebuggable. Once the handoff is logged, a partial answer is traceable to the agent that dropped the context, and the fix lands in the right place.

Decision three: measure everything, including the agents themselves. Observability was designed to oversee agentic execution rather than only count tokens and spend. Five things were built in:

• A model orchestrator that selects which model to invoke against defined rules, weighing the incoming query, the nature of the required output, how much human control the workflow demands, and the accuracy expected. This is what ended the practice of calling frontier models for trivial work.
• Cost decomposed below the token line, separating query decomposition, contextual search and response formatting, so spend attaches to the part of the pipeline that caused it.
• Chunk-level traceability, so a human can follow lineage down to the retrieved chunk with its confidence score.
• A repository of whitelisted models, so model adoption became a governed decision rather than a developer preference.
• Workflow limits that halt the process when a rule is breached, rather than logging the breach and continuing.

Step 3. Agents built against the priority and risk matrix

Only after the three decisions were closed did implementation begin, sequenced by the matrix from Step 1. Retrieval chatbots, API agents and voice agents were built onto a base that already had masking, audit and measurement in place, which meant none of them needed retrofitting to pass review.

Business Impact & Results

Summary: The organization moved from being unable to reconstruct what any agent did to having one platform where every decision is visible to the CTO, the CFO and product owners at once. Agent expansion stopped requiring a governance argument.

What became visible

One platform, four views, technical and functional metrics side by side.

View

What the business can now see

What is running well

Retrieval chatbots with answer confidence and user satisfaction. API agents with the full decision chain across the payload, and whether a human approved or the system auto-approved. Voice agents by call, language, conversational tone and the product actually pitched

What needs checking

Which agents breach rules most often and which pipeline is affected next. Low-confidence retrievals, and specifically which users received them. Capacity utilization and load distribution by geography

Latency by layer

Hop-wise latency across retrieval, inference and response. Latency spikes correlated to root-cause attributes. Voice agent latency tracked on its own benchmark, because conversational latency fails differently

Cost decomposed

Model-wise token consumption and weighted average cost per conversation. Infrastructure cost by regional pricing. Capacity utilization by weekday, time of day and campaign

The single sharpest change is the one that reads as least dramatic. The question what did the model do, for whom, and at what cost went from unanswerable to a dashboard filter.

Operational outcomes

Cost attribution became possible. Before, model spend was a line item. After, it decomposes to the pipeline stage and the campaign that generated it. Orchestration rules that route low-complexity work away from frontier models are the mechanism most directly attached to spend reduction, and internal modeling of those rules against the prior usage pattern indicates a reduction of roughly 23% in weighted average cost per conversation. That figure is modeled from the routing rules rather than measured over a full quarter, and we say so wherever it is used.

Incidents became investigations. With chunk-level lineage and confidence scores, a low-quality answer is traceable to its retrieval rather than attributed to the model and argued about.

Hallucination complaints were correctly reclassified. A meaningful share of what had been logged as model hallucination turned out to be context loss on agent handoff. Naming it correctly stopped the team spending cycles tuning prompts against a problem that lived somewhere else.

Review stopped being the bottleneck. Because masking, audit and measurement were already in place, new agents inherited compliance posture instead of negotiating it. This is the outcome the client values most and the one that is hardest to see from outside.

Long-term business value

The C-suite now has the confidence to upgrade existing agents and commission new pipelines, because the platform answers the questions that previously stopped them. Governance became infrastructure rather than a gate, and infrastructure scales in a way that a review committee does not.

Key Takeaways

1. Close the governance decisions before you build the agents. Masking, audit trail and measurement are cheap to design first and expensive to retrofit. The sequence is the whole insight, and it is the one most programs get backwards.
2. Mask upstream of the model, not downstream of the incident. Structured extraction at ingress means sensitive data never reaches the model layer. Post-hoc redaction only proves the exposure already happened.
3. Most hallucination complaints in agentic systems are context loss. If handoffs are not logged, the symptom appears at the user and the cause stays invisible, so teams tune prompts and fix nothing. Define the handoff log before the agent suite.
4. Cost governance is a routing problem, not a pricing problem. The waste is not what the model costs. It is calling the most capable model for work that never needed it. A rules-based orchestrator handles this in a way that developer discipline does not.
5. Observability is what makes AI expansion approvable. Once a business can answer what the model did, for whom, and at what cost, the next agent stops being an argument and becomes a decision.

Conclusion

The thing that unlocked this program was not a model choice or an architecture pattern. It was refusing to build agents until three decisions were closed.

That refusal looks slow for about six weeks and then pays for itself continuously, because every agent built afterwards inherits masking, audit and measurement instead of negotiating for them one review at a time. The client's ambition did not change through this engagement. What changed is that the architecture could finally hold it.

Any organization with AI initiatives spread across several teams will recognize at least four of the six intervention areas described here. The pattern is consistent regardless of sector, because the failure is structural rather than technical: adoption moves faster than governance, and the gap stays invisible until an incident makes it visible on someone else's terms. The remedy is equally transferable. Assess what is already running, close the decisions that are expensive to reverse, and instrument the system so that every decision it makes can be seen.

Start with the question the platform is built to answer. If you cannot say what your models did, for whom, and at what cost, that is the first project, not the last.

Looking to solve a similar business challenge? Connect with our experts to explore the right solution for your organization.

If you're scaling AI across more than one team, a structured assessment covering use case priority, architecture readiness, governance, and observability will tell you within a few weeks which decisions you can still afford to defer and which ones you can't.