Do you need access to our model or training data?

No. We work against the system as it runs: its inputs, its outputs and the decisions downstream. The model stays yours and untouched. We need read access to production logs for sampling and a place in your repository for the golden set and the harness.

What if our system has no decision point where a wrong answer costs money?

Then you do not need this, and we will say so in the first call. The assessment pays off where an error becomes a denied claim, a wrong figure to a regulated user, a missed defect or a misrouted case.

One senior engineer end to end, with a second senior for the labelling week. Both have shipped production AI in healthcare, fintech or defense. No junior bench, no hand-offs.

What happens after the three weeks?

You keep everything: the golden set, the harness, the baseline and the report. Your engineers run it from the CI from then on. If the verdict lists gaps you want closed, that is a separate, scoped engagement, and you are free to close them yourself.

Can this be done under NDA and inside our infrastructure?

Yes. Most assessments run inside the client’s cloud with no data leaving it. HIPAA and SOC 2 environments are the norm for us, not the exception.

AI Reliability Assessment

Know, field by field, how accurate your AI is today.

And what a wrong answer costs you. And whether swapping models is a config change or a gamble.

Three weeks. Fixed scope, fixed price. Your model stays untouched; you keep the golden set and the harness.

Book a 45-minute scoping call Read the scope

When this is the right call

You want to upgrade the model and nobody can prove the new one will not break what works.
An AI feature is built but not released to regulated users, because there is no answer to “are you sure”.
A client, auditor or regulator asked “how do you know this is correct” and the room went quiet.
The demo worked, production does not, and nobody knows where the two diverged.
A cheaper model looks tempting and there is no baseline to measure the risk.

How the three weeks go

Week 0
Intake, two to three days
Access, a read of the system, a walk through the pipeline. We pick the decision points where a wrong answer costs money and agree on one or two task types: extraction, classification, cited answers, routing.
Week 1
Golden set
100 to 300 labelled examples pulled from real production traffic, labelled field by field, never “mostly right”. This is an asset you keep and did not have before.
Week 2
Eval harness
Automated runs in your stack, triggered from CI. Accuracy per field. A baseline for the system as it is today. A failure taxonomy: invented field, dropped field, wrong format, silent truncation, refusal.
Week 3
Guardrails and verdict
Schema validation, behaviour on an invalid answer, retry and fallback policy, hand-to-human thresholds. Cost and latency measured. Two or three candidate models run against the golden set, so a model swap becomes a config change.

What you get

›Golden set in your repository
›Eval harness that runs from CI
›Accuracy of the current system, by field
›Failure taxonomy with frequency and cost
›Model comparison: accuracy, cost, latency
›Ship / no-ship verdict with the exact gaps
›One-hour walkthrough with your team

What is not in it

Training or fine-tuning models. Building an MLOps platform. A security or compliance audit. Replacing your team.

We hand over the tools and the baseline; your engineers take it from there. A consulting partner builds the kitchen, we put in the wiring.

Scope and price

Triage

1 week

One task type, no guardrails. Report and recommendations.

$4,500

fixed price

Standard

3 weeks

One system, up to two task types, the full scope above.

$12,000

fixed price

Extended

4 to 5 weeks

Several systems, or an evidence pack prepared for a regulator.

$18,000

fixed price

Standard is the usual starting point. Triage is for one question that needs an answer this week. Extended is for teams that have to show their work to a regulator.

Measured on real systems

96.4%

detection accuracy in production, verified client review

89%

physician-rated answer accuracy, clinical RAG over 50,000 documents

200+

certified metrics on one data platform, reporting that the business trusts

The method is written up in Eval frameworks: how to swap LLM models without breaking production.

Questions we get

Do you need access to our model or training data?: No. We work against the system as it runs: its inputs, its outputs and the decisions downstream. The model stays yours and untouched. We need read access to production logs for sampling and a place in your repository for the golden set and the harness.
What if our system has no decision point where a wrong answer costs money?: Then you do not need this, and we will say so in the first call. The assessment pays off where an error becomes a denied claim, a wrong figure to a regulated user, a missed defect or a misrouted case.
Who does the work?: One senior engineer end to end, with a second senior for the labelling week. Both have shipped production AI in healthcare, fintech or defense. No junior bench, no hand-offs.
What happens after the three weeks?: You keep everything: the golden set, the harness, the baseline and the report. Your engineers run it from the CI from then on. If the verdict lists gaps you want closed, that is a separate, scoped engagement, and you are free to close them yourself.
Can this be done under NDA and inside our infrastructure?: Yes. Most assessments run inside the client’s cloud with no data leaving it. HIPAA and SOC 2 environments are the norm for us, not the exception.

Next step

A 45-minute call tells us both whether this fits.

Bring the system and the decision it makes. We leave with a yes, a no, or a Triage scope.

Book the call

Know, field by field, how accurate your AI is today.

Intake, two to three days

Golden set

Eval harness

Guardrails and verdict

Triage

Standard

Extended

A 45-minute call tells us both whether this fits.