Quality assurance in a call center runs on a budget nobody writes down.
To score a call properly, a reviewer needs roughly twice its length. Once to listen, again to grade it against the rubric and find the evidence. So a full-time analyst gets through something like twenty hours of reviewed interaction in a week, and that twenty hours has to cover every agent on the floor.
The capacity is spoken for before anyone picks a call. Sales get graded first, for completion and validation, because a sale is the thing the business has to be able to stand behind. It’s the right priority. It also eats most of what there is.
Which means the reviewed population is almost entirely calls that worked.
Analysts in the war studied the bombers that came back and proposed armoring where the bullet holes were, until someone pointed out those were the survivable hits. The armor belonged where the returning planes showed no damage at all, because the planes hit there hadn’t come back to be measured.
A quality program built on sales calls is measuring returning bombers. The shorter calls and the opportunities that never closed are the ones that didn’t come back, and they carry most of what anyone would want to know. Compliance items sit across all of it. They happen on every call, converted or not, and they get whatever attention is left over.
This is an outbound floor where agents take card numbers and Social Security numbers to collect payments and open accounts. PCI and CPNI aren’t abstractions there. A primary barrier has to exist, and twenty hours a week of human listening is not one.
What changes when you score all of it
Full coverage reaches the planes that didn’t come back. Objections, competitor mentions and rejection reasons all live in calls that failed, which is why the operation’s account of why customers say no used to be assembled out of what agents remembered hearing.
Agents get a fairer read out of it too, since a bad month is no longer a handful of calls nobody can go back and re-examine. That tends to be what people ask about first.
It also means a sample can’t be cut. Slice a thin sample by offer, then by campaign, then by region, and there’s nothing left to stand on. Every rollup above the individual agent is either unavailable or quietly unsound. Score everything and the same figures hold at every altitude: region, site, campaign, supervisor, agent.
That’s what supervisors actually open it for. Not scores. Objections, competitor mentions, and take rate, broken out by offer, by campaign, and by region. If one carrier is coming up twice as often as another against a particular offer, that’s a morning meeting and a check after lunch on what worked.
It answers the other direction too. An agent pitching an add-on at half the rate of their peers looks like a coaching problem until you put take rate next to it. Sometimes they’re picking their moments. The delta is what’s worth talking about, and you can’t see a delta in a sample.
The same material rolls up into monthly and quarterly business reviews, by campaign and by region, which is only defensible because the coverage underneath is complete.
The part that was actually hard
Not the model. Knowing who was speaking.
The recordings are mono, both sides on one track, and the transcription is fuzzy. All of QA rests on attributing a line to the agent or the customer, and a system that gets that wrong doesn’t produce slightly worse scores. It produces confident nonsense. That problem got its own logic, its own model, and most of the research: speaker heuristics, what a local model can hold in context, and where the GPU stops being able to help. The rubric mattered as much as the scorer did, and both had to be right before any of it was worth running.
A scorer that can’t hallucinate
Scoring runs in two passes.
The first pass reads the transcript and extracts structured facts. It doesn’t score anything. The second pass grades against the full criteria using only that extraction. It never sees the transcript at all.
A model can’t invent a quote it can’t read. If something wasn’t extracted in the first pass, there’s no text for the second pass to score, so a competitor mention or a rebuttal that never happened can’t be credited. That’s a structural guarantee rather than an instruction in a prompt, and it’s the reason I’d hand the output to a supervisor without a caveat attached.
Every rating cites its evidence from the transcript, so any score can be checked in seconds by the person disputing it.
Reliability isn’t uniform, and pretending otherwise would be the mistake. Compliance language is close to fixed — the same disclosure, the same words, every call — and that’s where the model is most dependable. Competitor mentions and rejection reasons are a small set of repeated lines. Judgment calls are softer, which is why the quality team can override any rating, and why those overrides feed back into the rubric.
It runs on a self-hosted GPU box. Once warm it held an average of one call every seventy-six seconds across a full week. That works out to about 3,400 calls a week from a single machine, enough to score every transcript over two minutes for a floor of 110 agents. No transcript leaves the network, which is what makes any of it permissible.
What the model isn’t allowed to decide
A separate module holds deterministic rules that run after the model and correct it: quote verification against the transcript, product and carrier normalization, forced N/A where an item can’t apply, red-flag validation, and composite scoring with the renormalization that redistributes weight when no compliance items apply. These are unit tested, because the rubric is judgment and this is arithmetic, and arithmetic shouldn’t be probabilistic.
Sale credit is the clearest case. The model forms an opinion about whether the call converted, and that opinion gets thrown away. Conversion comes from the dialer’s disposition, the system of record the client and the floor already trust. The model’s verdict is used only where a disposition hasn’t finalized and the scorer proved customer acceptance with a real quote.
Dispositions settle days later, so a separate job re-derives sale credit across a rolling twenty-one-day window. It’s idempotent. A missed day heals on the next run.
The rubric is a file you can open
The rubric, the prompts, the repair and validation logic all live in a single standalone HTML file that runs in a browser as a working scorecard. Production doesn’t reimplement any of it. The worker extracts that file’s script and runs it in a sandbox, so the artifact a person can open and the logic production runs are the same thing and can’t drift apart.
In practice I’m the only person who has ever changed it. The architecture means a quality organization could own its own scoring standard without a developer in the loop. What it has actually bought so far is one definition of the rubric and no way for the running system to disagree with the document.
How a call reaches the scorer
People ask about the scorer, but the queue underneath it is the part that had to be right, because a scoring system that quietly drops calls is worse than not having one at all.
Calls arrive on a feed I don’t own and can’t change, refreshed every half hour. A sync job copies the eligible ones into a queue and a worker takes them one at a time. That worker stays up and polls rather than waking on a schedule, and the hourly job attached to it is a watchdog: it tries to take a lock, finds the running worker already holding it, and exits having done nothing. It only ever starts a worker when the last one died.
Claiming a call is one statement that marks the row and hands it back in the same breath, so two workers can’t take the same call even if two are somehow running. Before anything reaches a model, the calls the dialer already resolved as unanswered get filtered out, and a voicemail check runs in plain code. Neither of those costs a second of GPU. Both of them would otherwise cost seventy-six.
After that it’s three model calls in order, then the deterministic layer, then a single transaction. Either every table lands or none of them do.
The part I’d point at is what happens when that breaks. A worker can die holding a call, and if nothing notices, the call sits claimed forever and nobody finds out until someone asks why an agent’s numbers look thin. So a sweep runs on its own timer, finds anything held too long, and puts it back in the queue.
That shape repeats twice more. The worker exits at the end of each day so it comes back on whatever code was deployed that afternoon instead of running for weeks on a version I’ve since fixed. And sale credit is re-derived over a rolling window rather than tracked forward from the last successful run, so a morning the job didn’t fire gets picked up by the next one without anybody reconciling anything by hand.
What I would tell you in an interview
Swapping the model is an afternoon. The work was deciding which judgments a language model is allowed to make and then building the thing that enforces it: a scorer that can’t see enough to lie, and a rules layer that overrules it on anything arithmetic. The one number the business actually pays on comes from the dialer, not from either of them.