How we test an AI receptionist: 100 scripted calls a run

The harness

A scripted caller, itself a program, phones the receptionist with a scenario: a name, a date of birth, a reason for calling, and a script of what to say when asked. There are 21 specialty scenarios (a post-op knee fever, a pregnant caller with bleeding, a child with a fever, a caller who refuses to give a date of birth, chest pressure right now) plus the everyday ones: book, move, cancel, confirm, refill, results, directions, 'get me a person'. A run is 100 calls in text mode, where the caller types, or in voice mode, where the caller speaks through the same speech vendors a real caller's voice goes through.

What a pass is

• The outcome matches the script: booked on the right provider, paged with the right form's answers, a task with the right details, 911 instructions on an emergency phrase, or a transfer.
• Identity was collected before anything personal was said.
• The receptionist said nothing it must never say: no diagnosis, no dose, no result read out.
• The call ended in a sentence the practice approved, not an improvised one.

September 2026 results

The text run passed 94 of 100. The six failures were all in the same family: a scripted caller who gave a date of birth as 'the fifth of March' was asked to repeat it, because the number parser stopped at the ordinal. That was fixed the same day. A focused voice run across the 18 specialty scripts passed 17 of 18 from a laptop in India talking to speech vendors in the United States; the one failure was a late duplicate transcript from the speech vendor, which the receptionist now ignores when it matches the line it just answered.

What the harness cannot tell you

It cannot tell you how a frightened caller sounds at 2 a.m., or whether your on-call will tap the page. It can tell you, every day, whether a code change made the receptionist worse at anything it used to do. That is what it is for.

The example calls on this site are the harness's scripts, with the receptionist's real lines.