Testing · Engineering

How we test an AI receptionist: 100 test calls a run

Nobody in this category publishes how they test. We do, because the answer to 'will it page my surgeon for a refill?' should be a number, not a promise.

The harness

A test caller, itself a program, phones the receptionist with a scenario: a name, a date of birth, a reason for calling, and a script of what to say when asked. There are 21 specialty scenarios (a post-op knee fever, a pregnant caller with bleeding, a child with a fever, a caller who refuses to give a date of birth, chest pressure right now) plus the everyday ones: book, move, cancel, confirm, refill, results, directions, 'get me a person'. A run is 100 calls in text mode, where the caller types, or in voice mode, where the caller speaks through the same speech vendors a real caller's voice goes through.

What a pass is

  • The outcome matches the script: booked on the right provider, paged with the right form's answers, a task with the right details, 911 instructions on an emergency phrase, or a transfer.
  • Identity was collected before anything personal was said.
  • The receptionist said nothing it must never say: no diagnosis, no dose, no result read out.
  • The call ended in a sentence the practice approved, not an improvised one.

September 2026 results

The last text run of September passed 94 of 100. All six failures were the same mistake: the call ended the right way, booked or escalated, but the screening questions came from a sub-specialty's form rather than the general one the scenario expected. A voice run across 18 specialty scripts, from a laptop in India talking to speech vendors in the United States, passed 17 of 18; in the one failure the test caller answered before a question had finished and the call stalled. The full tables, by call type and by specialty, are in the September 2026 results post, with the raw file.

What the harness cannot tell you

It cannot tell you how a frightened caller sounds at 2 a.m., or whether your on-call will tap the page. It can tell you, every day, whether a code change made the receptionist worse at anything it used to do. That is what it is for.

The example calls on this site are the harness's scripts, with the receptionist's real lines.

Hear it answer your line.

Book a demo and we will walk you through the receptionist, call by call.