Evaluating voice agents: WER isn't enough

Word-error rate is useful for measuring speech recognition, but a caller does not experience an ASR model in isolation. They experience a complete agent: listening, interpreting intent, deciding what to do, responding at the right moment, and recovering when the conversation goes off script.

A voice agent can achieve a respectable WER and still fail the call. It may interrupt the user, miss a correction, choose the wrong tool, repeat itself, or transfer without the context a human agent needs.

Evaluate the outcome, not only the transcript

The primary question is whether the user’s goal was completed correctly. That requires a scenario-level test with a defined starting state, expected outcome, allowed actions, and failure conditions.

For a scheduling flow, success might require the correct date, timezone, customer identity, confirmation, and system update. A natural-sounding conversation that books the wrong slot is still a failure.

Use a multi-dimensional rubric

We separate voice-agent quality into dimensions so teams can diagnose the responsible component:

• Intent and entity accuracy: Did the system understand the goal and important details?
• Task completion: Did tools and workflows produce the correct final state?
• Conversation quality: Was the response clear, concise, and appropriate for the user?
• Turn taking: Did the agent handle pauses, interruptions, and backchannels naturally?
• Recovery: Did it recognise uncertainty, correct mistakes, and avoid loops?
• Latency: Were responses fast enough to preserve conversational flow?
• Safety and compliance: Did the agent follow consent, disclosure, privacy, and escalation rules?

Scores should include concrete anchors. Reviewers need examples of what qualifies as acceptable, poor, and critical for each dimension.

Test realistic acoustic conditions

Studio audio hides production failures. Evaluation sets should include background noise, speakerphone distortion, packet loss, varied microphones, accents, different speaking rates, and overlapping speech. These conditions should be tagged so regressions can be traced to a specific acoustic slice.

The same scenario should be tested with paraphrases and different voices. Otherwise, a system may appear reliable because it has learned the wording of the test rather than the underlying task.

Measure turns and sessions

Turn-level analysis finds recognition errors, slow responses, and awkward phrasing. Session-level analysis finds loops, accumulated misunderstanding, premature transfers, and failures to complete the job. Both are needed.

We also inspect the handoff. When escalation is correct, the agent should transfer the relevant context, explain the next step to the user, and avoid making the user repeat information unnecessarily.

Combine automation with calibrated human review

Automated checks can validate latency, tool outcomes, required phrases, and structured fields. Human reviewers are better suited to conversational naturalness, politeness, ambiguity, and whether recovery felt appropriate.

A representative sample should be double-reviewed during calibration. Disagreements reveal unclear rubric language or scenarios with multiple valid outcomes.

Turn production failures into regression tests

After launch, confirmed incidents and difficult calls should be anonymised and added to the evaluation suite. Release gates can then require minimum scores for critical dimensions and important slices—not just an improved average.

WER remains a valuable diagnostic metric. It becomes meaningful when placed inside a broader framework that measures whether the agent understood, acted correctly, communicated well, and recovered safely.