EVALUATING RESPONSIVENESS IN SPOKEN DIALOG SYSTEMS

Wataru Tsukahara and Nigel Ward

International Conference on Spoken Language Processing, 2000

Abstract:

Ratings of user satisfaction, although fairly easy to elicit for
today's spoken language systems, can be more elusive for systems which
operate at near-human levels of performance.  This problem can be
alleviated by adding a \scare{re-listening} phase before eliciting
judgements: in this phase the user listens to a recording of himself
interacting with the system while consulting a transcript of that
interaction.  This technique allows more sensitive judgements of
system quality because it avoids problems arising from attention
limits.