| Abstract
|
Objective: Evaluate the performance of a c … Objective: Evaluate the performance of a continuous-speech interface to adecision-support system.Design: We performed a prospective evaluation of a speech interface thatmatches unconstrained utterances of physicians with controlled-vocabulary termsfrom Quick Medical Reference (QMR). We assessed the performance of the speechinterface in two stages: in the real-time experiment, physician-subjects viewedaudio-visual stimuli intended to evoke clinical findings, spoke a description ofeach finding into the speech interface, and then chose from a list generated bythe interface the QMR term that most closely matched the finding. Subjectsbelieved that the speech recognizer decoded their utterances; in reality, ahidden experimenter typed utterances into the interface (Wizard-of-Ozexperimental design). Later, we replayed the same utterances through the speechrecognizer and measured how accurately utterances matched with appropriate QMRterms using the results of the real-time experiment as the gold standard.Measurements: We measured how accurately the speech-recognition systemconverted input utterances to text strings (recognition accuracy) and howaccurately the speech interface matched input utterances to appropriate QMRterms (semantic accuracy). Results: Overall recognition accuracy was less than 50%. However, usinglanguage-processing techniques that match keywords in recognized utterances tokeywords in QMR terms, the semantic accuracy of the system was 81%. Conclusions: We found that reasonable semantic accuracy can be attained whenlanguage-processing techniques are used to accommodate for speechmisrecognition. We also found that the Wizard-of-Oz experimental design offeredmany advantages for this evaluation and believe that this technique may beuseful to future evaluators of speech-input systems. future evaluators of speech-input systems.
|
| Author
|
William M. Detmer +,
Smadar Shiffman +,
Jeremy C. Wyatt +,
Charles P. Friedman +,
Christopher D. Lane +,
Lawrence M. Fagan +
|
| Bibtype
|
techreport +
|
| Institution
|
Knowledge Systems, AI Laboratory +
|
| Key
|
KSL-94-38 +
|
| Modification dateThis property is a special property in this wiki.
|
1 May 2009 14:05:20 +
|
| Note
|
Updated February 1995. +
|
| Number
|
KSL-94-38 +
|
| Tag
|
Computer science +
|
| Title
|
A Continuous-Speech Interface to a Decision-Support System: II. An Evaluation Using a Wizard-of-Oz Experiment +
|
| Tr id
|
KSL-94-38 +
|
| Year
|
1994 +
|
| Categories |
Technical Report,
Publication,
KSL Technical Report
|
