A Continuous-Speech Interface to a Decision-Support System: II. An Evaluation Using a Wizard-of-Oz Experiment

From Tetherless World Wiki

Jump to: navigation, search

Citation: William M. Detmer and Smadar Shiffman and Jeremy C. Wyatt and Charles P. Friedman and Christopher D. Lane and Lawrence M. Fagan. (1994) A Continuous-Speech Interface to a Decision-Support System: II. An Evaluation Using a Wizard-of-Oz Experiment. In , 1994.

Publication techreport ( Edit )
type Technical Report
bibtype techreport
Bibtex basics
author William M. Detmer and Smadar Shiffman and Jeremy C. Wyatt and Charles P. Friedman and Christopher D. Lane and Lawrence M. Fagan
title A Continuous-Speech Interface to a Decision-Support System: II. An Evaluation Using a Wizard-of-Oz Experiment
number KSL-94-38
institution Knowledge Systems, AI Laboratory
year 1994
Bibtex more
note Updated February 1995.
Access Paper
abstract Objective: Evaluate the performance of a continuous-speech interface to a decision-support system.Design: We performed a prospective evaluation of a speech interface that matches unconstrained utterances of physicians with controlled-vocabulary terms from Quick Medical Reference (QMR). We assessed the performance of the speech interface in two stages: in the real-time experiment, physician-subjects viewed audio-visual stimuli intended to evoke clinical findings, spoke a description of each finding into the speech interface, and then chose from a list generated by the interface the QMR term that most closely matched the finding. Subjects believed that the speech recognizer decoded their utterances; in reality, a hidden experimenter typed utterances into the interface (Wizard-of-Oz experimental design). Later, we replayed the same utterances through the speech recognizer and measured how accurately utterances matched with appropriate QMR terms using the results of the real-time experiment as the gold standard.Measurements: We measured how accurately the speech-recognition system converted input utterances to text strings (recognition accuracy) and how accurately the speech interface matched input utterances to appropriate QMR terms (semantic accuracy). Results: Overall recognition accuracy was less than 50%. However, using language-processing techniques that match keywords in recognized utterances to keywords in QMR terms, the semantic accuracy of the system was 81%. Conclusions: We found that reasonable semantic accuracy can be attained when language-processing techniques are used to accommodate for speech misrecognition. We also found that the Wizard-of-Oz experimental design offered many advantages for this evaluation and believe that this technique may be useful to future evaluators of speech-input systems.

KSL Technical Report ID: KSL-94-38
Facts about A Continuous-Speech Interface to a Decision-Support System: II. An Evaluation Using a Wizard-of-Oz ExperimentRDF feed
Abstract Objective: Evaluate the performance of a … Objective: Evaluate the performance of a continuous-speech interface to a decision-support system.Design: We performed a prospective evaluation of a speech interface that matches unconstrained utterances of physicians with controlled-vocabulary terms from Quick Medical Reference (QMR). We assessed the performance of the speech interface in two stages: in the real-time experiment, physician-subjects viewed audio-visual stimuli intended to evoke clinical findings, spoke a description of each finding into the speech interface, and then chose from a list generated by the interface the QMR term that most closely matched the finding. Subjects believed that the speech recognizer decoded their utterances; in reality, a hidden experimenter typed utterances into the interface (Wizard-of-Oz experimental design). Later, we replayed the same utterances through the speech recognizer and measured how accurately utterances matched with appropriate QMR terms using the results of the real-time experiment as the gold standard.Measurements: We measured how accurately the speech-recognition system converted input utterances to text strings (recognition accuracy) and how accurately the speech interface matched input utterances to appropriate QMR terms (semantic accuracy). Results: Overall recognition accuracy was less than 50%. However, using language-processing techniques that match keywords in recognized utterances to keywords in QMR terms, the semantic accuracy of the system was 81%. Conclusions: We found that reasonable semantic accuracy can be attained when language-processing techniques are used to accommodate for speech misrecognition. We also found that the Wizard-of-Oz experimental design offered many advantages for this evaluation and believe that this technique may be useful to future evaluators of speech-input systems. future evaluators of speech-input systems.
Author William M. Detmer and Smadar Shiffman and Jeremy C. Wyatt and Charles P. Friedman and Christopher D. Lane and Lawrence M. Fagan  +
Bibtype techreport  +
Has author William M. Detmer and Smadar Shiffman and Jeremy C. Wyatt and Charles P. Friedman and Christopher D. Lane and Lawrence M. Fagan  +
Has identifier KSL-94-38  +
Has publishing details 1994  +
Has title A Continuous-Speech Interface to a Decision-Support System: II. An Evaluation Using a Wizard-of-Oz Experiment  +
Has year 1994  +
Institution Knowledge Systems, AI Laboratory  +
Ksl tr id KSL-94-38  +
Note Updated February 1995.  +
Number KSL-94-38  +
Process note YES  +
Title A Continuous-Speech Interface to a Decision-Support System: II. An Evaluation Using a Wizard-of-Oz Experiment  +
Year 1994  +
Retrieved from "http://tw.rpi.edu/wiki/A_Continuous-Speech_Interface_to_a_Decision-Support_System:_II._An_Evaluation_Using_a_Wizard-of-Oz_Experiment"
Personal tools