A continuous-speech interface to a decision-support system: ii. an evaluation using a wizard-of-oz experiment

From Semantic Portal Wiki

{{#vardefine:category|Publication}}{{#vardefine:templatename|i.publication}}{{#vardefine:package|smwbp_instance_templates}}

Edit

Reference: {{#vardefine:pagename|a continuous-speech interface to a decision-support system: ii. an evaluation using a wizard-of-oz experiment }}

1. [[]]

bibtex

{{#vardefine:pagename|A continuous-speech interface to a decision-support system: ii. an evaluation using a wizard-of-oz experiment }}{{#vardefine:key| }}

abstract: Objective: Evaluate the performance of a continuous-speech interface to adecision-support system.Design: We performed a prospective evaluation of a speech interface thatmatches unconstrained utterances of physicians with controlled-vocabulary termsfrom Quick Medical Reference (QMR). We assessed the performance of the speechinterface in two stages: in the real-time experiment, physician-subjects viewedaudio-visual stimuli intended to evoke clinical findings, spoke a description ofeach finding into the speech interface, and then chose from a list generated bythe interface the QMR term that most closely matched the finding. Subjectsbelieved that the speech recognizer decoded their utterances; in reality, ahidden experimenter typed utterances into the interface (Wizard-of-Ozexperimental design). Later, we replayed the same utterances through the speechrecognizer and measured how accurately utterances matched with appropriate QMRterms using the results of the real-time experiment as the gold standard.Measurements: We measured how accurately the speech-recognition systemconverted input utterances to text strings (recognition accuracy) and howaccurately the speech interface matched input utterances to appropriate QMRterms (semantic accuracy). Results: Overall recognition accuracy was less than 50%. However, usinglanguage-processing techniques that match keywords in recognized utterances tokeywords in QMR terms, the semantic accuracy of the system was 81%. Conclusions: We found that reasonable semantic accuracy can be attained whenlanguage-processing techniques are used to accommodate for speechmisrecognition. We also found that the Wizard-of-Oz experimental design offeredmany advantages for this evaluation and believe that this technique may beuseful to future evaluators of speech-input systems.

download:

• paper:
• slides: