next up previous
Next: Accuracy Up: Evaluation Previous: Evaluation


Table 3 reports on the efficiency of the NLP components for the set of 1000 wordgraphs and test utterances. The first two rows present the results for sentences; the remaining rows provide the results for word-graphs. Listed are respectively the average number of milliseconds per input; the maximum number of milliseconds; and the maximum space requirements (per word-graph, in Kbytes).

For most word-graphs we used the nlp_speech_trigram method as described above. For large word-graphs (more than 100 transitions), we first selected the best path in the word-graph based on acoustic scores and N-gram scores only. The resulting path was then used as input for the parser. In the case of these large word-graphs, N=2 indicates that bigram scores were used, for N=3 trigram scores were used.

CPU-time includes tokenizing the word-graph, removal of pause transitions, lexical lookup, parsing, the robustness/disambiguation component, and the production of an update expression. 10

Table 3: Efficiency (1).
input method mean msec max msec max Kbytes
test sentence data-oriented 91 8632 14064
test sentence grammar-based 28 610 524
word graphs data-oriented 7011 648671 619504
word graphs grammar-based N=2 298 15880 7143
word graphs grammar-based N=3 1614 690800 34341

For word-graphs the average CPU-times are actually quite misleading because CPU-times vary enormously for different word-graphs. For this reason, we present in table 4 the proportion of word-graphs (in %) that can be treated by the NLP component within a given amount of CPU-time (in milliseconds).

Table 4: Efficiency (2).
method 100 500 1000 5000 10000
data-oriented 52.7 70.8 76.6 90.6 94.2
grammar-based N=2 58.6 87.0 94.6 99.5 99.8
grammar-based N=3 58.5 78.9 87.3 96.7 98.7

next up previous
Next: Accuracy Up: Evaluation Previous: Evaluation