Evaluation of the DAM, and the VIDIAM and PARADIME specific parts of the DAM.

16 march 2005

The DAM module will comprise a common part, and VIDIAM and PARADIME
specific parts.  In normal operation, both the VIDIAM and PARADIME
specific parts will be functioning simultaneously.  It will however be
possible to disable either of the two, and still have a fully functioning
DAM.  This way, we can evaluate functioning of the DAM as a whole, and of
the VIDIAM and PARADIME specific approaches.

We will specify a number of example dialogues (partially based on the
currently collected example dialogues) that illustrate the types of
dialogues that the system will be able to support.  Dialogues MT-01 and
SK-01 in the functional specification are initial examples. For each
version of the DAM, we will verify in how far the DAM is able to support
those types of dialogue.

Dialogue corpus collection is a part of the DAM projects.  VIDIAM and
PARADIME will conduct corpus collection experiments in collaboration.  We
will collect subjective information from the users by means of the
evaluation questions at the end of each dialogue.  The thus obtained
corpora will be used for several projects within IMIX.

VIDIAM and PARADIME will also use them for evaluation.  Besides using the
corpora for qualitative evaluation (i.e.  just looking at what happens and
what goes wrong), it will be easy to extract some quantitative evaluation
information, such as dialogue duration, certain types of error rate, task
success rate, and user satisfaction.

To be maximally useful for evaluation, we propose to collect some corpora
at a certain time, and in a certain way.  We propose collecting at least
one corpus for each finished version of the demonstrator (according to the
initial plans, there will be three versions of the demonstrator with
dialogue manager).  The last corpus can be used as a final evaluation of
the outcome of the projects. Each of these (three) corpora should be
collected using an appropriately controlled experiment (i.e. a properly
designed scenario with appropriate choice of users).