|
|
 |
|
Adaptive multimodal natural language generation
Speech, prosody & corpus-based methods
Multimodality, cognition and evaluation
Publications
Workshops
Multimodality, cognition and evaluation
The aim of this subproject is to gain insights in how users interpret and evaluate multimodal output, and whether or not users prefer particular combinations of modalities to other combinations or prefer unimodal variants. This knowledge can be used to make more informed decisions for automatic multimedia generation. So far, we have carried out several studies to investigate multimodal output and modality effect, focusing on the question how people process and evaluate unimodal and multimodal information in the medical domain.Van Hooijdonk and Krahmer (to appear) report on experiments that have been conducted to investigate which combinations of modalities (e.g., text vs. picture vs. film clip; text and picture vs. text and film clip; speech and picture vs. speech and film clip) are most efficient and effective for illustrating RSI exercises, paying special attention to cognitive load of learners and the special characteristics of procedural learning (i.e., acquiring skills rather than knowledge).
Another study focused on text-structure and spatial cognition. For this study, we conducted an exploratory thinking aloud experiment in which users had to find information in a large medical web site. The thinking aloud protocols that were collected in this study were analysed in order to gain insights in users’ spatial orientation and search strategies in this domain. In particular, we investigated the way in which spatial descriptions are used to conceptualize users' search tasks. See Van Hooijdonk et al. (2004, 2005, 2006).
Recently we have carried out a study investigating which (combinations of) modalities people choose to answer different types of questions when simulating a QA system. To this end we conducted two experiments, one in which we asked people to create multimodal answer presentations for different medical questions, and one in which these multimodal presentations were evaluated by other people (van Hooijdonk et al., 2007).
For an overview of all results in this subproject, see Van Hooijdonk (2008).
References:
- Hooijdonk, C.M.J. van (2008). Explorations in Multimodal Information Presentation. PhD thesis, Tilburg University, The Netherlands.
- Hooijdonk, C.M.J., van, W. Bosma, E. Krahmer, A. Maes and M. Theune (2011). Experiments in multimodal information presentation. In A. van den Bosch and G. Bouma (eds.), Interactive Multi-modal Question-Answering, Springer Verlag Berlin-Heidelberg, pp. 89-115. © Springer
- Hooijdonk, C.M.J., van and E. Krahmer (2008), Information modalities for procedural instructions: the influence of text, static and dynamic visuals on learning and executing RSI exercises. IEEE Transaction on Professional Communication, 51(1), pp. 50-62.
- Hooijdonk, C.M.J. van, A. Maes, and N. Ummelen (2004) Users' representation of spatial information on web sites. Proceedings of European Association for Research and Learning and Instruction, Special Interest Group Comprehension of Text and Graphics: Basic and Applied Issues, Valencia, Spain: EARLI SIG, pp. 243-247.
- Hooijdonk, C.M.J., van, A. Maes, and N. Ummelen (2005) Using thinking aloud data to characterize cognitive tasks in hypertext. Proceedings of the IEEE Professional Communication Society, International Professional Communication Conference, 10 - 13 July 2005, Limerick, Ireland, pp. 546-553.
- Hooijdonk, C.M.J., van, A. Maes and N. Ummelen (2006). "I have been here before": An investigation into spatial verbalizations in hypertext navigation. Information Design Journal 14(1), pp. 5-19.
- Hooijdonk, C. van, J. de Vos, E. Krahmer, F. Maes, M. Theune and W. Bosma (2007). On the role of visuals in multimodal answers to medical questions. In the Proceedings of the 2007 Conference of the IEEE Professional Communication Society (IPCC), 30 September - 3 October 2007, Seattle, USA.