Multimodal Comprehension of Language and Graphics
Graphs, in particular line graphs and bar graphs, are successful means to present data, both in the task of analyzing the data and in the task of communicating the results of data analysis. They are used extensively in scientific publications, textbooks, magazines and newspapers; in print media and in internet documents line graphs and bar graphs are the dominant, i.e. most frequently used, types of graphs in addressing non-experts in communication through graphs. In addition to text-graph documents, in many professional communication and classroom settings, graphs, language, and often gestures accompany each other in multimodal processes of communication.
Due to these common properties of language and graphs, namely coupling of syntax and semantics, we propose – on the basis of current models for language comprehension and models for graph comprehension – the parallelism between these processes leading to the construction of cross-modal referential links. The backbone of the cognitive architecture for multimodal comprehension is a common system of conceptual and spatial representations, which enables the coupling of internal representations computed in the two modality-specific comprehension paths. The architecture and the proposed mechanisms are justified by a series of experimental studies, in particular eye movement studies.
Gestures and graphical communications are visuo-spatial modalities, which share similar perceptual visuo-spatial features to convey meaning such as quantity, direction and relations. Therefore we focus on the similarities and the differences among the “vocabularies” of gestures, speech and diagrams as used in graph comprehension and graph description. In a series of experiment, we investigated the conceptualization of events and of graph-segments by focusing on gesture production and verbal descriptions of visually and of haptically perceived graphs, line graphs as well as bar graphs.
Based on the research concerning text-graph comprehension by seeing people, we extended our research focus to assisted haptic-graph exploration. Our long-term goal is to realize an automatic, i.e., computational, verbal assistance system that has the capability to provide instantaneous support for haptic-graph explorers during their course of exploration. For developing such an assistance system—beyond realizing components considering natural language generation and user-system interaction—empirical studies are needed to understand the underlying principles of haptic graph exploration, of conceptualizing graphs, and of communicating about graphs in a assisted graph-comprehension setting. We conducted experiments with single blindfolded participants exploring haptic graphs (of different types) as well as, experiments of participant pairs, a blindfolded haptic explorer and a verbal assistant who was able to observe the haptic explorations.