TITLE:

Multimodality in Human-Computer Interfaces

PRESENTERS:

Michael Johnston, AT&T Labs Research
Srinivas Bangalore, AT&T Labs Research

DESCRIPTION:

The ongoing convergence of the web with telephony, through
technologies such as Voice over IP, high-speed mobile data networks,
and handheld computers and smartphones, enables the creation of
natural and highly effective multimodal interfaces for human-human
communication and human-machine interaction with automated
services. These interfaces allow for user input and system output to
be optimally distributed over multiple different modes such as speech,
pen, and graphical displays.  Research on the computational processing
of language has primarily focussed on linear sequences of speech or
text where the primitive elements are phonemes, morphemes, or
words. Multimodal language can be distributed over two or three
spatial dimensions as well as the temporal dimension and involve
additional primitive elements such as gestures, drawings, tables, and
charts.  This tutorial provides an overview of the problem of
multimodal language processing and detailed examples showing how
representations and techniques from speech, language, and dialog
processing can be extended and applied to the parsing, integration,
understanding of multimodal inputs and the planning, generation, and
presentation of multimodal outputs.