Project acronym	MIAMM
Project full title	Multidimensional Information Access using Multiple Modalities
Proposal/Contract n°	IST-2000-29487
Operative commencement date of contract	1/09/01

Summary

The objective of the MIAMM project is to provide an integrated and comprehensive framework for the design of modular multidimensional/multimodal dialogue systems. The work within MIAMM will be achieved along the following lines:

The MIAMM platform will be based upon the definition of a series of interaction scenarios for multidimensional interaction, comprising constraints on the use of modalities, the relation with the underlying application and the graphical presentation procedures that have to be implemented;

A haptic and tactile device for multidimensional interaction will be designed and integrated within the platform after an initial evaluation phase of the current available technology. This will be integrated into the prototype with an adequate visualization environment and gestural interpretation algorithms;

Fully multi-lingual understanding and generation components will be integrated within the MIAMM platform by incorporating and adapting the latest technologies in speech recognition and synthesis, and combining several alternative parsing technologies (statistical, automata, local syntax). Interface through the multi-modal interface language (MMIL) will result in language easily pluggable software modules;

A dialog manager will be designed to combine the information available from the underlying application, the language modules, the haptic device and the graphical interface. Exchanging information by means of the MMIL language, it will manage its own user representation to integrate the latest scientific development concerning anaphora and deixis interpretation, as well as smooth dialog management to cope with both vocal and gestural interaction;

Human factor analysis will drive the development of the multi-modal user interface. For the haptic interaction human factor research has to find out for which purposes and actually how a person would use the haptic device. This will directly lead to semantic concepts related to haptic input and output gestures.

Objective(s)

Rationale
Available online information resources grow exponentially in their content and distribution.
Accessing data and navigating in it gets ever more complicated and unsatisfying, especially for the non IT specialist.
If the user queries the system with speech input, these problems are even more severe. Casual users of an information service usually rest on the capability of the dialog partner to disambiguate the query using background knowledge and a model of the user’s information state. Unfortunately, dialog system does not usually have access to these knowledge sources.
In most multimedia applications when the — inexperienced — user might not know how the information is structured or has no precise idea on what item he is looking for, the dialog system should be flexible enough to offer various interactions. The user should be able to use the most appropriate means (speech, text, gesture, pointing device, etc.) and get feedback from the system (speech, text, graphic, force feedback, etc.). Speech input might be convenient for the first query to the system, but later, when the search is narrowed, it might be difficult for the user to express everything by speech.
Innovative products in this field will facilitate

the development of e-commerce applications by making the systems more accessible
the access for non-IT skilled people to information services.

Both points correspond to major objectives of the IST programme and the eEurope initiative. The MIAMM project will directly address these needs by developing innovative research in multi modal dialog technologies to integrate them in innovative products.

Objectives
The main objective of the MIAMM project will be to develop new concepts and techniques in the field of multi-modal dialogs to allow fast and natural access to multimedia databases. This will imply both the integration of available technologies in the domain of vocal interaction and multimedia information extraction, and the design of novel technology for gesture designation and manipulation coupled with an adequate graphical presentation logic. In this domain, the MIAMM approach will be to produce a device independent result format that will account for future technological evolutions in interactive devices, in particular haptic (i.e. gestural with forcefeedback) interfaces. MIAMM will also explore how a coupled haptic / speech recognition device could be implemented in portable device.
Specific emphasis will be put in the project on:

Producing fully specified evaluation methods and protocols for multi-modal interaction which can serve as a roadmap for the development of future devices and systems;
Designing generic middleware components based on standardized exchange protocols.
Prototyping the integration of speech & haptic interfaces in a mobile device

Hence, the MIAMM prototype will serve as a test bed for exploring new types of man-machine interaction techniques and/or devices in the domain of information access applications.
The MIAMM platform is based upon a modular approach where the various components exchange results through a shared multi-modal interface language (MMIL). The MMIL will comprise, among other information components, gesture trajectory information, speech recognition and understanding information, queries and result sets addressed to or received from the multimedia database, as well as user specific information. Based on the XML format, it will integrate already standardized components (e.g. for speech) and will lead to specific propositions for extending those components (e.g. metadata features for multimedia information) or even for defining new standards (e.g. gesture information representation).
As a whole, the main contribution of the project will be to provide an integrated and comprehensive framework for the design of modular multidimensional/multi-modal dialog systems based on precisely defined scenario, design recommendations and protocols.

Validation
To validate the MIAMM platform, a typical application dealing with consumer electronics (electronic TV program guide, music title selection from CD/MD jukebox and MP3 Internet archive) will be developed. This application is large enough to demonstrate the need for advanced user interface mechanism and small enough for a dialog system to be implemented and real evaluation procedure to be undergone. In such application, the data delivered by the information service can be structured and accessed in multiple dimensions.
One of the MIAMM research topics will be on reducing the size of the search along certain dimensions using a haptic device with force feedback, and its combination with natural language interaction. Such an experiment can only be contemplated if the relevant information is presented to the user in a way that is compatible with multi-modal designation. Furthermore, speech recognition is error prone, especially when a large list of possible alternatives exists and the language model cannot reduce the alternatives drastically. In such cases handwriting might be a useful input modality.
This application will allow the MIAMM project to develop novel representation and access techniques for multimedia document access and retrieval. The application will also aim at combining recent international standards (RDF, Dublin Core) to provide a homogeneous representation of a whole range of media, with enough flexibility to account for possibly novel media that may appear in the future.
The difficulty of this type of application lies in the possible gap between the user and the system. Requests with an approximate (e.g. wrong music title) or vague (I would like to listen to some cheerful music) description or broad questions on the database (What Jazz music do you have) might result in a huge number of titles. The task of the dialog system is then to help the user to reduce the number of alternatives along some of the above-mentioned dimensions until a small set of titles is selected.

Benefits to the users
The MIAMM project directly aims at providing the benefits of the Information Society to user of new multimedia information services, including people who would otherwise be left aside. Direct benefit will include:

An easy access to information systems by providing flexible access through flexible interfaces
adapting to the context of the query
Access to new sources of information, education, entertainment or commerce for people who
might have reluctance to use information retrieval systems.

Measuring achievements
Measurement of achievements will be precisely specified by all participants at the beginning of the project. Past experience of similar projects will be considered. Measurements will be done according to the following criteria:

Degree of acceptance of the prototype by non-expert users as evaluated in the final phase of the
project;
Evaluation of final results concerning haptic based multi-modal interfaces through wide
interaction with the corresponding scientific community (publication in majors conferences in
the field);
Availability of autonomous linguistic modules for the three languages considered in the project.