|Project full title||Multidimensional Information Access using Multiple Modalities|
|Operative commencement date of contract||1/09/01|
The objective of the MIAMM project is to provide an integrated and comprehensive framework for the design of modular multidimensional/multimodal dialogue systems. The work within MIAMM will be achieved along the following lines:
Available online information resources grow exponentially in their content and distribution.
Accessing data and navigating in it gets ever more complicated and unsatisfying, especially for the non IT specialist.
If the user queries the system with speech input, these problems are even more severe. Casual users of an information service usually rest on the capability of the dialog partner to disambiguate the query using background knowledge and a model of the users information state. Unfortunately, dialog system does not usually have access to these knowledge sources.
In most multimedia applications when the inexperienced user might not know how the information is structured or has no precise idea on what item he is looking for, the dialog system should be flexible enough to offer various interactions. The user should be able to use the most appropriate means (speech, text, gesture, pointing device, etc.) and get feedback from the system (speech, text, graphic, force feedback, etc.). Speech input might be convenient for the first query to the system, but later, when the search is narrowed, it might be difficult for the user to express everything by speech.
Innovative products in this field will facilitate
the development of e-commerce applications by making the systems more accessible
the access for non-IT skilled people to information services.
Both points correspond to major objectives of the IST programme and the eEurope initiative. The MIAMM project will directly address these needs by developing innovative research in multi modal dialog technologies to integrate them in innovative products.
The main objective of the MIAMM project will be to develop new concepts and techniques in the field of multi-modal dialogs to allow fast and natural access to multimedia databases. This will imply both the integration of available technologies in the domain of vocal interaction and multimedia information extraction, and the design of novel technology for gesture designation and manipulation coupled with an adequate graphical presentation logic. In this domain, the MIAMM approach will be to produce a device independent result format that will account for future technological evolutions in interactive devices, in particular haptic (i.e. gestural with forcefeedback) interfaces. MIAMM will also explore how a coupled haptic / speech recognition device could be implemented in portable device.
Specific emphasis will be put in the project on:
Producing fully specified evaluation methods and protocols for multi-modal interaction which can serve as a roadmap for the development of future devices and systems;
Designing generic middleware components based on standardized exchange protocols.
Prototyping the integration of speech & haptic interfaces in a mobile device
Hence, the MIAMM prototype will serve as a test bed for exploring
new types of man-machine interaction techniques and/or devices in the
domain of information access applications.
The MIAMM platform is based upon a modular approach where the various components exchange results through a shared multi-modal interface language (MMIL). The MMIL will comprise, among other information components, gesture trajectory information, speech recognition and understanding information, queries and result sets addressed to or received from the multimedia database, as well as user specific information. Based on the XML format, it will integrate already standardized components (e.g. for speech) and will lead to specific propositions for extending those components (e.g. metadata features for multimedia information) or even for defining new standards (e.g. gesture information representation).
As a whole, the main contribution of the project will be to provide an integrated and comprehensive framework for the design of modular multidimensional/multi-modal dialog systems based on precisely defined scenario, design recommendations and protocols.
To validate the MIAMM platform, a typical application dealing with consumer electronics (electronic TV program guide, music title selection from CD/MD jukebox and MP3 Internet archive) will be developed. This application is large enough to demonstrate the need for advanced user interface mechanism and small enough for a dialog system to be implemented and real evaluation procedure to be undergone. In such application, the data delivered by the information service can be structured and accessed in multiple dimensions.
One of the MIAMM research topics will be on reducing the size of the search along certain dimensions using a haptic device with force feedback, and its combination with natural language interaction. Such an experiment can only be contemplated if the relevant information is presented to the user in a way that is compatible with multi-modal designation. Furthermore, speech recognition is error prone, especially when a large list of possible alternatives exists and the language model cannot reduce the alternatives drastically. In such cases handwriting might be a useful input modality.
This application will allow the MIAMM project to develop novel representation and access techniques for multimedia document access and retrieval. The application will also aim at combining recent international standards (RDF, Dublin Core) to provide a homogeneous representation of a whole range of media, with enough flexibility to account for possibly novel media that may appear in the future.
The difficulty of this type of application lies in the possible gap between the user and the system. Requests with an approximate (e.g. wrong music title) or vague (I would like to listen to some cheerful music) description or broad questions on the database (What Jazz music do you have) might result in a huge number of titles. The task of the dialog system is then to help the user to reduce the number of alternatives along some of the above-mentioned dimensions until a small set of titles is selected.
Benefits to the users
The MIAMM project directly aims at providing the benefits of the Information Society to user of new multimedia information services, including people who would otherwise be left aside. Direct benefit will include:
An easy access to information systems by providing flexible access through flexible interfaces
adapting to the context of the query
Access to new sources of information, education, entertainment or commerce for people who
might have reluctance to use information retrieval systems.
Measurement of achievements will be precisely specified by all participants at the beginning of the project. Past experience of similar projects will be considered. Measurements will be done according to the following criteria:
Degree of acceptance of the prototype by non-expert users as evaluated in the final phase of the
Evaluation of final results concerning haptic based multi-modal interfaces through wide
interaction with the corresponding scientific community (publication in majors conferences in
Availability of autonomous linguistic modules for the three languages considered in the project.