LIMAH: Linking Media in Acceptable Hypergraphs

Available multimedia content is rapidly increasing in scale and diversity, yet today, multimedia data remain mostly unconnected, i.e., with no explicit links between related fragments. The project investigate multimedia content linking, where linking  refers to the creation of explicit and meaningful links between multimedia documents, or fragments of documents.

The idea of explicit links between multimedia contents can be traced back to Memex (Bush, 1945), the vision of Vannevar Bush which elaborates on the notion of hypertext and hyperlink in multimedia data. Significant work has been undertaken in the field of hypermedia, with pioneering experiments such as HyperCafe (Sawhney et al., 1996), focusing on the storage and representation of the links as well as on user experience. However, automatic authoring of links is disregarded. Most of the research effort in automatic multimedia content analysis has been devoted to describing and indexing content on which core tasks—e.g., information retrieval, summarization, recommendation—are built to develop multimedia applications for users. Creating and characterizing explicit links within multimedia collections, as illustrated in the figure, builds on multimedia content analysis techniques to go one step further and potentially improve core tasks and applications to ultimately provide better services to users.

In this global picture, LIMAH seeks to develop methodology and technology for the automatic or semi-automatic creation and characterization of links in multimedia data by exploring hypergraph construction techniques from multimedia collection in close relation with acceptability and usage. The key idea is to provide structured collections, with various types of links along with link characterization, in an hypergraph structure which offers meaningful advantages for core tasks and user-oriented applications. Meaningfulness encompasses multiple facets (e.g., relevance of the links, exploitability by users, added-value on core tasks) and will be investigated in a number of ways, from objective measures on core tasks to user acceptability and satisfaction on key pilot use-cases, along with prospective concerns on the evolution of media consumption modes and of laws. While not directly related to technology, these last concerns appear as crucial to find the right societal outcome for technological advances and to nurture prospective thinking on multimedia science.

Creating explicit links heavily relies on the ability to measure the proximity between two items, e.g., based on collaborative filtering or on information retrieval techniques for text, audio and images. LIMAH focuses on content-based proximity, with a strong emphasis on language data which provide links of high semantics. While we will make heavy use of existing technology for content comparison in all modalities, LIMAH will further develop fine grain linguistic analysis of texts, transcripts and social data for two main reasons. On the one hand, pairing linguistic elements at a fine grain semantic level is a crucial step towards links of high added-value for users, in particular regarding social data which have seldom been exploited in media aggregation scenarios. On the other hand, such techniques provide the basis to characterize links by providing users with a (semantic) explanation of the link.

Application use cases

In a nutshell, LIMAH develops scientific and methodological basis to organize multimedia collections as hypergraphs combining links of different natures and evaluates the impact of structured collections on usages and technology in a global perspective encompassing ICT, law and human science. However, creating links and evaluating usages can hardly be done ex nihilo and requires use cases and usage scenarios on which evaluation will be performed. LIMAH will study hypergraph authoring and acceptability in two distinct complementary use-cases, namely, navigation in news data and learning with online courses. Each use case rely on common technological foundations while exhibiting a different problematic:

• Navigation in news data builds upon well-known multimodal material, including newspapers and broadcast news, along with associated user generated and social data. We will build on a solid experience of some of the partners in this domain, reusing existing data whenever possible, possibly with human generated links to compare with. We will exploit existing techniques for content comparison, mostly contributing on hypergraph construction and evaluation, on social data processing and on the legal and sociological prospective.
• Learning in online courses targets educational data, in relation with the educational platform of Comin Labs, with the goal of establishing links between lecture videos, lecture notes, slides, related resources on the Web (e.g., Wikipedia), students' questions, comments and opinions. Linking educational material has been somewhat less studied, in terms of content-analysis, in terms of targeted applications and usages. Moreover, educational data are less structured than news data, offer a greater variety of contents and have links targeting very precise media fragments (i.e., answer to a question, pointer to a definition, slide synchronization). Such data also raise different copyright and editorial liability issues than professional media.

Methodology

With the goal of providing advanced technology to organize news and educational multimedia collections with links in an hypergraph structure, contributions in LIMAH are organized around two main interrogations:

1. How to automatically build from a collection of multimedia documents an hypergraph which provides exploitable links?
• We will develop hypergraph construction algorithms exploiting pairwise comparison of contents and relying on nearest neighbor graphs paradigms as a starting point. Accounting for the timeliness of the data, pruning graphs to avoid explosion in the number of links while maintaining link relevance, combining multiple nearest neighbor graphs (e.g., resulting from comparisons along different angles and modalities) are key questions that the project will address. Another key issue that we will study in tight relation with usages is the granularity of the links sources and targets.
• We will improve the state of the art in fine grain linguistic comparison of language data, with a strong focus on social data. In this line of work, the two main contributions will be on multilevel text comparison and on opinion analysis in social data. Combining sentence alignment techniques—which provides fine semantic proximity—with coarse grain comparison using the vector space model will be considered in the former. In the latter, we will contribute advances in contextual interpretation and dynamic evolution of opinions in degraded text data.
• We will develop technology and methodology for the characterization of links so as to provide user-friendly explanations of the links established. We will investigate characterization techniques to qualify links in all their dimensions (similar/opposite, redundant/complementary, cause/consequence, etc.), developing typology elements and related machine learning approaches while considering serendipity in relation with usages. Moreover, characterization of opinions directly contributes to link characterization.
2. How hypergraph collections with explicit links modify usage of multimedia data in all aspects?
• We will evaluate impact of various hypergraph structures on users, performing user studies in a lab setting. Adequacy between collection structuring algorithms and usability of structured collections for end-users will be studied from different angle—graph construction strategy, types and targets of links, characterization of links, etc.—with particular attention on user disorientation which might be emphasized in linked media. In the lecture scenario, we will focus on evaluating what structure information help e-learners and what use they make of such information.
• As an illustration of the impact of hypergraph structuring on core tasks, we will revisit multiple document summarization in the framework of linked media contents. We will investigate the use of links along with their characterization in multi-document summarization. We will extend existing criteria such as maximal marginal relevance, taking into account links to limit redundancy and increase coverage.
• We will study the evolution of usages and practices from the sociological and legal standpoints. On the one hand, we will consider the different aspects of copyrights in linked media contents, especially exploring professional data. The qualification of works resulting from linked media will be studied, exploring exceptions to copyrights to deal with new, fragmentary, linked contents. On the other hand, we will study the conditions and possibilities for fixing new media uses and explore the negotiations between media actors and their mobilization of action potentialities.

The main outputs of the project are technological advances to organize a multimedia collection with links of different natures, evaluation of such a structuring on media practice along with demonstration platforms on the use-cases considered, and a better insight regarding the evolution of media practices and law in light of linked media content.