Challenges in Multimedia Information Retrieval & Filtering 薛向阳 xyxue@fudan.edu.cn 复旦大学计算机科学与工程系 上海市智能信息处理重点实验室

Outline Potential Applications, Query Examples & Achievements Basic Concepts & Architectures Key Techniques & Problems 2018/11/22 薛向阳 － 复旦大学计算机科学系

Many Potential Applications Broadcast media selection (e.g. radio channel, TV channel) Cultural services (e.g. history museums, art galleries) Digital libraries (e.g. image catalogue, musical dictionary, bio-medical imaging catalogues, film, video and radio archives) Journalism (e.g. searching speeches of a certain politician using his name, his voice or his face) Multimedia directory services (e.g. yellow pages, Tourist information) …… 2018/11/22 薛向阳 － 复旦大学计算机科学系

Video Query Examples（TREC） a specific person I want all the information you have on Ronald Reagan a specific thing I'm interested in any material on Hoover Dam. I'm looking for a picture of the OGO satellite 2018/11/22 薛向阳 － 复旦大学计算机科学系

Informedia – CMU Establishment of large video libraries as a searchable information resource Full content information retrieval in both spoken language and video/image domains Integration of speech, image and natural language understanding for library creation and exploration Fully automated transcriptions generated entirely speech recognition or with closed captions Information summaries at varying detail, both visually and textually 2018/11/22 薛向阳 － 复旦大学计算机科学系

CueVideo – IBM Developing fully automatic means for indexing, hyper-linking and preparation of media material for effective searching and browsing by users Combines several automated indexing，searching and browsing tools Video analysis and summarization Use of speech recognition for spoken media retrieval 2018/11/22 薛向阳 － 复旦大学计算机科学系

Outline Potential Applications, Query Examples & Achievements Basic Concepts & Architectures Key Techniques & Problems 2018/11/22 薛向阳 － 复旦大学计算机科学系

An Instance of IR System Query String Document Corpus Ranked Documents 1. Doc1 2. Doc2 3. Doc3 . 2018/11/22 薛向阳 － 复旦大学计算机科学系

Information Retrieval Information Retrieval (IR) Deals with: Representation (or Modeling) Storage Organization Access of / to Information Items 2018/11/22 薛向阳 － 复旦大学计算机科学系

Architecture：IR offline Multi - Modal User Interface Representation ， Modeling Relevance feedback Description (MPEG -7/XML) Multimedia Query Processi ng Database Organizing: Index Structure Searching Ranking 2018/11/22 薛向阳 － 复旦大学计算机科学系

Information Filtering Generally, the goal of an Information Filtering (IF) system is to sort through large volumes of dynamically generated information and present to the user those which are likely to satisfy his or her information requirement 2018/11/22 薛向阳 － 复旦大学计算机科学系

Architecture：IF representation 2018/11/22 薛向阳 － 复旦大学计算机科学系

Applications using MPEG7 2018/11/22 薛向阳 － 复旦大学计算机科学系

Comparison：IR & IF Information Retrieval Information Filtering User Information Needs or Query – Varying Database or Collection – Static Information Filtering User Information Needs or Profile – Static Incoming Data – Varying Common to both how to represent information how to select relevant information 2018/11/22 薛向阳 － 复旦大学计算机科学系

Outline Practical Applications, Query Examples & Achievements Basic Concepts & Architectures Key Techniques & Problems 2018/11/22 薛向阳 － 复旦大学计算机科学系

Digital TV Program Filtering & Searching System DVB-S DVB-C MPEG2 TS Filtering XML Search Engine Database: >2TB User Shot Key-frame Speech Ocr Scene Face Motion … Template 2018/11/22 薛向阳 － 复旦大学计算机科学系

Representation – extract low level features Text Features stop word elimination, stemming, index term selection, thesauri, word cut… Image and Video Features color, texture, shape, motion, … Audio (Speech,Music) Features zero-crossing ratio, short time energy Spectral, Spectral Flux, Spectral Centroid, LPC, MFCC Pitch,Rhythm,Timbre,… Requirements － Good Representation, Fast, Automatic, Robust 2018/11/22 薛向阳 － 复旦大学计算机科学系

Representation – get high level features Structured Video Analysis Video – Scene – Shot – Key frame Summaries at varying detail, both visually and textually Audio & Visual Object Recognition Face,Character,Car,… Word Spotting,Speech Recognition,Speaker,… Problem - Low Precision, Infant, Inevitable Incompleteness in the Representation,… 2018/11/22 薛向阳 － 复旦大学计算机科学系

2018/11/22 薛向阳 － 复旦大学计算机科学系

Retrieval Model Boolean Model Vector Model Probabilistic Model Fuzzy Set Model Neural Network …… 2018/11/22 薛向阳 － 复旦大学计算机科学系

Storage & Organization Standardized Descriptors - MPEG-7 Management of XML Documents Index Structures – For Fast Query Inverted File for Text Index Structure for XML Documents Index Structures for High Dimensional Vector (Visual Features) - Dimensionality Curse 2018/11/22 薛向阳 － 复旦大学计算机科学系

Curse of Dimensionality An Intuitive Explanation Assume n-dimensional points distributed in super-cubic. Selectivity can be computed： When n increasing, P(n) will go down to zero exponentially. In order to find relevant points, searching window should be enlarged! D （0,0） r 2018/11/22 薛向阳 － 复旦大学计算机科学系

Multi-Modal Interface － 1 Input Information Needs Key Word,… Example Image, Example Face, Example Video Clip,… Speech, Humming,… Relevance feedback How to submit user’s query easily and friendly to IR system? How can IR system understand user’s query intention? People are unable to specify that which they don't know There is inevitable uncertainty in the representation or understanding of information problems 2018/11/22 薛向阳 － 复旦大学计算机科学系

Multi-Modal Interface － 2 Output Query Results Enable user to browse full content in hierarchy or web Visualization is important for presentation 2018/11/22 薛向阳 － 复旦大学计算机科学系

How to Compute Relevance? Relevance is a dynamic and idiosyncratic relationship between person and information object Information objects mean many different things to different people (or the same person at different times) There is inherent uncertainty in the relevance relationship 2018/11/22 薛向阳 － 复旦大学计算机科学系

Comparison：IR & DR DR（数据检索） IR（信息检索） Matching Exact Partial, Fuzzy Inference Deduction Induction Model Deterministic Probabilistic Classification Monothetic Polythetic Query Language Artificial Natural Query Specification Complete Incomplete Items Wanted Relevant Error Response Sensitive Insensitive 2018/11/22 薛向阳 － 复旦大学计算机科学系

Conclusion Many types of data without strict structure in huge multimedia database Almost all algorithms of intelligent information processing and recognition (audio & visual) are necessary for better representation Seeking good retrieval model may be key to reduce gap between person and computer Uncertain & chaotic task – unable to be formulated 2018/11/22 薛向阳 － 复旦大学计算机科学系

Q/A？ Thank You！ 2018/11/22 薛向阳 － 复旦大学计算机科学系