19^th International CODATA Conference
Category: Data Visualization

Caliph & Emir: Semantics in Multimedia Retrieval and Annotation

Mathias Lux (mlux@know-center at), Werner Klieber (wklieber@know-center at), Michael Granitzer (mgrani@know-center at)
Know-Center Graz, Austria
http://www.know-center.at

Applying Information Retrieval methods directly to multimedia objects like images, video audio documents or interactive presentations, is often not possible without adaptations of existing algorithms. Aiming for cross media retrieval, which allows retrieval across different formats and media types, a common denominator for different media types and formats has to be found. A common approach is the usage of metadata in a way that does not differentiate between single media types and formats. For multimedia objects one of these already specified metadata schemes is MPEG-7. On the one hand MPEG-7 allows low level and technical descriptions like color distributions or blockiness of an image. On the other hand manual annotation of multimedia objects in different levels, ranging from free text annotations to complex computer-readable structured semantic descriptions, is supported. The paper we present is twofold. In the first half a current research project prototype called "Caliph & Emir" is presented. It allows annotation and retrieval of semantic descriptions, created and stored as directed graphs, in the context of digital photos. The annotation and retrieval capabilities are based on MPEG-7. In addition to semantic descriptions common content based image retrieval mechanisms were implemented and widely used metadata formats for digital photos like EXIF, Adobe's XMP and IPTC IIM were integrated. Caliph, which stands for "Common And Light weight PHoto Annotation", is the annotation part supporting graphical annotation and graph visualization. Emir, which is short for "Experimental Metadata based Image Retrieval", is the retrieval engine. Besides the prototype description the value added by semantic descriptions is discussed. The second part gives an outlook on future work. The next step in development is merging the retrieval capabilities of "Caliph & Emir" with another research project, a cross media retrieval and visualization application. We also outline further developments towards cross media applications in general and we discuss the integration of existing ontologies for semantic annotation and retrieval in combination based on MPEG-7.