Research Projects

Smartphones have evolved considerably in processing power over the last years. They now feature multi-core CPUs as well as GPUs and consumer-quality cameras up to HD resolution. This makes them an interesting platform for graphics and vision and opens new opportunities for research.

The aim of AR4DOC is to facilitate the task of document inspection by a human operator. This requires the person to have detailed knowledge about the nature of a document, which may be outdated or even unavailable at the time of inspection.

We seek to provide this information in an interactive way using Mobile Augmented Reality (AR), so that a well-grounded decision on the vailidity of a document is possible. This involves several tasks such as document localization, recognition, tracking, presentation as well as interaction.

The generation of realistic 3D models of whole cities has become a vibrant and highly competitive market through the recent activities of, most notably, Goggle Earth and Microsoft Virtual Earth. While the first generation of these systems only delivered high-quality zoomable images of the ground, the current trend is heavily geared towards 3D – that is, users can access three-dimensional height- fields of the terrain, and even 3D models of individual buildings. Simple building models, basically extruded polygons with different types of roofs, can be generated today from aerial images completely automatically. This is a solved problem. Far from solved, however, is the problem of generating automatically detailed buildings with façades. Input data for this problem are registered range maps obtained by stereo matching and sequences of highly overlapping thus redundant images (taken from a car driving in the road) where each pixel has not only a color but also a depth, a z-value. Although range maps can be directly rendered in principle, the data size is huge and, more importantly, the pixels have no semantics: A priori there is no difference between a pixel on the floor, on the wall, or on a door. But these shape semantics are required by all downstream applications using the city model. Shape grammars, on the other hand, have recently become (again) a popular method in research for representing 3D buildings. Their great advantage is that they allow to parameterize buildings, which can be used for populating virtual cities with believable architectural buildings, e.g., for 3D games. The goal of the CITYFIT project is, given highly redundant input imagery and range maps from an arbitrary building in Graz, to synthesize a shape grammar that, when evaluated, creates a clean, CAD- quality reconstruction of that building that fits the original data very closely and makes the semantics of all major architectural features explicit. These shape semantics can even be transferred back to inform the original data, so each of these “semantically enriched” data points can tell whether it belongs to ground, wall, or door.

The systematic creation of models of the real world to support the locational awareness on the Internet can be achieved if previously required massive manual labor gets replaced by automated procedures. A particular challenge exists in the automation of the extraction of the 4 classical map features buildings, circulation spaces (e.g. road networks), vegetation and water bodies, as well as their interaction. Decennia of research have been unable to automate the extraction of these features from classical aerial photography towards an economically viable result. However, we believe that we can succeed in the proposed project to develop automated procedures to create feature data for three reasons. First is the recent advent of digital aerial sensors producing highly redundant digital large format aerial photography. Redundancy will be obtained by using high forward and side overlaps, say at 80% and 60%, so that every point in the terrain is imaged at least 10 times, and any algorithm can rely on multiple analysis results that then can either reinforce or cancel one another. Second, the geometric redundancy gets augmented by a radiometric redundancy using 4 spectral bands, adding an infrared band to the classical red, green and blue color channels. Third, we will combine the classical "object reconstruction" approach available from stereo procedures, by new recognition methods. While classically a "car" on a street may have been seen via a "point cloud" and would have to get recognized simply by a representation of local height anomaly on an otherwise flat reference surface, recognition includes the use of stored images of cars in a data base to actually recognize a car as a human would do when inspecting an aerial image. The project is split up into five work packages which will focus on how reconstruction and recognition techniques can help each other and how additional information either from a previous mission or GIS can be integrated in the 3D modeling framework. One work package will address the assessment of the obtained quality, another will address project management and dissemination activities. Within the project we will develop an extensive library of combined recognition/reconstruction methods, and apply them to a range of test data sets. Test data will vary in geometric resolution (pixel size), overlaps, and types of terrain scenarios.

Multimedia data has a rich and complex structure in terms of inter- and intra-document references and can be an extremely valuable source of information. However, this potential is severely limited until and unless effective methods for semantic extraction and semantic-based cross-media exploration and retrieval can be devised. Today’s leading-edge techniques in this area are working well for low-level feature extraction (e.g. colour histograms), are focussing on narrow aspects of isolated collections of multimedia data, and are dealing only with single media types. MISTRAL follows the following lines of radically new research: MISTRAL will extract a large variety of semantically relevant metadata from one media type and integrate it closely with semantic concepts derived from other media types. Eventually, the results from this cross-media semantic integration will also be fed back to the semantic extraction processes of the different media types so as to enhance the quality of the results of these processes. MISTRAL will focus on most innovative, semantic-based cross-media exploration and retrieval techniques employing concepts at different semantic levels. MISTRAL addresses the specifics of multimedia data in the global, networked context employing semantic web technologies. The MISTRAL results for semantic-based multimedia retrieval will contribute to a significant improvement of today’s human-computer interaction in multimedia retrieval and exploration applications. New types of functionalities include but are not limited to:

• cross-media-based automatic detection of objects in multimedia data: For example, if a video contains an audio stream with barking together with a particular constellation of video features, the system can automatically consider the features in the video as an object "dog".
• semantic-enriched cross-media queries: A sample query could be "find all videos with a barking dog in the background and playing children in the foreground".
• cross-media synchronisation: The idea is to synchronize independent types of media according to the extracted semantic concepts. For example, if users see somebody walking in a video, they should also hear footfall from an audio.

In this project textual, formal and structural interactions between fundamental theology and media society will be researched. The most important facts which influence the media society of the western world are internet, multi media lifestyle and religious symbolism. This project which is carried out together with the Institute for Fundamental Theology, Graz University and Joanneum Research Graz.

The Institute for Computer Graphic and Vision mainly deals with the semi automatically retrieving of religious symbols. Religious symbols were in former times mainly used only in liturgical events. Today they are media effective prepared and presented. Examples are the presentation in TV of the war "good against bad" after 11th September or the Star Wars Trilogy. In the project a semi automatic digital film footage structuring and analysis according to religious symbols should be implemented. Joanneum Research will provide its Content Analysis Module, the search infrastructure and already existing annotation tools.

Robust and Adaptive Approaches to Scene and Object Recognition: The goal of this joint project is to investigate new robust and adaptive approaches in the area of object and scene recognition. Object and scene recognition is a necessary requirement for developing truly cognitive systems as well as for the development of advanced and novel multimodal interfaces leading to ambient intelligence. Having a robust object and scene recognition system the following applications will greatly benefit: novel user interfaces which understand human activities, intelligent surveillance, indexing multi-media databases and content analysis of images, autonomous mobile systems and robotics, industrial inspection and robotics, etc. The goal is to develop computer vision based systems that can recognize objects, and in the context of environment perform localization and navigation. The major challenge is to develop systems and methods that can work under realistic unconstrained conditions (i.e., outside the lab). The three partners proposing this project (Center for Machine Perception, Czech Technical University Prague, CMP, Computer Vision Lab, Faculty of Computer and Information Science, University of Ljubljana, CVL, and Institute for Computer Graphics and Vision, Graz University of Technology ICG) have considerable expertise in this area and developed complementary methods and techniques. The goal of the project is to join the efforts and combine the expertise. In particular, we do the following activities:

• Organization of joint workshops and colloquia
• Exchange of PhD. students
• Joint research/joint papers