Mobile Visual Computing

Mobile devices have grown to be a major computing platform that impacts the everyday life. The latest generation of devices offers great graphics and computing capabilities together with a rich set of input sensors from cameras, GPS and orientation sensors. This course will introduce concepts, methodologies and tools to develop interactive, 3D graphics applications on mobile devices that connect to the real world and location of the user. After completing the course successfully, you will be able to develop your own mobile, interactive, 3D application.

Slides

• 01 Introduction
• 02 Development Environment
• 03 OpenGL ES 2.0
• 04 Sensors
• 05 Location
• 06 Cameras
• 07 Features & Pose
• 08 Image-based Tracking
• 09 Vuforia
• 10 Visual Search
• 11 Kooaba - VisualSearch

Materials

• Development environment
• Device drivers
• Example collection 1
• Examples 1.1
• Examples 2
• Examples 2.1 - implementations from lab 2
• Examples 3
• Examples 3.1 - alternative particle system + improved vuforia tracking app including an .obj loader.
• Examples 4 - Kooaba test app

Assignments

1. Extend the per vertex lighting shaders from the GL2Main example to perform per pixel lighting calculations.

   Required input to the fragment shader will be: position, normal, color, light direction. These need to be in the camera coordinate system.

   

2. Use sensors input to build one of the following to projects:
   • A marble balance board/game. Similar to a game like this Balance Board

     Use the sensor input to change the orientation of a board or the gravity in the game and move a sphere according to a simple physics simulation.

   

   • A location-based viewer similar to Wikitude

     Use a config file with a fixed set of objects with lat/long coordinates and show them relative to the users location.

3. Use Image-Based Tracking to create a real balance board game

   Using an image target glued to a board, move a virtual marble or object on that real board. You will need to combine the acceleration sensor information with the tracking information to get gravity with respect to the board.

Links

Android development

• USB drivers http://developer.android.com/sdk/oem-usb.html
• NDK information http://developer.android.com/sdk/ndk/index.html
• Making menus http://developer.android.com/guide/topics/ui/menus.html
• Vuforia Tracking SDK https://developer.qualcomm.com/mobile-development/mobile-technologies/augmented-reality
• Vuforia Dev Guide https://ar.qualcomm.at/qdevnet/developer_guide
• Camera calibration tool http://graphics.cs.msu.ru/en/science/research/calibration/cpp
• Kooaba API https://github.com/kooaba/kooaba-api/tree/beta/query_api_v2

OpenGL ES 2.0

• Android tutorial http://developer.android.com/resources/tutorials/opengl/opengl-es20.html
• Examples http://code.google.com/p/opengles-book-samples/wiki/Instructions
• Reference card http://www.khronos.org/opengles/sdk/2.0/docs/reference_cards/OpenGL-ES-2_0-Reference-card.pdf