Haptics is a tactile feedback technology which takes advantage of the sense of touch by applying forces, vibrations, or motions to the user. This mechanical stimulation can be used to assist in the creation of virtual objects in a computer simulation, to control such virtual objects, and to enhance the remote control of machines and devices.

Haptics is widely used in simulators and other immersive virtual reality environments, but also for controlling remote robotic tools in the automobile manufacturing industry, or even for performing remote surgery. I used two different kinds of haptic devices for this project, the Sensable Phantom Omni and the Phantom Premium 1.5 (picture on the left).

Augmented reality (AR) is about augmenting the real world environment (captured via a camera for example), with virtual information like 3D models or 2D images. Like haptics, the primary objective of augmented reality is to improve people’s senses and skills to perform actions in the real world using additional computer-generated information that are not available in the first place.

My research focused on making those two technologies seamlessly work together, and help the user to have a better perception of his augmented environment.

The first step was to implement a way to haptically feel augmented objects. Once I completed this task, I figured that it was really hard to grab augmented objects using the haptic device due to the lack of visual and haptic cues. This is the reason why I started to work on strengthening the notion of depth in the scene as well as finding ways to make the relative positions of objects to one another easier to perceive.

Attachment

I worked on shadow casting (top-left image), occlusion of augmented objects by real objects (top-right image) and reflections of augmented objects in real mirrors (bottom-left image). Haptics is already used for remote surgery, and there is active research on how to use augmented reality in this field as well. Surgeons need a very accurate perception of their working environment, the visual cues presented here would surely become very handy!

Finally, I decided to integrate a physics engine to my framework to enhance the interaction between augmented objects and the real world. I used Newton Game Dynamics for this purpose and make several samples to illustrate the result (bottom-right image). The video below presents the results of my research in more details.

Chapters

Project presentation

Technologies involved

Project requirements

First results

Object manipulation

Occlusion problems

Augmented shadows

Augmented reflections

Augmented physics

Download the video

English Francais