ISMAR 2018
IEEEIEEE computer societyIEEE vgtc

Sponsors

Platinum Apple
Silver MozillaIntelDaqriPTCAmazon
Bronze FacebookUmajinDisney Research
SME EnvisageAR
Academic TUMETHZ

Caterina Battisti, Stefano Messelodi, and Fabio Poiesi. Seamless bare-hand interaction in mixed reality. In Adjunct Proceedings of the IEEE and ACM International Symposium for Mixed and Augmented Reality 2018 (To appear). 2018.
[BibTeX▼]

Abstract

Unrealistic Mixed Reality (MR) experiences can be caused by unprocessed occlusions between real and augmented objects during interactions. Depth knowledge is indeed key to achieve a seamless visualisation when a bare hand interacts with an augmented object. This can be addressed by blending real-time 3D finger tracking information with the visualisation of the hand in MR. We propose an approach that automatically localises the hand in RGB images and associates the respective depth; estimated with an auxiliary infrared stereo camera used for hand tracking; to each RGB hand pixel. Because misalignments between the outline of the hand and its depth may occur due to tracking errors; we use the distance transform algorithm to densely associate depth values to hand pixels. In this way hand and augmented object depths can be compared; and the object can be rendered accordingly. We evaluate our approach by using an MR setup composed of a smartphone and a Leap Motion mounted on it. We analyse several hand configurations and measure the erroneously classified pixels when occlusions occur.