ISMAR 2018
IEEEIEEE computer societyIEEE vgtc


Platinum Apple
Silver MozillaIntelDaqriPTCAmazon
Bronze FacebookUmajinDisney Research
SME EnvisageAR
Academic TUMETHZ

Caterina Battisti, Stefano Messelodi, and Fabio Poiesi. Seamless bare-hand interaction in mixed reality. In Adjunct Proceedings of the IEEE and ACM International Symposium for Mixed and Augmented Reality 2018 (To appear). 2018.


Unrealistic Mixed Reality (MR) experiences can be caused by unprocessed occlusions between real and augmented objects during interactions. Depth knowledge is indeed key to achieve a seamless visualisation when a bare hand interacts with an augmented object. This can be addressed by blending real-time 3D finger tracking information with the visualisation of the hand in MR. We propose an approach that automatically localises the hand in RGB images and associates the respective depth; estimated with an auxiliary infrared stereo camera used for hand tracking; to each RGB hand pixel. Because misalignments between the outline of the hand and its depth may occur due to tracking errors; we use the distance transform algorithm to densely associate depth values to hand pixels. In this way hand and augmented object depths can be compared; and the object can be rendered accordingly. We evaluate our approach by using an MR setup composed of a smartphone and a Leap Motion mounted on it. We analyse several hand configurations and measure the erroneously classified pixels when occlusions occur.