Real-Time Hand Pose Tracking using 6-Axis IMUs

Abstract

We introduce a real-time system for tracking hand pose using 6- axis inertial measurement units (IMUs) without requiring magnetometers or external sensors. Accurate hand pose tracking with only 6-axis IMUs is known to be fundamentally challenging due to the absence of a shared heading reference, leading to severe drift and inter-sensor misalignment. To overcome these limitations, we propose a hybrid method that combines a learning-based pose estimation approach followed by a late-stage Extended Kalman Filter (EKF). The learning-based model estimates noisy yet reasonable hand poses and is trained with drift-insensitive features like gravity vectors and wrist-relative gyroscope signals. On the other hand the EKF can appropriately filter the noise from pose estimates leading to robust tracking. Evaluated on a 12-hour dataset spanning 23 interaction tasks across 10 participants, our system improves joint angle accuracy by 40% over an EKF-only baseline and by 18% over a learning-only approach, achieving a mean joint error below 10°. The resulting framework enables real-time hand tracking invariant to magnetic perturbations, occlusion, or lighting changes, and is well suited for robotics, human–robot interaction (HRI), and human-computer interaction (HCI) applications.