Sarker, AnikKou, ZiyiRistani, ErgysGuan, LiNiehues, Taylor2026-04-072026-04-072026-03-16https://hdl.handle.net/10919/142746We introduce a real-time system for tracking hand pose using 6- axis inertial measurement units (IMUs) without requiring magnetometers or external sensors. Accurate hand pose tracking with only 6-axis IMUs is known to be fundamentally challenging due to the absence of a shared heading reference, leading to severe drift and inter-sensor misalignment. To overcome these limitations, we propose a hybrid method that combines a learning-based pose estimation approach followed by a late-stage Extended Kalman Filter (EKF). The learning-based model estimates noisy yet reasonable hand poses and is trained with drift-insensitive features like gravity vectors and wrist-relative gyroscope signals. On the other hand the EKF can appropriately filter the noise from pose estimates leading to robust tracking. Evaluated on a 12-hour dataset spanning 23 interaction tasks across 10 participants, our system improves joint angle accuracy by 40% over an EKF-only baseline and by 18% over a learning-only approach, achieving a mean joint error below 10°. The resulting framework enables real-time hand tracking invariant to magnetic perturbations, occlusion, or lighting changes, and is well suited for robotics, human–robot interaction (HRI), and human-computer interaction (HCI) applications.application/pdfenCreative Commons Attribution 4.0 InternationalArticle - Refereed2026-04-01The author(s)https://doi.org/10.1145/3757279.3785628