input: fix hand tracking with no interaction profile and/or grip pose by ImSapphire · Pull Request #261 · Supreeeme/xrizer

ImSapphire · 2025-12-28T08:07:56Z

Fixes hand tracking on WiVRn (WiVRn/WiVRn@1e2823c). Closes #165

Hand tracking is still pretty scuffed, we don't support the 'augmented' inputs from hand tracking that SteamVR supports (/input/[finger]_pinch trigger input, /input/index_point trigger input, /input/system left side button input, plus rebinds for basic trigger and grip pull).

I don't like the pose_is_synthesised hack, but this is what OC did and I'm not good enough at math to figure out how to fix the wrist pose when we're using a fake grip.

Tested in VRChat and Resonite.

Supreeeme · 2026-01-18T17:18:22Z

src/input/devices.rs

+                    Hand::Left => Vec3::from_array([0.09, -0.03, -0.09]),
+                    Hand::Right => Vec3::from_array([-0.09, -0.03, -0.09]),


Where are these values coming from?

they're pulled from OpenComposite

These values derived from OC, are pulled from SteamVR's SteamLink for hands, so they should be spot on.

This could use a comment or be a const or something.

Supreeeme · 2026-01-18T17:18:48Z

src/input/devices.rs

+                offset *= Mat4::from_euler(
+                    glam::EulerRot::XYZ,
+                    0.0,
+                    if data.hand == Hand::Left {
+                        -FRAC_PI_4
+                    } else {
+                        FRAC_PI_4
+                    },
+                    if data.hand == Hand::Left {
+                        -FRAC_PI_2
+                    } else {
+                        FRAC_PI_2
+                    },
+                );


What's up with this transform?

it's from OC. without this transform, when the fingertips are facing forward and palms are facing down, in-game the palms will be facing inward and fingers are pointed 45deg down

Most likely also derived from SteamLink ditto the above comment by me.

Can this be a comment?

src/input/devices.rs

Supreeeme · 2026-01-18T17:23:29Z

src/input/devices.rs

+        self.interaction_profile.or_else(|| {
+            if matches!(self.device_type, TrackedDeviceType::Controller { .. })
+                && self
+                    .get_hand_skeleton(xr_data, session_data.tracking_space())
+                    .is_some()
+            {
+                Some(&VRLinkHand)
+            } else {
+                None
+            }
+        })


Is hand tracking only something that can be dynamically switched to, and does the runtime change the interaction profiles when hand tracking is enabled? If the profile doesn't change this will be stuck with the old profile and never work.

at least on Quest through WiVRn the interaction profile will change to NULL when switching to hand tracking.

i don’t think it’s critical that we return the VRLinkHand profile in all cases where hand tracking is active, i just did this so hand tracking works with no interaction profile

src/input/devices.rs

Supreeeme · 2026-01-18T17:25:24Z

src/input/devices.rs

+                TrackedDevice::new(
+                    TrackedDeviceType::Controller(ControllerData {
+                        hand: Hand::Left,
+                        hand_tracker: create_hand_tracker(Hand::Left),
+                        skeleton_cache: Mutex::new(Box::new(None)),
+                        pose_is_synthesised: false.into(),
+                    }),
+                    None,
+                    None,
+                ),
+                TrackedDevice::new(
+                    TrackedDeviceType::Controller(ControllerData {
+                        hand: Hand::Right,
+                        hand_tracker: create_hand_tracker(Hand::Right),
+                        skeleton_cache: Mutex::new(Box::new(None)),
+                        pose_is_synthesised: false.into(),
+                    }),
+                    None,
+                    None,
+                ),


Why are controllers being automatically created at startup?

without this hand tracking will not work if controllers aren’t connected at least once

Why? We should not report controllers if they don't actually exist. If I'm understanding the VRLinkHand profile properly, we just need to add bindings for the hand_interaction profile. In fact, couldn't we just do that instead of checking for the hand skeleton?

If I'm understanding the VRLinkHand profile properly, we just need to add bindings for the hand_interaction profile. In fact, couldn't we just do that instead of checking for the hand skeleton?

It's possible to have XR_EXT_hand_tracking data available while XR_EXT_hand_interaction is not supported, I only added the VRLinkHand profile for the properties

Fair enough, but that should be a comment on the profile, that's not really immediately obvious. In any case though, we should not create controllers when they don't exist, that's just confusing. There should be a way to check if hand tracking is available and try to pull a pose from it without this. XrSystemHandTrackingPropertiesEXT::supportsHandTracking might be useful.

In any case though, we should not create controllers when they don't exist, that's just confusing

I do agree, but there's lots of functions that rely on the controllers existing, like IsTrackedDeviceConnected and Get*TrackedDeviceProperty, so I'm pretty sure we have to do this for hand tracking to work without having gotten a valid interaction profile for the hands, right?

It would make more sense to take an approach similar to the info set used to determine if there are controllers before reporting them to the game - each frame, you could attempt to get a pose from the hand tracker, and when it succeeds you can add a controller. I don't have a real example on this but I'm pretty sure lying about controllers being connected can actually mess some games up. (should've written a test, rip)

src/input.rs

Supreeeme · 2026-01-18T17:36:33Z

src/input/devices.rs

+    pub skeleton_cache: Mutex<Box<Option<xr::HandJointLocations>>>,
+    pub pose_is_synthesised: AtomicBool,


You could instead have an enum like:

enum SkeletonPose { Real(Box<Option<xr::HandJointLocations>>), Synthesized(Box<Option<xr::HandJointLocations>>) }

and then just have a field like:

skeleton_cache: Mutex<SkeletonPose>

Also, I don't think all of the fields in this struct need to be pub.

synthesised pose doesn't really have anything to do with the openxr hand skeleton though, so i'm not sure it would make sense to wrap this in an enum here

Supreeeme · 2026-02-19T00:13:51Z

src/input/devices.rs

+                    Hand::Left => Vec3::from_array([0.09, -0.03, -0.09]),
+                    Hand::Right => Vec3::from_array([-0.09, -0.03, -0.09]),


This could use a comment or be a const or something.

Supreeeme · 2026-02-19T00:14:08Z

src/input/devices.rs

+                offset *= Mat4::from_euler(
+                    glam::EulerRot::XYZ,
+                    0.0,
+                    if data.hand == Hand::Left {
+                        -FRAC_PI_4
+                    } else {
+                        FRAC_PI_4
+                    },
+                    if data.hand == Hand::Left {
+                        -FRAC_PI_2
+                    } else {
+                        FRAC_PI_2
+                    },
+                );


Can this be a comment?

Supreeeme · 2026-02-19T00:24:03Z

src/input/devices.rs

+    let pose_from_hand_tracking = || {
+        if let Some((joints, velocities)) =
+            controller.get_hand_skeleton(xr_data, session_data.tracking_space())
+        {


This could be a method on ControllerData.

Supreeeme · 2026-02-19T00:30:40Z

src/input/devices.rs

+    pub fn get_hand_skeleton(
+        &self,
+        xr_data: &OpenXrData<impl crate::openxr_data::Compositor>,
+        base: &xr::Space,
+    ) -> Option<(xr::HandJointLocations, xr::HandJointVelocities)> {
+        let TrackedDeviceType::Controller(data) = self.get_type() else {
+            return None;
+        };


This should just be a method on ControllerData.

Supreeeme · 2026-02-19T00:42:02Z

src/input/devices.rs

+                TrackedDevice::new(
+                    TrackedDeviceType::Controller(ControllerData {
+                        hand: Hand::Left,
+                        hand_tracker: create_hand_tracker(Hand::Left),
+                        skeleton_cache: Mutex::new(Box::new(None)),
+                        pose_is_synthesised: false.into(),
+                    }),
+                    None,
+                    None,
+                ),
+                TrackedDevice::new(
+                    TrackedDeviceType::Controller(ControllerData {
+                        hand: Hand::Right,
+                        hand_tracker: create_hand_tracker(Hand::Right),
+                        skeleton_cache: Mutex::new(Box::new(None)),
+                        pose_is_synthesised: false.into(),
+                    }),
+                    None,
+                    None,
+                ),


Why? We should not report controllers if they don't actually exist. If I'm understanding the VRLinkHand profile properly, we just need to add bindings for the hand_interaction profile. In fact, couldn't we just do that instead of checking for the hand skeleton?

Supreeeme · 2026-02-19T00:55:07Z

src/input/skeletal.rs

        hand: Hand,
-        transforms: &mut [vr::VRBoneTransform_t],
-    ) {
+    ) -> Option<[vr::VRBoneTransform_t; Count as usize]> {


Why is this changed?

…ilable

ImSapphire marked this pull request as draft December 28, 2025 08:08

ImSapphire force-pushed the hand-tracking-fixes branch from 560abea to 562665a Compare December 28, 2025 13:04

ImSapphire changed the title ~~Hand tracking fixes~~ input: fix hand tracking with no interaction profile or grip pose Dec 29, 2025

ImSapphire changed the title ~~input: fix hand tracking with no interaction profile or grip pose~~ input: fix hand tracking with no interaction profile and/or grip pose Dec 29, 2025

ImSapphire force-pushed the hand-tracking-fixes branch 2 times, most recently from 6f35c5b to eb3646e Compare January 7, 2026 07:30

ImSapphire marked this pull request as ready for review January 7, 2026 07:31

ImSapphire force-pushed the hand-tracking-fixes branch 2 times, most recently from 21bb680 to b836a4d Compare January 12, 2026 01:15

Supreeeme reviewed Jan 18, 2026

View reviewed changes

ImSapphire force-pushed the hand-tracking-fixes branch from b836a4d to ad4b5c1 Compare January 19, 2026 14:00

ImSapphire force-pushed the hand-tracking-fixes branch from 7339c29 to b05146c Compare February 7, 2026 17:31

ImSapphire requested a review from Supreeeme February 7, 2026 17:32

ImSapphire mentioned this pull request Feb 17, 2026

Skeletal Summary 2: Electric Boogaloo #313

Merged

Supreeeme reviewed Feb 19, 2026

View reviewed changes

ImSapphire force-pushed the hand-tracking-fixes branch from b05146c to a9ccf05 Compare February 19, 2026 02:11

ImSapphire added 8 commits February 18, 2026 20:15

input: move hand tracker into device

5303254

input: cache openxr hand skeleton

2c47649

input: cache openvr bone transforms

573fea0

input/skeletal: simplify estimated skeleton lerp logic

cf94db7

input: fallback to deriving poses from hand tracking if grip is unava…

54d6953

…ilable

input/devices: initialise hand devices even with no interaction profile

f0c5082

input: report VRLinkHand profile when only hand tracking is active

1f12fd9

input: store skeletal tracking level per-hand

dd94b89

ImSapphire force-pushed the hand-tracking-fixes branch from a9ccf05 to dd94b89 Compare February 19, 2026 02:18

		Hand::Left => Vec3::from_array([0.09, -0.03, -0.09]),
		Hand::Right => Vec3::from_array([-0.09, -0.03, -0.09]),

		pub skeleton_cache: Mutex<Box<Option<xr::HandJointLocations>>>,
		pub pose_is_synthesised: AtomicBool,

Uh oh!

Conversation

ImSapphire commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ImSapphire Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ImSapphire Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ImSapphire Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ImSapphire commented Dec 28, 2025 •

edited

Loading

ImSapphire Jan 19, 2026 •

edited

Loading

ImSapphire Feb 19, 2026 •

edited

Loading

ImSapphire Jan 21, 2026 •

edited

Loading