Cardboard SDK sensor API bypass dispatchSensorEvent

[lrq3000]

So it seems that recently, an update to the Google Cardboard SDK introduced its own sensor API with automatic drift correction and stuff like that. For VR apps, it's now the advised way to access the sensors, instead of the android.hardware.Sensor API.

The bug was first spotted with Cmoar Cinema Pro (not the Free, it doesn't use the Cardboard API) but it can easily be reproduced using VRTV Video Player Free: this app provides an option "Orientation Provider" with 4 choices:

1. Google Cardboard SDK (default): this produces the aforementioned bug.
2. Manual touch input: use the touch screen to move the screen instead of the sensors. It also bypasses our module but that's normal.
3. Improved Orientation Sensor: no problem, the module works.
4. Calibrated Gyroscope: no problem also.

It seems this Cardboard sensor API is quite new so I don't have much more info.

[lrq3000]

What do you think about it Kjos? I'm not really used to hacking into sourcecode using Xposed. If I find the documentation, do you think you can hack some quick code? (I can try it out and enhance it, I just don't know how to hook properly).

[Kjos]

My thought on this, is that the new Cardboard SDK uses the NDK to get the Gyro readings.
See: https://source.android.com/devices/halref/sensors_8h_source.html and other source files. I'm not very familiar with using the NDK, but I'm pretty sure accessing the Gyro through the NDK omits and Java calls. Xposed only reads the Java calls, so there's no way to access those.
A quick fix would be to look into the Cardboard jar and see what functions are exposed. If any of them accesses the Gyro data, you can hook into that. This used to be the case, as I did that earlier, but not sure with the new SDK, chances are slim tbh.
If you're unfamiliar with hooking, I can recommend reading the tutorial here: http://forum.xda-developers.com/showthread.php?t=2709324 If you're hooking in Android it helps to have the source code available (and take any changes between Android versions in to mind).
I'll take a look at the Cardboard sdk and report what I find.

[lrq3000]

Ok thank you very much for the infos and for looking at this issue, I am
also reading the Cardboard sourcecode to try to find somewhere to hook.

2015-09-17 13:20 GMT+02:00 Kjos notifications@github.com:

My thought on this, is that the new Cardboard SDK uses the NDK to get the
Gyro readings.
See: https://source.android.com/devices/halref/sensors_8h_source.html and
other source files. I'm not very familiar with using the NDK, but I'm
pretty sure accessing the Gyro through the NDK omits and Java calls. Xposed
only reads the Java calls, so there's no way to access those.
A quick fix would be to look into the Cardboard jar and see what functions
are exposed. If any of them accesses the Gyro data, you can hook into that.
This used to be the case, as I did that earlier, but not sure with the new
SDK, chances are slim tbh.
If you're unfamiliar with hooking, I can recommend reading the tutorial
here: http://forum.xda-developers.com/showthread.php?t=2709324 If you're
hooking in Android it helps to have the source code available (and take any
changes between Android versions in to mind).
I'll take a look at the Cardboard sdk and report what I find.

—
Reply to this email directly or view it on GitHub
#4 (comment)
.

[Kjos]

I think this function needs hooking:
private native void nativeGetCurrentEyeParams(long l, com.google.vrtoolkit.cardboard.HeadTransform headTransform, com.google.vrtoolkit.cardboard.Eye eye, com.google.vrtoolkit.cardboard.Eye eye1, com.google.vrtoolkit.cardboard.Eye eye2, com.google.vrtoolkit.cardboard.Eye eye3, com.google.vrtoolkit.cardboard.Eye eye4);

in

package com.google.vrtoolkit.cardboard;

class: CardboardViewNativeImpl

Looking at the source, it looks like Cardboard is also applying some lowpass filtering. However it's no use to mess with that, as I'm pretty sure the native impl doesn't use it (it's in compiled code).

Add to first thing:
Maybe better to hook into
public void getCurrentEyeParams(com.google.vrtoolkit.cardboard.HeadTransform head, com.google.vrtoolkit.cardboard.Eye leftEye, com.google.vrtoolkit.cardboard.Eye rightEye, com.google.vrtoolkit.cardboard.Eye monocular, com.google.vrtoolkit.cardboard.Eye leftEyeNoDistortionCorrection, com.google.vrtoolkit.cardboard.Eye rightEyeNoDistortionCorrection) { /* compiled code */ }

package:
com.google.vrtoolkit.cardboard.CardboardViewApi

Although then you would hook into native ndk as well original Java code (so if not take into account, it could hook twice for the same data with Java Cardboard SDK apps). On the other hand that would be more future proof, as the native function is private and might change.

[Kjos]

Unfortunately as it's all compiled code, you can't hook directly to the gyro, so you need to filter the results instead.

[lrq3000]

Yes it was to be expected that there would be no direct access to the gyroscope since the authors of the Google Cardboard project wants to provide fusionned/virtual sensors to all Cardboard apps to avoid drifts and miscalibration issues.

Well, I think that hooking to the results of the virtual sensors will be as good as we can get. I'm reading the documentation and I'll try to do that :)

[lrq3000]

Just a question: do you have any access to the source code of the Cardboard SDK? I can't find the source code for the latest version...

[lrq3000]

Found another possible hook:

https://github.com/rsanchezsaez/cardboard-java/blob/master/library/src/main/java/com/google/vrtoolkit/cardboard/sensors/HeadTracker.java#L58

What do you think about it?

[Kjos]

I used the latest Cardboard jar from the Github and Intellij automagically decompiles it except for the native code. You can use any other decompiler probably. It's not obfuscated code either, all the names are intact.
You can hook in the SensorEvent but it's not used with the NDK code. You need to hook in the results, as posted above.

[lrq3000]

Ah ok I see, then I'm currently trying to hook that, I'll keep you updated :)

[Kjos]

Here's the class decompiled: http://pastebin.com/1YkXXXca

[lrq3000]

Thank's :)

[Kjos]

http://pastebin.com/xpbabKid This is HeadTransform. I'm not sure if it's better to use reflection or anything, but in theory you could include the Cardboard.jar or only needed source files such as HeadTransform.java to get the headview float array, which holds the rotation. Apply a filter to that, and it will work nicely.

[Kjos]

Ah, you could just use reflection like it is done here as well:
https://github.com/Kjos/XposedJitteryGyroFix/blob/master/GyroscopeNoiseFilter/src/net/kajos/gyronoisefilter/GyroscopeNoiseFilter.java#L171
I forgot it was done like that. Just use a similar way to get the float[] array of headview.