Whether the program is common to HTC Vive devices

[SSSSamZhu]

Hi,
For the need of robot display and interaction, we hope to repeat the work on VR head-mounted display (through Vive remote operation of the upper body cloud movement of the robot).But our VR headset doesn't seem to be the Oculur you're using.I have noted your relevant discussion. Does this mean I can reproduce your results on Vive at present?
#60
#71
from iCub Shenzhen01
Thanks！

[S-Dafarra]

Hi @SSSSamZhu, thanks for opening the issue.

The pipeline we used in https://www.youtube.com/watch?v=r6bFwUPStOA is still in progress, and it is not thoroughly documented yet. In that case, we are using https://github.com/robotology/human-dynamics-estimation/tree/master to estimate the operator's body posture. Then, we use the Xsens_module of this repo to send references to the walking-controller (see https://github.com/robotology/walking-controllers).

For what concerns the use of the HTC Vive, we exploit this device: https://github.com/ami-iit/yarp-device-openxrheadset. Also in this case, its use is not very documented yet (but I am working on a README https://github.com/ami-iit/yarp-device-openxrheadset/tree/readme).

#71 was a first rough attempt to use the VIVE joysticks to command the robot arms. On the other hand, we are heading in a different direction, and that PR probably will not be merged.

I think that the easiest thing you can try would be to control just the robot neck with the HTC VIVE. The steps more or less would be:

• run the transform server as in https://github.com/robotology/walking-teleoperation/blob/master/app/scripts/transformServer.xml
• run the camera devices and the yarp-device-openxrheadset as in https://github.com/robotology/walking-teleoperation/blob/master/app/scripts/Xprize-VisualRetargeting.xml. Note that I am assuming you have properly configured the headset and you have installed yarp-device-openxrheadset. Regarding the second point, in future releases of the robotology-superbuild it will be automatically installed (see Add OpenXR and OpenVR related software to Teleoperation Component robotology/robotology-superbuild#1115). Also, notice that this configuration file is required https://github.com/robotology/walking-teleoperation/blob/master/app/robots/iCubGenova09/openXRHeadsetParameters.ini. You may need to copy it in a folder with your robot name
• then you can run OculusRetargetingModule --OPENXR::move_icub_using_joypad 0 --GENERAL::useSenseGlove 1. Also in this case you would need a configuration file similar to https://github.com/robotology/walking-teleoperation/blob/master/app/robots/iCubGenova09/oculusConfig.ini
• Run prepareTeleoperation and runTeleoperation on the /oculusRpc RPC port.

Again, apologies for missing some good documentation about the whole pipeline, but it is an active research areas at the moment and many things are changing.

Let me know if you need additional details.

[SSSSamZhu]

Hi @S-Dafarra ,
Thanks for your help.I'm sorry thatI need to reopen this issue until 18 days later. I have recently been learning to use icub's libraries, and this is my first experience with VR.I hope you won't mind my trivial but useful question.

1. Do yarp-device-openxrheadset need to be installed on the Windows and used together with Oculus_module?Or it can connect to a Vive VR device directly on superbuild of Linux？I noticed that openXR-related plug-ins have been added in the latest release of 2022.05. https://github.com/robotology/robotology-superbuild/releases
2. Does that mean I need to fully configure the environment here https://github.com/robotology/walking-teleoperation before adding the module yarp-device-openxrheadset you mentioned?
3. As for the whole system installation, should I install yarp on superbuild and use Windows system?
4. Can the robot perform upper body motion without the walk-Controllers module,that captures the end effectors of the hands and head of the human operator and commands the respective movement?If you want to use the walk-Controllers module, should yarp and icub-main be installed on your Linux system as well?

Thank you for your answer and I'm sorry for taking up your time.

[SSSSamZhu]

Hi @S-Dafarra ,
I'm surprised you guys made such a great robot https://www.youtube.com/watch?v=r6bFwUPStOA .We plan to use iCub more for hospitality and presentations .Therefore, we hope to reproduce your project on iCubShenzhen01 and get more detailed guidance from you. The only devices we have right now are the iCub V2.0 robot itself, and some VR devices like the VIve HTC I mentioned earlier .If necessary, we can purchase the same equipment as in your experiment for reproduction.

Thank you so much for helping us.

[S-Dafarra]

Apologies for the late answer, it has been a busy period. Let me try to go through your questions.

Do yarp-device-openxrheadset need to be installed on the Windows and used together with Oculus_module?Or it can connect to a Vive VR device directly on superbuild of Linux？I noticed that openXR-related plug-ins have been added in the latest release of 2022.05. https://github.com/robotology/robotology-superbuild/releases

Yes, now the yarp-device-openxrheadset is installed by the superbuild when ENABLE_TELEOPERATION is set to ON. This should make the installation much easier. On the other hand, you still need to run yarp-device-openxrheadset on Windows since the use of SteamVR and the VIVE headset on Linux is not very stable. We did try once, but we had some issues with the visualization inside the headset. For for starting I would suggest running `yarp-device-openxrheadseton Windows. TheOculusRetargetingModule`` can run on Linux instead.

• Does that mean I need to fully configure the environment here https://github.com/robotology/walking-teleoperation before adding the module yarp-device-openxrheadset you mentioned?
• As for the whole system installation, should I install yarp on superbuild and use Windows system?

Yes, you would need to setup the Windows system using the superbuild. I would suggest following these instructions: https://github.com/robotology/robotology-superbuild/blob/8e2869cef7ff91a3bbf1fa2151a2e79ceb1deb69/doc/conda-forge.md#source-installation

4. Can the robot perform upper body motion without the walk-Controllers module,that captures the end effectors of the hands and head of the human operator and commands the respective movement?If you want to use the walk-Controllers module, should yarp and icub-main be installed on your Linux system as well?

Yes, to use walking-controllers you would also need yarp and icub-main installed. Once more though, the use of the VIVE controllers to control the robot hands is not fully supported by us yet. It is currently a work in progress.

[SSSSamZhu]

Hi @S-Dafarra ,
Thanks for your nice reply.

In that case, we are using https://github.com/robotology/human-dynamics-estimation/tree/master to estimate the operator's body posture. Then, we use the of this repo to send references to the (see https://github.com/robotology/walking-controllers).Xsens_modulewalking-controller

Based on what you did before, can I understand that I can use Vive HTC to reproduce some other functions of your project? Because I saw the vive HTC used in the video .Maybe this part of the function is to send the iCub robot's eyes back to the Vive HTC? This part of the functionality is also expected !

1. Could you please give more details about using yarp-device-openxrheadset yarp superbuild on Windows10. For example, what command do I need to enter on the terminal?
2. In Linux system, I have read the Readme file. The first step is to import the XML file https://github.com/robotology/walking-teleoperation/blob/master/app/scripts/Xprize-VisualRetargeting.xml you mentioned into yarpmanager and modify the configuration file ini at the same time. But I didn't find these files in superbuild . Does that mean I need to clone walking-teleoperation to the superbuild file directory?
   
   Also, does walk-teleoperation not need to be compiled? Because I saw you write in the Readme

git clone https://github.com/robotology/walking-teleoperation.git
cd walking-controllers
mkdir build && cd build

Finally, thank you for your patience.

[S-Dafarra]

Based on what you did before, can I understand that I can use Vive HTC to reproduce some other functions of your project? Because I saw the vive HTC used in the video .Maybe this part of the function is to send the iCub robot's eyes back to the Vive HTC? This part of the functionality is also expected !

For the control of the gaze, you can check the SRanipalModule: https://github.com/robotology/walking-teleoperation/tree/42004109a4068f584fdaae348f448b8812efecdf/modules/SRanipal_module

Could you please give more details about using yarp-device-openxrheadset yarp superbuild on Windows10. For example, what command do I need to enter on the terminal?

I would suggest referring to https://www.yarp.it/latest/ and https://github.com/robotology/robotology-superbuild

In Linux system, I have read the Readme file. The first step is to import the XML file https://github.com/robotology/walking-teleoperation/blob/master/app/scripts/Xprize-VisualRetargeting.xml you mentioned into yarpmanager and modify the configuration file ini at the same time. But I didn't find these files in superbuild . Does that mean I need to clone walking-teleoperation to the superbuild file directory?

This probably means that the superbuild did not compile correctly.

[SSSSamZhu]

Hi @S-Dafarra ,
I am so sorry that I have agreed to this merger request due to my misoperation. I hope it will not cause too much negative impact on your work. If you need me to delete or withdraw the operation, please tell me the detailed steps. I am not familiar with this process. Please forgive me for my rudeness.
#71

As I mentioned earlier, we want to add solid presentation and interaction capabilities to icub that we already have, since it may need more interactive entertainment in the future. I noticed in your video, published in 2019, https://www.youtube.com/watch?v=yELyMYkCyNE there are used to Ocular fragments inside. I think this will be a stable technical solution and this is the effect of using Joypad + Oculus. Could you tell us the matching version and model you are using?

In order to make this project looks more clean and efficient communication, maybe we can discuss our subsequent transfer to https://github.com/robotology/community/discussions.

Finally, thank you very much for your patience.

[S-Dafarra]

Hi @S-Dafarra , I am so sorry that I have agreed to this merger request due to my misoperation. I hope it will not cause too much negative impact on your work. If you need me to delete or withdraw the operation, please tell me the detailed steps. I am not familiar with this process. Please forgive me for my rudeness. #71

No problem at all, do not worry 😉

As I mentioned earlier, we want to add solid presentation and interaction capabilities to icub that we already have, since it may need more interactive entertainment in the future. I noticed in your video, published in 2019, https://www.youtube.com/watch?v=yELyMYkCyNE there are used to Ocular fragments inside. I think this will be a stable technical solution and this is the effect of using Joypad + Oculus. Could you tell us the matching version and model you are using?

We were using an old Oculus Rift (one of the first). It is very old, and it cannot be bought anymore. In that case, we were using this other device, but since we moved to the use of the VIVE we are not using it anymore (hence it is not maintained).

Notice that also in this way we needed walking-controllers to work on the robot.

In order to avoid using walking-controllers, we can resort to the following guide: https://github.com/robotology/human-dynamics-estimation/blob/f55e70d7adebc66552f4359f62b90fbed8b8d7ae/doc/how-to-run-whole-body-retargeting.md
On the other hand, this solution would require the use of the XSens suit.

As mentioned above, we are still working on a simpler retargeting application using just the controllers, but this is still a work in progress.

[SSSSamZhu]

We were using an old Oculus Rift (one of the first). It is very old, and it cannot be bought anymore.

Do you mean Oculus Rift CV1? There seems to be some stock in the Chinese market. It would be great to have your exact brand.

In that case, we were using this other device, but since we moved to the use of the VIVE we are not using it anymore (hence it is not maintained).

In the superbuild introduction, I saw the oculus module's function description.
https://github.com/robotology/robotology-superbuild/blob/master/doc/cmake-options.md#oculus
Is this corresponding to the function mentioned above? I'm not sure what the specific function of yarp-device-ovrHeadset is.

Notice that also in this way we needed walking-controllers to work on the robot.

In order to avoid using walking-controllers, we can resort to the following guide: https://github.com/robotology/human-
dynamics-estimation/blob/f55e70d7adebc66552f4359f62b90fbed8b8d7ae/doc/how-to-run-whole-body-retargeting.md On the other hand, this solution would require the use of the XSens suit.

All in all, does this mean that using Joypad + Oculus alone is not feasible? https://github.com/robotology/walking-teleoperation/wiki I saw it on a wiki, and it seems a little old.

but since we moved to the use of the VIVE we are not using it anymore (hence it is not maintained).

However, I remember you said that the use of the VIVE controllers to control the robot hands is not fully supported yet. You mean you use VIVE as an expression recognition module? Use XSens for full-body repositioning, not VIVE or Oculus.

[S-Dafarra]

but since we moved to the use of the VIVE we are not using it anymore (hence it is not maintained).

However, I remember you said that the use of the VIVE controllers to control the robot hands is not fully supported yet. You mean you use VIVE as an expression recognition module? Use XSens for full-body repositioning, not VIVE or Oculus.

Back in time, we did use the Oculus joysticks to control the robot (though walking-controllers), but we do not use that headset anymore, nor we support anymore the code that was working back in time.

Anyhow, I guess we are getting a bit off-topic.

If your aim is to use a VIVE headset to control the robot, right now the easiest that you can try is to work with the head and gaze retargeting only, as mentioned in #98 (comment). This requires you to install the robotology-superbuild as mentioned above. Once we are there, we can discuss how to proceed 😉

[SSSSamZhu]

If your aim is to use a VIVE headset to control the robot, right now the easiest that you can try is to work with the head and gaze retargeting only, as mentioned in #98 (comment).

So sorry, maybe my first question was misleading. What we want to do is reproduce an interactive technology solution on ICub, like you did before, using Joypad + Oculus to control the movement of the upper body of the robot. The first solution I came up with was to use VIVE, because we happen to have a VIVE in our lab.

Thanks.

[S-Dafarra]

If your aim is to use a VIVE headset to control the robot, right now the easiest that you can try is to work with the head and gaze retargeting only, as mentioned in #98 (comment).

So sorry, maybe my first question was misleading. What we want to do is reproduce an interactive technology solution on ICub, like you did before, using Joypad + Oculus to control the movement of the upper body of the robot. The first solution I came up with was to use VIVE, because we happen to have a VIVE in our lab.

Thanks.

That's clear.

The systems you have seen in the videos have been very complex to set up and require a lot of configuration and testing. In all those cases, we were controlling all the robot joints through the walking-controller.

Right now I don't have any how-to guide, nor any steps to follow. We do not have anything ready at the moment to control just the upper body using the VIVE or any other joystick, but it should be possible. In any case, this will require running a series of different modules and many things can go wrong in the middle.

Hence, as a first step, I was suggesting to start controlling only the neck, so that at least we can start testing the system.

I understand this is not exactly what you want, but I don't know how to help you otherwise.

[SSSSamZhu]

Moring, @S-Dafarra

We do not have anything ready at the moment to control just the upper body using the VIVE or any other joystick, but it should be possible. In any case, this will require running a series of different modules and many things can go wrong in the middle.

I also noticed that you used tracker for relocation, is this scheme still supported and maintained? It seems like a great idea.
https://github.com/ami-iit/yarp-openvr-trackers

[S-Dafarra]

Moring, @S-Dafarra

We do not have anything ready at the moment to control just the upper body using the VIVE or any other joystick, but it should be possible. In any case, this will require running a series of different modules and many things can go wrong in the middle.

I also noticed that you used tracker for relocation, is this scheme still supported and maintained? It seems like a great idea. https://github.com/ami-iit/yarp-openvr-trackers

Yes, we do use trackers. They are also supported in https://github.com/ami-iit/yarp-device-openxrheadset. With the same pipeline is possible to get the position and orientation of joysticks. Controlling the robot accordingly is another story

[SSSSamZhu]

Hi, @S-Dafarra
I have learned how to connect YARP nodes to enable Windows hosts to communicate with Linux. But I still have some problems, and I hope to get your advice.

1. I have created a new profile with the robot name in the corresponding folder, but I don't think my actions have worked. I hope you can get more detailed suggestions.

log in Windows.

(robsub) C:\Users\samgmzhu>yarprun --server /icub-virtualizer
[INFO] |yarp.os.Port|/icub-virtualizer| Port /icub-virtualizer active at tcp://192.168.1.102:10021/
[INFO] Yarprun successfully started on port:  /icub-virtualizer
[INFO] |yarp.os.impl.PortCoreInputUnit|/icub-virtualizer| Receiving input from /tmp/port/2 to /icub-virtualizer using tcp
STARTED: server=/icub-virtualizer alias=:icub-virtualizeryarpdev--from:openXRHeadsetParameters.ini0 cmd=yarpdev --from openXRHeadsetParameters.ini  pid=6992
[INFO] |yarp.os.impl.PortCoreInputUnit|/icub-virtualizer| Removing input from /tmp/port/2 to /icub-virtualizer
[ERROR] |yarp.os.Property| cannot read from openXRHeadsetParameters.ini
[INFO] |yarp.dev.Drivers| Welcome to yarpdev, a program to create YARP devices
[INFO] |yarp.dev.Drivers| To see the devices available, try:
[INFO] |yarp.dev.Drivers|    yarpdev --list
[INFO] |yarp.dev.Drivers| To create a device whose name you know, call yarpdev like this:
[INFO] |yarp.dev.Drivers|    yarpdev --device DEVICENAME --OPTION VALUE ...
[INFO] |yarp.dev.Drivers|    For example:
[INFO] |yarp.dev.Drivers|    yarpdev --device fakeFrameGrabber --width 32 --height 16 --name /grabber
[INFO] |yarp.dev.Drivers| You can always move options to a configuration file:
[INFO] |yarp.dev.Drivers|    yarpdev [--device DEVICENAME] --from CONFIG_FILENAME
[ERROR] |yarp.dev.Drivers| Unable to find --device option in file openXRHeadsetParameters.ini. Closing.
CLEANUP :icub-virtualizeryarpdev--from:openXRHeadsetParameters.ini0 (6992)
[INFO] |yarp.os.impl.PortCoreInputUnit|/icub-virtualizer| Receiving input from /tmp/port/2 to /icub-virtualizer using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/icub-virtualizer| Removing input from /tmp/port/2 to /icub-virtualizer
[INFO] |yarp.os.impl.PortCoreInputUnit|/icub-virtualizer| Receiving input from /tmp/port/2 to /icub-virtualizer using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/icub-virtualizer| Removing input from /tmp/port/2 to /icub-virtualizer
[INFO] |yarp.os.impl.PortCoreInputUnit|/icub-virtualizer| Receiving input from /tmp/port/2 to /icub-virtualizer using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/icub-virtualizer| Removing input from /tmp/port/2 to /icub-virtualizer

2. I was wondering how do I change the host here? Now I have changed the original icub-console to my host name issacliang-LC3. May I ask how to modify icub-console-gui? He couldn't start for that reason.
   
   Looking forward to your reply.