panda_robotiq controller not working as expected

[akuramshin]

Problem: sending only a change in gripper position results in the entire robot changing its pose.

Steps to reproduce:

1. Run python mani_skill/examples/demo_manual_control.py -e StackCube-v1 -c pd_ee_delta_pose --enable-sapien-viewer
   This should open up the StackCube environment with the panda_robotiq agent. The demo_manual_control script allows the user to manually control the robot with every action command being a single environment step.

2. Send a command to only open/close the gripper by pressing the f/g keys on your keyboard (you might have to first press on the matplotlib window for it to register keyboard commands).

Since we are using the pd_ee_delta_pose controller and we are only sending a gripper action, the expected outcome is for all the joints to remain static except for the gripper. However, the entire robot moves along with the gripper.

In contrast, changing [this line (https://github.com/montrealrobotics/ManiSkill/blob/main/mani_skill/envs/tasks/tabletop/stack_cube.py#L24) to use the "panda" embodiment instead will show you the expected outcome.

[akuramshin]

Currently following up in the ManiSkill repo in a thread here: haosulab#636 (comment)