supervisor.step() returns too early

[KelvinYang0320]

supervisor.step() returns too early
I want the robotic arm to rotate its joints -0.05, 0, or 0.05 in each step, but the robot hasn't done its action, and directly returns current values from its position sensors. Then the supervisor sends next action to the robot.
However, the experiment still works and seems good.

To make the robotic arm rotate its joints precisely, I let the robot send a specific message which denotes that it is still moving, and let the supervisor send robot a special action number, say -1, which denotes that supervisor is not sending an action to the robot but asking whether the robot stops moving.

The above solution can solve the problem, but I am looking for some more efficient ways.
Thanks!

[tsampazk]

Hey @KelvinYang0320,

This issue sounds peculiar. Can you share controllers or point me to code that reproduces this behaviour?

[KelvinYang0320]

My deepbots-panda issue2 repo

• supervisor.step()
• 
• my solution:
  • supervisorController
  • robotController

Still lots of dirty code, sorry.
And thank you for your reply!

[tsampazk]

Still lots of dirty code, sorry.

No worries, i will check it out and get back to you!

[tsampazk]

Hello @KelvinYang0320, i took a look at the code and the example and - correct me if i am wrong - i think that a robot action takes more than one step to complete. As a consequence the supervisor has to wait for the previous action to complete before getting the new observations and sending back the new action.

Webots motors controlled by setPosition run an internal PID controller to reach the position you asked for. This can take multiple steps. The solution you implemented seems perfectly fine to me. I think you are forced to implement such a scheme (or anything similar really) to wait for the action to complete.

I would suggest that you use the RobotSupervisor scheme, you can see an example here. Note that the linked example uses the latest dev release of deepbots which you can find here.

With the robot-supervisor scheme we use the robot node as a supervisor, without needing any emitter-receiver communication, and all needed functions are implemented in one controller. Moreover, the example uses the new reset procedure that simplifies reset and makes it far-far faster. Switching from the emitter-receiver to the robot-supervisor scheme is more a matter of copy-pasting and deleting some stuff, no big deal really (i have done it myself in the example i linked). The new scheme will make your code far simpler and more efficient.

I think that by switching to the new scheme your issue will be solved by default.

Feel free to ask any other questions and i'll be happy to help.

[KelvinYang0320]

@tsampazk Thank you very much for helping me!
I have fixed the problem that you mentioned, thanks for pointing out the problem.

As a side note, one problem i spotted is that when the arm hits the floor and tries to move through it, the target position is never reached and the simulation hangs, never going through the handshake process, as seen in the screenshot below.

I have trained the DDPG agent from scratch without using the pretrained model from my IKPY-based-RL-ENV, and it seems to converge after more than 10000 episodes which takes more than one day. It seems to work well!
For more details, please refer to my deepworlds fork.

[tsampazk]

Happy to help @KelvinYang0320. Whenever you feel ready, open a pull request so we can include the problem in the deepworlds examples, which will make a great addition to the repository!

[KelvinYang0320]

@tsampazk I have done refactor but found that I forgot to save my models during training.
Now I have to train DDPG agent again to ensure everything goes well.
It might take about two days, sorry:cry:

[tsampazk]

No worries, we are not including trained models in the deepworlds repository anyway, at least not yet. The code that is used for training is enough!

[KelvinYang0320]

Ok, thank you @tsampazk .
I will open a pull request for this code that is used for training.
If there are new issues after two days, I will fix them and open another new pull request.