Improvement for the upcoming competition, while the metal is hot.

[omichel]

Once the ICRA 2023 competition is over (that is this Friday), we plan to organize another very similar competition to be held during the IROS 2023 conference. Before we proceed with improving the rules and organization of this new competition, we would like to gather the feedback from the current participants. to help us improve the competition and make it more interesting from a scientific point of view, but also from an entertainment point of view as we believe that both are not incompatible.

Feel free to post on this discussion thread any suggestion that you believe could contribute to improve the competition.

Thank you!

[MJavadZallaghi]

First of all, at least for future competitions please give awards also to the second and third place teams.😅✌🏼

Is the future competition again based on humanoid robots?

[omichel]

Thank you for your suggestion. Yes, the future competition will be very similar to this one, involving wrestling humanoid robots.

[AKRNANDULA]

It was a good competition. and looking forward to participate again in your next event.
Kindly, consider the latest team name and participant name/s while preparing the certificate for top 32 teams in leader board.

The only suggestion was to give some more time (may be 1 or 2 days) to update the controller program after selecting the top 32 or before the start of finals.

[JustLuckyRobotics]

i agree with this.

[JustLuckyRobotics]

Brilliant comp. was fun and helped hone a lot of skills. the only change i think would be beneficial to everyone would be the ability to choose to fight a specific team in a singular fight. Id have this as well as the current system of a commit triggering a leaderboard climb.

I think one of the most challenging aspects of such a competition is having a training partner, IE a robot to fight in local sims. A potential solution could be to have an "effective" enemy bot which uses a precompiled code (not visible) anyone and everyone can fight at anytime in local sims along side the tutorial code which is visible (perhaps our winning code could be compiled somehow and made ready for next comp) Alternatively, a solution could be if a team is able to request a fight with a specific team on the leaderboard but the outcome has no effect on the leaderboard state. IE a singular fight limited to 5minutes (to prevent busted code hogging resources) and also give these actions a lower priority if someone has made a normal code commit and intends the climb the leaderboard.

This would allow for a lot of innovative strategies and counter strategies a lot faster and more effectively, with the current system if you win against a particularly clever and innovative controller, you have to either sandbag your code so you can fall down the board to be able to test out ideas and counters, or wait until they choose to update their code. With my intended change it would allow people to test their code against anyone, while still having to climb the leaderboard as now.

[MJavadZallaghi]

Great competition, and congratulations to the winner.

I think you can make it possible that before a new match in the final phase of the competition, a team could change its controller until a defined deadline.

I'm going to mention an example. Does FIFA force the national teams in the world cup to continue the games with the initial players and strategy during all the final phases? Obviously no. So I think after any stage of the games, a team must be able to make some changes to improve its performance against the next competitor as we did it during the 32-team selection stage frequently.

Thank you and best regards.

[JustLuckyRobotics]

i strongly disagree, the intention of this competition is to build a automated controller good against all situations the programmer can anticipate, as should any good robot controller be. All this would achieve is people using their upcoming competitors replays to tailor their controller against a specific team each round, entirely against the theme of the competition.

This competition is also international, any time window given will be advantageous to some more than others, the only way to solve this would be needlessly time consuming and complicated for the organiser, or would need 2 days between games which is also not compatible with being run during ICRA.

The current format is best suited in my opinion, the only change should maybe be additional time, maybe a day or 2, between the end of the leaderboard "qualifiers" and the actual competition.

[MJavadZallaghi]

Okay. Make sense.
But if the nature of the competition is to develop an autonomous controller, I believe any interval for controller change after the 32-team selection must not be allowed strictly, as it's not fair.

[amirhnourian]

I agree, especially since the end date for pushing the controller was so strict. I also did not know the date it would be great if we can change the controller before each match and also be more fun and engaging. In this sense, those empty hours before each game ( 24 hours ) would mean something!

[christopherpeers]

In response to some of the ideas in this thread:

"a team must be able to make some changes to improve its performance against the next competitor as we did it during the 32-team selection stage frequently."

This change would undoubtedly increase the impact of luck as it would encourage behaviour as seen on the leaderboard where people would either run several different simple scripts in the hope of winning some matches or simply watch the replay of their next opponent and attempt to cheese them out of a win.
If one was capable enough of making a robust controller for their robot that could sense and react in the correct manner then this change would be completely pointless, unless it was for the tuning of some parameters in which case the suggestion from another post about having a time frame after the leaderboard closes to adjust code would suffice and not this. Also, the last point concerning ICRA.

"But if the nature of the competition is to develop an autonomous controller, I believe any interval for controller change after the 32-team selection must not be allowed strictly, as it's not fair."

I don't believe there was any debate concerning whether the competition was asking for development of a autonomous controller, however, this is a large contradiction compared to being able to continuously adjust code during the tournament stages.

"I also did not know the date"

The competition lasted for several months, joining later in the competition will of course incur a handicap compared to those who started early. The first thing any participant should do is read the rules thoroughly.

"In this sense, those empty hours before each game ( 24 hours ) would mean something!"

As the competition was held during ICRA, I don't think its fair that people who were not attending would have much more time to adjust their code as opposed to those who were.

[MJavadZallaghi]

Also it would be great if the matches be in 3 win from 5 match template, as the impact of luck can be reduced more.

[omichel]

Thank you all for sharing your ideas for improving this competition. We are considering all of them and will soon make some decisions regarding the organization of the new edition.

[amirhnourian]

I think the best part of this match was templates, I learned Webots, motion planning, inverse kinematic implementation, and many more advanced programming and robotic topics. It would be best if we have more APIs or templates in the future so we could implement and generate more complex motion behavior and strategies.

I also would recommend introducing and talking about these winners' controllers so others can advance their robotic skills and knowledge even further.

It would be fun if you have a winner in some categories such as vision, motion and etc.

Finally, it was a great experience and a chance to further improve my robotic knowledge through hands-on experience. @omichel thanks for the opportunity!

[AndrejOrsula]

Thank you once again for organizing the competition! I have some points for a possible discussion.

Wait until controllers are initialized before starting Webots

The start-up time of the controllers is currently not deterministic. One of the controllers might start running before the other, and it is not possible to predict.

A synchronization mechanism for the start-up of controllers could eliminate this stochasticity. Here are some suggestions for discussion that could improve the situation (although they might be challenging to implement):

• Using docker-compose, the container of Webots service could wait until the participant & opponent controllers (services) are running via depends_on options. This will still not account for the start-up time of the specific controller code, but at least the process wouldn't be dependent on the time it takes to load each image from the SSD/HDD and spin up the container.
• Have an internal mechanism within Webots Supervisor that waits until each Robot controller is initialized (with some timeout, of course). This could be triggered automatically upon chaining up the parent implementation of the Robot constructor in subclasses that implement it (or something similar).

Partitioning of CPU cores

As the current README.md states, the evaluation is performed on i7-6700K CPU with 4 physical cores. With hyperthreading currently enabled, this results in a total of 8 logical CPU cores (2 logical cores for every physical core).

In the current setup:

• Webots is run with docker run --cpuset-cpus=0-1 ...
• The controller of participant (RED) is run with docker run --cpuset-cpus=2-4 ...
• The controller of opponent (BLUE) is run with docker run --cpuset-cpus=5-7 ...

This means that:

• Logical CPU cores 0 and 1 are assigned to Webots
• Logical CPU cores 2, 3 and 4 are assigned to RED robot controller
• Logical CPU cores 5, 6 and 7 are assigned to BLUE robot controller

I believe that lscpu --all --extended or cat /proc/cpuinfo on the machine would indicate something like this:

• CPU 0 (physical) => CPU 0 and CPU 4 (logical)
• CPU 1 (physical) => CPU 1 and CPU 5 (logical)
• CPU 2 (physical) => CPU 2 and CPU 6 (logical)
• CPU 3 (physical) => CPU 3 and CPU 7 (logical)

I am not sure if this is the most equal configuration. First of all, the L1/L2 cache of each physical CPU would be shared between the controllers (or Webots), which could favour one or the other participant. Similarly, the priority of which thread is run on the physical CPU could also be affected by the scheduler in one way or another. It might have a negligible effect that is difficult to quantify, but it seems like a simple change to make it appear more equal. For example:

• Webots could be assigned CPU 0 (physical) via docker run --cpuset-cpus=0,4 ...
• The controller of RED robot could run on CPU 1 (physical) + share CPU 3 (physical) via docker run --cpuset-cpus=1,5,3 ...
• The controller of BLUE robot could run on CPU 2 (physical) + share CPU 3 (physical) via docker run --cpuset-cpus=2,6,7 ...

This is just one of the possible configurations. An alternative would be to disable hyperthreading on the machine; however, that would prevent any CPU-based concurrency for the controllers (unless the machine gets an upgrade).

Limits of GPU VRAM usage

In my opinion, specifying the limit of GPU VRAM usage could also be enforced. Currently, nothing prevents one of the participants from quickly allocating all of the available VRAM in order to prevent the opponent from running anything on the GPU. Similarly, it makes it difficult for participants to know how much VRAM they can actually use without running into issues or causing problems for other participants.

RAM usage is limited to 6 GB via docker run --memory=6g ...; however, I do not believe it is possible to limit VRAM in the same way on the utilised GPU. Therefore, regular run-time checks of the VRAM usage might be the only option. If that is too challenging to implement, then I think having at least some guidelines would be appreciated (e.g. your controller must not exceed X.Y GB of VRAM).

Height detection of KO

Each robot is currently considered KO if the HEAD_SLOT frame is below 0.9 m for more than 10 s.

wrestling/controllers/referee/referee.py

Line 97 in 46a97b6

if position[2] < 0.9 or position[2] > 1.5 or abs(position[0]) > 1.5 or abs(position[1]) > 1.5:

I have a feeling that the minimum limit of 0.9 m might be too much for certain controller schemes and strategies. Slightly lowering this limit (or checking at a different frame) could allow participants to explore more complex strategies that do not rely on the robot always standing straight (without risking self-induced KO). I think it would also be very interesting to see if some participants managed to design a controller that is able to recover from falling down the ring and climb back up, which is probably impossible to achieve with the current height/time constraints.

On the other hand, I understand how some of the resulting controllers could be considered as exploits by participants. You have spent more time tuning this number, so you know the best. 😄

[christopherpeers]

On the comments regarding the height detection of KO, I always thought this should be more akin to real life wrestling where the participant with the higher frame height is considered to be winning over the one who is lower.

The change I would make would be to make only the participant with the lowest frame height below the threshold begin their countdown as this would encourage grappling motions as they would potentially cause a self-induced KO with the current ruleset.

[JustLuckyRobotics]

I get your point with the starting at the same time syncing up, and ensuring both docker images are ready before starting. however i dont agree that webots should wait until both code bases are ready to start acting. If participant A has a heavy NN implementation and takes a while to config and start, participant B should not be punished for this.

If B decides to forgo any setup config of any NN's to maximise its reaction time, that should be a respected strategy and in some ways might involve innovative design decisions to get around not having NN classifiers like yolo.

On the flip side it also eliminates the need to optimise things, if two teams fight, both with NN's, both take time to setup, both lag. Now its down to who's code is faster to configure, faster to react. This would give another avenue for innovation and design, maybe a lighter weight NN implantation which is good enough and boot faster is better than the advantages a heavier NN might have.

If the rest of your points are implemented as i believe they should be, i think what i have discussed above should be left alone as is. The competition is designed to be run asynchronously which should allow for differences in such things. Think of it like this, if this was real life comp the robots should be stood with no code running, at the start of the clock the competitors press "go" on their code. In a real life situation these differences in reaction time would be present.

Either idea works, i just feel as if such a change would shoehorn people into a certain route of development.