Soccer Simulation Demo 2021 - Issues and questions

[Felipe]

Hello, everyone.

This topic was created to concentrate all questions and suggestions you might have about the Soccer Simulation Demo of February 2021.

As we mentioned in the call, the idea of this demo competition was to serve as a trial for the official simulation competition to be held in June 2021. To help us improve, we welcome your suggestions and observations about the whole process.

If you want to report a bug or issue observed during the matches, please try to give as much detail as possible to help us identify the problem: identify your team, the exact match and time stamp you are referring to, and explain what you think the problem is.

The links for results and videos of all games can be found in the following topic: Soccer Simulation Demo 2021 - Results

On behalf of the RCJ Soccer TC and RCJ Soccer Simulation OC, I thank you all in advance!
Felipe.

P.S. 1: Please, note that we will not re-run matches. As stated before, we are having this demo competition to learn what we need to improve in order to have a good competition in June. We understand that this might be frustrating, but running the competition is taking much more time and effort than we anticipated and we simply don’t have the capacity to replay matches at this point.

P.S. 2: If you are looking for the simulator or competition rules, check out the post below: Soccer Simulation Demo Competition 2021

[thomastseng0380]

Hello @Felipe,

We are Team Reset from Taiwan (in Group 1 for the group phase) that participated in the demo competition and were wondering how the programs were assigned to the robots. When we looked at all four of the recorded matches, we realized that there seems to be one set of code on two different robots (one set of code seems to be unused), and this only happened when we were facing the blue goal (it was normal when we were trying to score on the yellow goal). We did double-check our code and we didn’t see how this problem could be possible since at no point was the offensive code ever copied into the defensive program. We also tested the sets of code that we submitted, with one set of code assigned to one robot for both sides (blue and yellow) locally on Webots, and we didn’t have the same problem.

In “Screenshot 1” of this folder, you can see that when we were attacking the yellow goal, our defensive robot was normal (stayed near the penalty area). However, in that exact same match, after halftime when the robots switched sides and we were attacking the blue goal, as you can see in “Screenshot 2” of this folder, our defensive robot was operating the offensive program.

So we just wanted to ask if there’s a chance that there’s an assignment error when initializing the competition, and if it is possible to have our robots reassigned code such that each robot gets a unique set of code instead of two robots running the same set of code and one set of code being left unused.

Thank you!

Sincerely, Team Reset

[david]

Hello @thomastseng0380,

what we ended up doing is editing everybody’s code to make the imports not collide. Changes I made to your code look like this:

What this does is it moves everything that you imported to team_libraries which gets mounted into the Docker container that runs the simulation as /app/controllers/team_033_libraries, your robot code gets mounted from robot1.py to /app/controllers/rcj_soccer_player_y1/rcj_soccer_player_y1.py or /app/controllers/rcj_soccer_player_b1/rcj_soccer_player_b1.py depending on what side you are playing. If spot anything that could have changed your bots’ behavior let me know, we’ll try and fix it for the future; this demo competition will have remain on the current code base though.

I have just run your code against the demo robots (as Team 999) and whichever side I put them on the robots did all move and there were no import-errors (or errors of any kind) on the Webots-console (other than that the container has no audio and my laptop has low GPU power:

For the future we’re hoping to have an upload format that requires fewer or no changes and maybe a test-upload so that you can see your bots run under “competition conditions”.

Best Regards,
David

[Zvono]

Hello rcj TC & simulation OC,

I’m writing on behalf of GJH_team, as to the latest matches.

While watching our games today, we couldn’t help but notice, that one of our robots was exhibiting unusual behavior and acting exactly like the default code.

So we decided to test this on our local simulation (whether there is a mistake on our side) by extracting the same .zip file, and when right controllers were selected on both sides, all worked perfectly, and we were unable to recreate the behavior exhibited by the robot in the matches (unless we selected default code for one of the robots, in which case we had 100% the same results). This behaviour was directly responsible for multiple goals we received (our robots rely on each one fulfilling its role), and basically meant we were playing 2v3 for one half.

After further review, this seemed to happen on every match we faced blue, on the Y2 robot exclusively. Again, our code was the same for all robots (with slight include changes), and 100% identical between Y2 & Y3. Our Y3 robot worked fine.

Now we may be wrong and will be doing some more testing, but we deem it that there was a mistake on our side (alas still very possible).
Could you please check if the correct codes were selected?

Thank you for the brilliant competition,

let’s hope we can get to the bottom of this,

GJH_team

[Leo]

Hello,

There is a high possibility that all the Y2 robots are running the demo code.

We have randomly chosen one sample from each group in the Group Phase, one sample from the Round of 16, and one sample from the Quarters Final.

Below is the list of samples:

Group1 - 2 vs 33
Group2 - 17 vs 46
Group3 - 19 vs 36
Group4 - 1 vs 9
Group5 - 11 vs 32
Group6 - 26 vs 5
Group7 - 39 vs 27
Group8 - 8 vs 28
Round of 16 - 3 vs 10
Quarters Final - 14 vs 47

We circled the robot in the following videos that has a similar behaviour to the demo robot.

Videos: https://drive.google.com/drive/folders/1U8ct_VFgJf9OcKtF4b-7nOp2JHMR8Ewm?usp=sharing

Leo and Jerry
TFA 312

[Felipe]

Hello, everyone.

It seems that the behavior reported by @thomastseng0380, @Leo and @Zvono points to the same issue. Thanks to all of you for reporting the problem, and special thanks to @Leo and Jerry for their excellent report with the videos!

After investigating the problem, we identified a bug in the script we used to automate the simulation process. Indeed, robot 2 from the Yellow team was pointing to the wrong code. In this particular case, because of the way we changed team sides, all teams played with Yellow in one half of each match, which means that there was some balance at the end.

We are going to invite all teams to a meeting in the next few weeks, in which we plan to explain how we ran the competition and collect feedback and suggestions from all of you.

I must point out that we cannot re-run matches now, as mentioned before. Identifying problems and solving them before the actual competition in June is one of the goals of this Demo Competition, and you are contributing a lot!

Thanks again for your collaboration and understanding!

Felipe.
On behalf of the RCJ Soccer TC and RCJ Soccer Simulation OC.

[rlaszlo]

Hi,

We found an interesting moment when the ball apparently left the field, and a few seconds later one of the teams got a goal scored. The screenshot shows the match and the time when this happened.

This is rather kind of funny, and probably earns just a low priority as a very rare issue. Maybe RCJSoccerReferee.check_progress() could check if the ball’s coordinates are out of the field, and if so, just mark it as lack of progress for the ball (before the ball reach check_goal() 's GOAL_X_LIMIT thresholds). To make the referee code absolute fool safe, the same check could run for all robots as well, because if any of the robots would somehow get out of the field, it would not have the chance to get back unless for lack of progress.

Best regards,
Robert

[david]

Thanks @rlaszlo for reminding us of this, I have created an issue (https://github.com/RoboCupJuniorTC/rcj-soccer-sim/issues/77) on the GitHub repository, most likely solution is to reset the ball as you suggested but maybe also to increase the tick rate so the ball cannot pass through the wall in one tick and thus avoid collision detection.
Best Regards,
David

[david]

@thomastseng0380 you were affected by the bug mentioned below, my apologies for looking in the wrong place first when investigating.
Best Regards,
David

[Zvono]

Hello all,

Firstly, we understand that this is a demo competition, and that there is only so much time the TC/OC has allocated for this demo.

However, that does not nullify the time and effort spent by teams preparing for this competition. Since the rules/game are likely to change substantially for the non-demo, it’d be nice to get to see our robots playing real matches in which all robots are functional.

@Felipe

which means that there was some balance at the end

Considering how differently teams approached strategy, it affected all teams in a different way. A team that statically assigned a goalkeeper role to the Y2 robot would be in a significant disadvantage. Strategies (like the one that we built) that dynamically switch robots are essentially punished, since they rely on the fact that every robot is doing its job, otherwise certain positions are not covered, and all robots are severely handicapped in their decision making by assuming that all robots are functional, while in reality one is not.

While its is our opinion that the finals should be re-matched (the results in the group phase were pretty significant in margin, this would also be a lower workload for the TC/OC). We understand that the TC/OC does not wish to rematch games, and respect that.

Once again, thank you for bringing robocup to us, the students, despite the crisis,

GJH_team

[adrianmatejov]

@Zvono We will release all of the submissions so you can easily download them and run your code against any team on your machine. Stay tuned!

[Felipe]

Dear teams,

During the meeting on Sunday (7th of March 2021) we will use the tool below to collect questions. You can already enter your questions and vote for the ones you find interesting!

Slido

We hope to talk to you on Sunday!
Felipe.

[thomastseng0380]

Hello @Felipe,

Will the meeting be recorded either in Word form or video form? Because of time zone issues, we may not be able to actively participate during the meeting time.

Thank you!

Sincerely, Thomas

[Felipe]

Hi, @thomastseng0380.

We will have the same meeting in two different moments, as explained in the e-mail invitation. You can participate in either one. In any case, we will try to record the meeting to make it available to those that are not able to attend.

Regards,
Felipe.

[david]

If anyone is looking for the meeting link for the Q&A: https://meet.google.com/zqt-ircp-oan

Regards,
David

[david]

Hello @Zvono,

the code @adrianmatejov spoke of is available on GitHub (https://github.com/RoboCupJuniorTC/rcj-soccer-sim-tournament-utils) now if you want to try to re-run any matches. We will try and improve documentation (Questions welcome, Pull Requests even more welcome) for you to be able to run the code, our main focus will be on making improvements for the June tournament however so issues specific to running old code again you will have to overcome on your own. If you fix any bugs on the way or document the process so that other teams can do the same we will of course happily merge and publish that work.

Best Regards,
David

[adrianmatejov]

Hello everyone!

During this week we managed to fix a couple of issues in the simulator.

1. If the robot or the ball leaves the field, lack of progress will be called immediately. (Related PR: https://github.com/RoboCupJuniorTC/rcj-soccer-sim/pull/82)
2. In case of lack of progress, picking nearest neutral spot has been made non-deterministic. We now pick 3 nearest spots and choose randomly from one of them where the ball is going to be placed. (Related PR: https://github.com/RoboCupJuniorTC/rcj-soccer-sim/pull/84)

One more bug I found during testing is that the ball was always being put to the center neutral spot after lack of progress. That is now fixed within point 2.

If you find more bugs, please report them, so we can fix it as soon as possible.

[adrianmatejov]

Hello again!

We have changed the structure of robot controller folders. From now on, there are just two folders, one for blue team and the second one for yellow team.
It is now easier to import shared code between robots and you don’t have to use “path” hacks to do so. Check the docs here https://robocupjuniortc.github.io/rcj-soccer-sim/how_to/

If you have further questions, do not hesitate to ask.

Adrian

[thomastseng0380]

Hello @adrianmatejov,

After downloading the updated code (with the new rules and fixes) from Github, we noticed quite a few problems that appear in the simulation.

Firstly, the robots all seem to run the same program. When we changed both of the motor speeds from left_speed and right_speed to 0 in robot3.py, all of the robots, not just robot3, stopped moving. Additionally, when we edited the code in robot1.py and robot2.py, the changes do not seem to have any effect on the robot as robot1 and robot2 continue to run robot3’s code. We are assuming that this problem could have something to do with the new structure of the robot controller folders.

Secondly, since the simulation runs pretty slowly at regular speed, when we do testing, we usually press the Run as fast as possible button. However, not only do all of the robots run at a much faster speed compared to before the demo competition when the original code was being used (which may not really be a problem), the match time does not speed up as it should be doing. Previously, if the simulation was run at double its original speed, the match time would also change at double its original speed. However, now, when the simulation is run at 7 times its original speed, the match time only runs at its original speed.

Thirdly, when the robots do not move at all, the ball moves automatically. Previously, this issue wasn’t as significant, but now, it ball does move quite a bit (having some velocity) without having any contact with the robots at the beginning of the match.

Lastly, we would like to readdress the issue of the robots slowing down. After having the opportunity to run the simulation with our robots playing against other robots (teams from the demo competition), we realized that the issue of the robots slowing down did not only occur for our robots. However, the slowing down of robots varied in degree in different matches (by chance) and on different teams, and worsens as the match lengthens (usually doesn’t occur in the first few minutes). This does affect the result of the simulation since the ball, quite strangely, does not appear to be slowing down. On a side note, when recording a video using the Record Button in Webots, even though the robots still slow down, it is not obvious in the recorded videos. So we do not think, as previously mentioned, that this issue is intrinsic to our code, and would like to know if other teams are experiencing the same problem and if the problem has something to do with the system itself.

Videos regarding these issues could be found with the link below:

https://drive.google.com/drive/folders/1SWcilGJeMwMDShFlEagDM_RcQP_0RoRg?usp=sharing

Thank you!

Thomas

[adrianmatejov]

@thomastseng0380 Thanks for reporting the issues.

1. The first problem (robots running the same code) is my fail , the fix is on its way to be merged soon (https://github.com/RoboCupJuniorTC/rcj-soccer-sim/pull/88)

2. Created the pull request for this to make supervisor synchronous (note we made robots run asynchronously, so if one robot enters infinite loop, it won’t stuck other robots) – https://github.com/RoboCupJuniorTC/rcj-soccer-sim/pull/89

3. Yeah, when robots are not moving, the ball moves just a bit. I’ll play with it and try to fix it if possible, but at first it looks like a Webots issue.

4. Regarding robots slowing down. Not sure what could be causing this. It might be that the code is inefficient, storing more and more data, which could cause the computation to be slower and slower. We will investigate this. Please, let us know if you happen to find the reason why robots slow down during the game

Thanks again!

Adrian