Hello! We have doubts about the performance limitations in the Notes section of the announcement board (" Codes that cause great delay in their allotted timestep process (for example, each second simulated equals 5 seconds in real time) will be terminated after 15 minutes real time."). First of all: what hardware is going to be used to run the simulation? as this will greatly impact performance, and we cannot measure if our code complies with restriction otherwise. Even then, it’s likely that our team will get disqualified for this reason. We didn’t know this restriction was in place and, correct us if we are wrong, we don’t believe it has been mentioned before. If we knew about this before, we would have made radically different decisions in the design of our code to take it into account. It is almost certainly impossible for us to improve performance so drastically in the few days we have left, it would involve an almost complete redesign of something we have been working on for six months. Is there anything else we can do about this?
While our rule did not specify it is real-time, nor Webots time either. .
Therefore, for all fairness, we need to allow common sense to dictate instead of allowing exploitation of loopholes. When it said 8 minutes, we’ll allow some “reasonable” extra time so that no individual team would manipulate the runtime setting arbitrarily. Honestly, we have never had team exceeding more than 9 minutes. Allowing 15minutes real-time is already considered to be over-kill to complete a supposedly 8-min run.
First of all, thank you for replying!
With “our rule” are you referring to the note: “Codes that cause great delay in their allotted time step process (for example, each second simulated equals 5 seconds in real time) will be terminated after 15 minutes real time.” or to the 8 minute of simulated time limit? The first one clearly refers to real time and the second one is controlled automatically by the supervisor, which works in simulated time.
Our code is very complex. In my computer, for example, it usually runs at 0.15x real time. But this also depends on the hardware. It is possible that it will take more than 15 mins. We did the math based on the time that the videos of the first round were cut off, and it runs more or less at the same rate. Therefore I conclude that our hardware is similar.
In 2021 we had a code of similar complexity (under team name Titan), that also ran very slowly, so it is likely that there is a precedent for a run taking more than 15 minutes real-time.
We perfectly understand that for physical limitations you can’t wait for hours for a run to finish, but a really complex code running at, let’s say, 0.3x real-time (~24min for an 8 min run) in moderate hardware (that we assume because of the time comparison), to us at least, does not seem unreasonable. After much optimization we were able to get it running close to that.
Is there any chance that the limit might be updated?
We are not trying to abuse the system or any loopholes. We just did something really complex with the objective of innovating and bringing something new to the table, and we had little time to think about performance.
We completely understand, though, if you can not do it. We will keep in mind performance for future competitions.
I just learned that it took ~20 minutes realtime to run 1.5 min webots time. It means it will take 106+ minutes just to finish one 8-min run. That means it will take up 5+ hours to just run your controller. Hope you understand this is far from being reasonable.
Yes, we understand, that is why we have made great improvements in performance. If our code takes hours to run we are completely okay with you stopping it. We just thought a 15 min limit was a bit too strict. We didn’t know it wasn’t a hard limit. Also. if it took 20 minutes, it means it performs considerably differently in your hardware, we will take this into account for future tests.