The 1.20 release
1.20 has been released on the German world 1 – quite successfully I would say as there haven’t been so much problems, although quite a few bugs of different severity haven’t been found during the beta testing and we had / have to fix a lot of bugs.
I have to say that I’ve been a bit disappointed by the beta testers regarding cheating: I would expect beta players to try to cheat in the game wherever possible. For example: It’s been possible for attackers to set the starting position on the flag points just by modifying the requests that are sent to the server. I only got aware of that when some players were affected by another bug that occurred when they tried to join more than one fort fight. That happened only after the 1.20 release on world 1.
That’s been a quite bad bug that shouldn’t have been released, yet it got through. Over the months of development I just always “moved” the responsibility of checking illegal positions to other code parts that I had not been working on at that moment – i.e. when getting the values from the players and I wrote it into the database, I expected to filter illegal positions on the fortbattle server. When I implemented the fort battle server, I expected that the values in the database would be correct. Due to the different times when I wrote these parts, I never implemented that checking anywhere. Nice, isn’t it? Yet, such glitches are normally found by evil cheaters quite quickly and they can be used to their advantage. Just imagine a group of attackers spawning on the flag just when the battle starts. Wouldn’t be so nice for the defenders…
So I would like to remind our beta testers: Try to break our rules on purpose! Don’t avoid unfair play but enforce it! You won’t get banned for that (unlike as on the public game worlds). Some of you did already do some testing like that, but we need more of that (like making fort member towns ghost towns (sorry for not having fixed that issue yet on the beta server, but we are not community managers and we had quite busy weeks…)).
Anyway, I wanted to write here now about what all went wrong during the release and how it happened.
Thursday, 7:45 am
I turned up at work at that time because it was the plan to make the release in the morning time so that the first forts would be finished around 4 pm / 5 pm. We thus expected that then the first battles would happen at Friday around 4 pm / 5 pm. I started working on several minor bugs, like fixing a glitch in the fort management and a problem with that particular piece of javascript when executed with Internet Explorer browsers.
9:30 am
About at 9:30 am Anthraxx pushed the button to make the release. About 9:31 am Eiswiesel ran into our office and yelled “Build time for Forts is 30 seconds”. That’s how it’s been on the beta worlds. Uops. We immediately fixed that wrong value, but when we did that, 3 forts had already been built by players. 8 other followed because we forgot to change the values in the task queues that had already been inserted into the database. These players had of course a minor advantage over other players since they were the first to have forts. Some players asked to undo that unfair development but that is not a simple request. We just decided that these lucky ones were just lucky winners in the fort build lottery. Their prize was the immediate attack of other players.
And that was how it came that the first fort battle was to be fought at 9:37 am on Friday instead of 4pm as we planned to. But that was not really to our disadvantage however.
10 am
A few minutes later the first bugs reports dropped in. It was my change at that morning that on the one hand stabilized the IE browser but also introduced a bug that made it impossible to invite other towns. But other bugs got visible too – IE 6 users were unable to build forts because the name field of the founded fort did not turn up on their screens. A little bit later other players reported that a county had 4 forts while the one below had 3. Oups. Another oups when players found a cozy little location of another fort:
Quite a nice location I’d say, but also quite wrong. Later it also turned out that the county with 4 forts had not 4 but 3 forts and there was one fort missing on the map that is shown in the minimap. The other county with 2 forts in the minimap had however 3 forts and the 3rd one doesn’t show up in the minimap.
We still have no clue what to do with Fort Los Angelus, neither do we know what is causing the minimap to behave that strange.
Thursday afternoon
Ok, so the first battle was to be fought on Friday morning. I checked if the policy port (1028) was reachable and after a few tweaks it worked. I should have checked the other port as well then (1578? Too lazy to look it up now), but I didn’t. Instead I spent some time on implementing battle logs so that we can later replay the battles. Don’t ask for a playback function, I still don’t know if the recorded actions are complete since I lack a playback function myself. I did also fix some other errors, just like my 3 colleagues in the office.
Around the late afternoon a bug turned up that always existed but never became visible: Due to the fort battles, for the first time in west it made sense for players to gather at a single location. When about 150 players had been luring at the first fort that was declared war on, the player list functionality broke due to too many players at one spot. That was also eventually fixed at afternoon.
When I left the office I had an intensive 11 hour day behind me and I had a bad feeling about the 200 players who were about to fight the first fort battle on a public server. Due to the mishap in the morning time, I had to turn up around 8 am on Friday as well (I normally come in around 10 am).
Friday, 9 am
I still had not checked the second port when the first battle was about to be fought. I just expected that if the policy port was running that the other port would be forwarded as well. Well, checking the port was not that simple because I had no application to test that in a valid way anyway. Had no time to write such a tool.
And this is what went wrong when the battle started at 9:37:
We instantly noticed that we couldn’t connect with flash to the game. That was bad. Looking into the server logs we saw that the server was bombarded with policy file requests that got answered promptly. But no connect request came through. After a couple of WTFs all over the place, hastily hacking on console windows and checking connectivity we realized that of the 3 ports that the server opened, 2 were forwarded correctly and that was the policy port – and the maintenance port that shouldn’t be available to the Internet at all. The game port that is used for the flash communication was simply not addressed by anyone.
After we figured that out we were able to fix it in time and about 10 minutes later the connection worked to my great relief. We could view the battle taking place and the final report to be sent to the spectators.
Later on Friday we noticed that players reported that they were playing on the wrong locations when they joined multiple battles. Also we noticed that the final report was not successfully sent to the users (but that was not noticed by the players due to the lack of information that there should be a report shown). Turned out that the reports became larger than 65kb which was the upper limit for reports at that point. I fixed that bug along with the bug of illegal player positions.
Over the day several battles were fought and the battle server stabilized. We made a few minor changes then – and now I am curious how stable the java server is going to be over the weekend. That is quite a stress test that is currently running. But I am quite optimistic that since the server has survived now for quite some time that it will run stable for some time. Yet I am not sure if we fixed all memory leak issues and so I don’t know if the server is not going down within the next 48 hours. I can’t predict anything there yet, but we’ll see.
I hope that the next week is going to be a bit more calm so that we can fix a couple of other problems and do a bit planning for upcoming features. I would expect that after this huge version release that 1.21 is not going to introduce such large features but will only provide minor features that we couldn’t get into 1.20 yet – but we’ll have to see what we will work on for that release.
We will also see next week when we are going to unroll 1.20 on other worlds and also other languages. This could take the whole week maybe, but I am not sure on that. I hope that the players of other worlds are patient – it shouldn’t take so long anymore.
PS: I am getting constantly requests to increase the number of beta players on the beta world. I am aware that a lot of players want to get into the public beta but you have to understand that I can neither change the player limit nor can I make the decision myself. It has however been decided that the player limit is going to be raised in the future. I can’t really tell you more on that because I simply don’t know more about it myself.
I am Eike Decker, also as known as Zet. I am working at InnoGames as a developer since January and I have been assigned to The West at the end of february. The first project that I’ve been working on for The West was the tutorial mode for the game which was introduced in 1.19. Since then I’ve been working on the fort battles non-stop. I wrote the flash client as well as the HTML frontend for the battles and also the parts on the game server that are related to the game play. If something doesn’t work as expect in these departments I am probably the person to blame. For anything else I am less likely to know what’s going on – I have only a limited overview on other features than the fortbattles. Simply because I have not done much else apart from that.
