1.20 has been released on the German world 1 – quite successfully I would say as there haven’t been so much problems, although quite a few bugs of different severity haven’t been found during the beta testing and we had / have to fix a lot of bugs.
I have to say that I’ve been a bit disappointed by the beta testers regarding cheating: I would expect beta players to try to cheat in the game wherever possible. For example: It’s been possible for attackers to set the starting position on the flag points just by modifying the requests that are sent to the server. I only got aware of that when some players were affected by another bug that occurred when they tried to join more than one fort fight. That happened only after the 1.20 release on world 1.
That’s been a quite bad bug that shouldn’t have been released, yet it got through. Over the months of development I just always “moved” the responsibility of checking illegal positions to other code parts that I had not been working on at that moment – i.e. when getting the values from the players and I wrote it into the database, I expected to filter illegal positions on the fortbattle server. When I implemented the fort battle server, I expected that the values in the database would be correct. Due to the different times when I wrote these parts, I never implemented that checking anywhere. Nice, isn’t it? Yet, such glitches are normally found by evil cheaters quite quickly and they can be used to their advantage. Just imagine a group of attackers spawning on the flag just when the battle starts. Wouldn’t be so nice for the defenders…
So I would like to remind our beta testers: Try to break our rules on purpose! Don’t avoid unfair play but enforce it! You won’t get banned for that (unlike as on the public game worlds). Some of you did already do some testing like that, but we need more of that (like making fort member towns ghost towns (sorry for not having fixed that issue yet on the beta server, but we are not community managers and we had quite busy weeks…)).
Anyway, I wanted to write here now about what all went wrong during the release and how it happened.
Thursday, 7:45 am
I turned up at work at that time because it was the plan to make the release in the morning time so that the first forts would be finished around 4 pm / 5 pm. We thus expected that then the first battles would happen at Friday around 4 pm / 5 pm. I started working on several minor bugs, like fixing a glitch in the fort management and a problem with that particular piece of javascript when executed with Internet Explorer browsers.
9:30 am
About at 9:30 am Anthraxx pushed the button to make the release. About 9:31 am Eiswiesel ran into our office and yelled “Build time for Forts is 30 seconds”. That’s how it’s been on the beta worlds. Uops. We immediately fixed that wrong value, but when we did that, 3 forts had already been built by players. 8 other followed because we forgot to change the values in the task queues that had already been inserted into the database. These players had of course a minor advantage over other players since they were the first to have forts. Some players asked to undo that unfair development but that is not a simple request. We just decided that these lucky ones were just lucky winners in the fort build lottery. Their prize was the immediate attack of other players.
And that was how it came that the first fort battle was to be fought at 9:37 am on Friday instead of 4pm as we planned to. But that was not really to our disadvantage however.
10 am
A few minutes later the first bugs reports dropped in. It was my change at that morning that on the one hand stabilized the IE browser but also introduced a bug that made it impossible to invite other towns. But other bugs got visible too – IE 6 users were unable to build forts because the name field of the founded fort did not turn up on their screens. A little bit later other players reported that a county had 4 forts while the one below had 3. Oups. Another oups when players found a cozy little location of another fort:
Quite a nice location I’d say, but also quite wrong. Later it also turned out that the county with 4 forts had not 4 but 3 forts and there was one fort missing on the map that is shown in the minimap. The other county with 2 forts in the minimap had however 3 forts and the 3rd one doesn’t show up in the minimap.
We still have no clue what to do with Fort Los Angelus, neither do we know what is causing the minimap to behave that strange.
Thursday afternoon
Ok, so the first battle was to be fought on Friday morning. I checked if the policy port (1028) was reachable and after a few tweaks it worked. I should have checked the other port as well then (1578? Too lazy to look it up now), but I didn’t. Instead I spent some time on implementing battle logs so that we can later replay the battles. Don’t ask for a playback function, I still don’t know if the recorded actions are complete since I lack a playback function myself. I did also fix some other errors, just like my 3 colleagues in the office.
Around the late afternoon a bug turned up that always existed but never became visible: Due to the fort battles, for the first time in west it made sense for players to gather at a single location. When about 150 players had been luring at the first fort that was declared war on, the player list functionality broke due to too many players at one spot. That was also eventually fixed at afternoon.
When I left the office I had an intensive 11 hour day behind me and I had a bad feeling about the 200 players who were about to fight the first fort battle on a public server. Due to the mishap in the morning time, I had to turn up around 8 am on Friday as well (I normally come in around 10 am).
Friday, 9 am
I still had not checked the second port when the first battle was about to be fought. I just expected that if the policy port was running that the other port would be forwarded as well. Well, checking the port was not that simple because I had no application to test that in a valid way anyway. Had no time to write such a tool.
And this is what went wrong when the battle started at 9:37:
We instantly noticed that we couldn’t connect with flash to the game. That was bad. Looking into the server logs we saw that the server was bombarded with policy file requests that got answered promptly. But no connect request came through. After a couple of WTFs all over the place, hastily hacking on console windows and checking connectivity we realized that of the 3 ports that the server opened, 2 were forwarded correctly and that was the policy port – and the maintenance port that shouldn’t be available to the Internet at all. The game port that is used for the flash communication was simply not addressed by anyone.
After we figured that out we were able to fix it in time and about 10 minutes later the connection worked to my great relief. We could view the battle taking place and the final report to be sent to the spectators.
Later on Friday we noticed that players reported that they were playing on the wrong locations when they joined multiple battles. Also we noticed that the final report was not successfully sent to the users (but that was not noticed by the players due to the lack of information that there should be a report shown). Turned out that the reports became larger than 65kb which was the upper limit for reports at that point. I fixed that bug along with the bug of illegal player positions.
Over the day several battles were fought and the battle server stabilized. We made a few minor changes then – and now I am curious how stable the java server is going to be over the weekend. That is quite a stress test that is currently running. But I am quite optimistic that since the server has survived now for quite some time that it will run stable for some time. Yet I am not sure if we fixed all memory leak issues and so I don’t know if the server is not going down within the next 48 hours. I can’t predict anything there yet, but we’ll see.
I hope that the next week is going to be a bit more calm so that we can fix a couple of other problems and do a bit planning for upcoming features. I would expect that after this huge version release that 1.21 is not going to introduce such large features but will only provide minor features that we couldn’t get into 1.20 yet – but we’ll have to see what we will work on for that release.
We will also see next week when we are going to unroll 1.20 on other worlds and also other languages. This could take the whole week maybe, but I am not sure on that. I hope that the players of other worlds are patient – it shouldn’t take so long anymore.
PS: I am getting constantly requests to increase the number of beta players on the beta world. I am aware that a lot of players want to get into the public beta but you have to understand that I can neither change the player limit nor can I make the decision myself. It has however been decided that the player limit is going to be raised in the future. I can’t really tell you more on that because I simply don’t know more about it myself.
o man :/ we will be patience! =]
but we are so crazy about that =]
Can’t wait for 1.20 to drop on English servers, everything looks great so far.
Hey zet,
Nice post… well I hope, that there won’t be so many probs any more… but what you forgot to talk about: there is a huge build-bug. I actually don’t know if it was there on friday but on saturday it definetly was. Here’s a link to that bugreport:
http://forum.the-west.de/showthread.php?t=23853
Even if anthraxx said it would be fixed, it isn’t. Just to remember you 😉
Greez Micky
“Just imagine a group of attackers spawning on the flag just when the battle starts. Wouldn’t be so nice for the defenders…”
HAHAHAHAHAHAHAA
Sitting here finishing my translations and red that, gave me a great laugh!
Keep it up! Love reading this!
//Best Regards
Patrik Persson
The West.se CoMa
@Micky1992 – yeah, I did not mention that bug – but I also didn’t mention all bugs we were facing ;). And we also have not fixed all the bugs by now. Next week will be fixing time and we also have to write tools for the community managers for managing the forts. Currently us devs are the only ones who can fix stuff that is related to the forts. If we would release 1.20 now on other worlds, we wouldn’t get anything done.
And that is also the reason why 1.20 is not going on other worlds too soon. Unless we have fixed the outstanding issues and provide some tools for the CMs to allow them changing ownership of forts, cancelling or postponing fort battles. Oh and we really have to set a time when fort battles are not allowed to take place. We’ve been thinking to prevent fortbattles to happen between 10am and 11am so that we can update the software without the problem of interrupting running fortbattles. Currently it’s like “oh, can’t update now, there’s this fort battle running”. And then we watch the battle and hope that one side is going to finish the battle quickly enough so that we don’t run into the next battle if we try to update then…
Dear zet,
as a beta-tester i’m kinda sorry for not cheating the flash-communication, i just thought a) You protected it well, b) i would get kicked in the ass for trying such things. But i promise i’ll try to cheat on that part when i can 🙂
Nice article after all, thanks for telling us such infos, and good luck for the next week. Bye!
@mrbay: Don’t worry about cheating on the beta. Even if you managed to corrupt the database resulting in a world reset on the beta we would still want to have to you on the beta world, especially then (though you might be not so popular among the other testers then :P).
The only rule is that you should then cooperate with us telling us what is exploitable. And if you don’t overdo it with the cheating we probably don’t bother to keep you playing with the stuff that you cheated for yourself – so there’s a small chance that you can keep the stuff your character gets a hand on through cheating ;).
For the next blog entry I am working on explaining how to find cheats and how to use them, so you might be interested in that.
Thanks then.
A few questions, if u don’t mind..
I’ve participated in two battles in the last 2 days, and.
1) The attackers won the first one (yesterday, againts Fort RO) with holding the flag for 10 rounds (i see 1 defender with + health, so i guess we won by holding the flag). However, in the fort log it shows we just held the flag for 9 rounds, not 10.
2) I’d like to ask if could You please write a result field in the battle report as well? 🙂 Coz’ now it sends out a wonderful report about the battle, but i have to check every health value to see who won.
3) So far i thought that You can move any number of sectors as far as theres free place in the next sector, and theres noone from the enemy already there. Did i know this wrong, or has it been changed lately?
4) Could You please clarify the Help site and the game? Because the Help says You need 20.000 construction points to construct a Large (Big) fort, but the game says 41.800 (it’s not a big thing, just a bit annoying that i don’t know which is true 🙂 )
Regarding the cheating.. I would be really happy to test it, and i have a few tools to check the network communication to do so, but fort battles right now are so fast that i have only a little time to check things.. 🙂
Thanks!
@mrbay:
1+2) You can also look on the top where the flag is standing – left side: Defender has won, otherwise the attacker has won. But yeah, there should be some text description, but we lacked some proper screendesign for that. The round counting is just not updated for the last round so I’d need to update that.
3) You can move only to adjacent sectors (which is sometimes quite far).
4) Can’t do that. Actually I have no idea what is correct either since I didn’t do anything about the number values…
About the cheating: if you find anything suspicious, look me now. Wireshark is a helpful tool for that.
Well happy bugfixing 😉
I forgot to talk about your blog at all in my last posting… it was really funny i have to say… especially the part of “break our rules” and so on made me laughing a lot.
Keep on writing in that style, it’s funny, nice and also informative.
Micky1992
I am enjoying trying to test what I can on beta but the one suggestion I would have for the future would be to start people off with a few levels so they can really test out stuff.
For example there would probably be a lot better testing of forts if people started at like lvl 30 ish so they could build their towns up faster to try and test the larger forts. also will open the jobs faster and easier so people could do the jobs to get the needed items for different aspects of forts.
example ships bells. those require quite a few levels to do the job. being able to do the jobs etc would allow people to better stress out the new items and test it better since thats all that matters for beta is a complete testing of new stuff
just my humble suggestion
Proflooney
I have a question, why is it only range weapons for the fort?
Is it possible for a vigor dueller to use thouse weapons?
Zet, would you say that it could be a few weeks before Forts are released onto the English servers?
@proflooney: What you say is very true and I am sorry for my earlier statement on my disappointment. We really need to change a few things so that it becomes easier for you. I will ask if it’s possible to raise the yield rate of items since this would help you already to get more money and getting easier more products. We are also going to raise the player limit on the beta so that you also get more products and money for constructing forts.
About more information about what to test: I think we could figure out how to publish a change log along with all updates we do. I think that would involve not much work from our side and could make it easier for you to spot what needs testing.
@Gravedigger: i don’t know the rules by heart, all that I can say is that we consider a lot of different options on modifying the battles a bit.
@Dupo: We are pushing as much as we can, but we still have problems to stabilize 1.20 on world 1. It sucks, I know, but that’s how it is. We hope that it’s stable enough to release it sometime next week on other German worlds, but there is still a lot that can happen.
So yeah,WHEN will the forts come?Can you zet or some1 else give me a time/date/week or something closer?
i played on world one yesterday and i wanted to go back today but everytime i want to log in they keep sending me back to the homepage
can you help me with my problem? thank you
@naomi99
please ask in the forum for such problems.