How the Open Beta launch went for us

Leaving the team in Erik's capable hands, Jonathan and I flew to California last week to show Path of Exile to an array of journalists. We got back a few days before Open Beta was due to begin and soon realised the effect that our press tour had had on the exposure of the game. Website traffic was rising by the day and dozens of positive articles were appearing online. We quickly ordered more servers. This turned out to be a good idea.

I will always remember the day before Open Beta as a day of controlled chaos. With less than 24 hours to go we were putting finishing touches on most systems and performing final quality assurance. That day saw a lot of improvements to the final boss fight of Act Three and the general stability of the 0.10.0 patch. We managed to get everything done that we wanted, but it took the team all the time we had available. By midnight we were down to a core group of developers mostly working on creating videos and descriptions for the microtransaction effects. Several of us worked through the night, but by 7am we had a final build prepared for deployment. By this time, our sales figures were going absolutely crazy as everyone rushed to buy their last minute Closed Beta kiwi packs. We were receiving new Diamond supporters at the rate of several per hour. It was with great hesitation that I pressed the button to end the Closed Beta and deploy the 0.10.0 patch.

Our "twenty minutes of expected downtime" was over before I even noticed. It turns out that our server hardware is indeed as fast as Jonathan was telling me. With this deployment complete, 0.10.0 was live on the realm but players were not yet able to log in. We quickly logged in to check that everything was working, which it thankfully was. We had around 90 minutes before the countdown hit zero, so we let players download the patch and grabbed some breakfast.

Like all companies that want to transfer a lot of data, we were using a CDN (Content Delivery Network) to deliver the patches to our users. CDNs have a lot of servers all around the world so that people can download files quickly from the closest ones to them. During the Closed Beta, we frequently received emails commending us on how quickly people could patch from us.

This did not scale as well as we hoped when approximately fifty thousand people tried to download the 1.5gb patch at the same time. We had warned the CDN in advance, and they had bolstered our capacity, but more was needed. As they added servers to their network for us, more and more users started to patch. By the time the Open Beta countdown hit zero and users started to join the game, it was apparent that some users near very crowded CDN servers were going to take many hours to patch. We are very sorry about this and are doing what we can to keep it fast in the future.

Each hour, approximately 9000 users finished patching. We watched the number of online players rise steadily while the servers held up fine. Just as we began celebrating, a cake arrived, sent by a longtime fan and Diamond supporter tpapp157. We assembled the team together and enjoyed the cake. I made some vague semi-speech about how well sales, traffic and server stability were going, but I had been awake for 30 hours by that point so it probably didn't make much sense to the team.

Our peak hours for concurrent users are usually in the morning, NZ time (which is in the evening for Europe and in the afternoon for America - the only timezone asleep then is Asia, where we have done no marketing). This meant that by early afternoon we had hit a local maximum user count that the servers could comfortably handle. We prepared the 0.10.0b patch (patch notes here) and I went home to sleep. The patch was deployed successfully, though our CDN bandwidth issues meant that it took a while for some users to download the 8mb that the patch needed.

By about 5am, the realm had crashed several times, booting everyone off each time. When I arrived at the office, Jonathan was preparing a fix which was deployed, resolving that issue. There are now no known or unknown realm crashes. Since the last crash, the server has been very stable.

Today was spent answering emails. Our support email and address and PM box have been receiving literally thousands of requests, ranging from the simple account issues to complex tech support questions. Despite having full-time support staff, we moved many developers over to help with answering questions in an attempt to catch up with the queue. There is still a giant backlog while we look into hiring more support workers (only apply for that job if you're in Auckland, NZ). If you've written to us, please be patient because it may take us a few days to get to your message.

We also changed our policy on the deletion of microtransaction items today, and offered some extra stash tabs to Closed Beta supporters who bought a lot of tabs before the Open Beta started. We are constantly trying to improve our microtransaction shop so that it's as fair for everyone as possible.

We hit a peak online player count of 56,700 players online today. I expect we'll beat this total in coming days. Jonathan has a lot of interesting stats he plans to share with you about the Open Beta, so we'll try to post those over the weekend. Expect the 0.10.0c patch sometime within a few days also.

Overall, the day we entered Open Beta we had as many visits to the website as the entire first year of traffic after Path of Exile was announced in 2010. We sold approximately $200,000USD of Kiwi packs in the last two days of the Closed Beta. This has been earmarked for server and bandwidth costs, which are pretty high right now. I believe in the Closed Beta we crowdfunded a total of around US$2.5m over 35 weeks, though of course much of that has been spent on development costs.

Sales have of course been much more sane in Open Beta now that we're not selling Kiwi packs or Closed Beta keys and players have a lot of microtransaction credit to spend before they need to top up. Every purchase helps the development of the premier Action RPG for hardcore gamers. We are deeply thankful for the ongoing support, and look forward to sharing many adventures in Wraeclast with you!