Howdy, Stranger!
It looks like you're new here. If you want to get involved, click one of these buttons!
Quick Links
Why are so many MMO launches a disaster?
Gnostik
Member Posts: 47
EDIT: Just to stave off future nitpicking: yes, I realize it's not actually "every" launch. I changed the title of the thread to try to prevent more pedantic posts that contribute nothing to the discussion. Please forgive the hyperbole in what follows.
---
As I see that Final Fantasy XIV's online sales are still suspended because they can't handle the server load, I'm left to wonder: why are MMO launches still following this pattern in the year 2013?
This is a serious question, and I'm hoping someone with some technical expertise, particularly in networking, can help me understand.
There are two things that make it kind of shocking to me that a major MMO would, once again, be completely overwhelmed at launch. The first is that developers have so many examples to learn from at this point. They may not have all the technical specs on other MMOs' launches, but it's common knowledge that they always fail to handle the server load. So I have to assume any upstart games knows this, and would try their best to not have it happen to them.
This brings me to my second point (a question, really) and this is where my knowledge is clearly spotty, because if things worked the way I thought they did, we wouldn't keep seeing one debacle after another. Isn't it a lot easier to scale server capacity up or down today than it was several years ago? I'm always hearing about how things are moving to the cloud. Shouldn't a company be able to easily scale up their capacity for the launch rush, then scale it back down after the numbers drop?
This is what I don't get. Companies don't want to shell out for a ton of servers that won't be needed after the launch period? That's understandable. But why aren't they taking temporary measures that can be scaled back later? There has to be some sort of service out there that rents servers for a temporary period.
What makes FF especially boggling, is that (to my understanding) they let people on to their servers in the order that they pre-purchased the game - i.e. not all at the same time. So it really seems they had every opportunity to prepare and adjust capacity, yet they still had to suspend sales.
So, it seems I'm left with 2 conclusions. The first is that my paltry technical knowledge is way off, and it's not at all easy to rent or temporarily scale server capacity for a game launch. The second is that it is possible, but studios are too cheap to pay for the capacity they need, and thus always come up short. I'm not sure which is more likely. It seems like any short-term savings you might gain by not meeting capacity would be more than washed away by the bad PR of yet another failed launch. And when you're forced to stop selling your product because you can't handle the load, it's hard to see how that makes business sense.
So please, someone try to explain to me why every gaming company without fail is unprepared for the server load when they launch their MMO.
Comments
EQ2 launch, despite its bugs, server load was not one of them
SOE had tons of servers available prelaunch and SOE let players prepick servers prelaunch
I remember the DCUO PC launch and Free Realms PC launch
I saw no issues w Server loads w both games
EQ2 fan sites
The way I see it ...
Part of it is that MMOs are gigantic, complex pieces of client-server software with a huge surface of things that can go wrong under stess and no matter how good your testing there will likely be more obscure load-related bugs found when it hits the real world.
Part of it is that marketting and IT have diametrically opposed needs. The hype ensures that servers are hit with more traffic in the first day than the game is expected to be able to handle at any other time in its entire life cycle.
Why would launches get better?
Why Beta test more than just as a marketing tool?
Why not just launch and have people pay to beta test your game because it's proven people will spend money to get into a beta and make excuses for you when you launch and it goes bad?
You'll save a lot of money that way so it makes good business sense.
Different devs take different approaches to that. Some use overload servers (ex: GW2). Some add new servers and then consolidate afterwards. Some just ride out the first few weeks if they feel it will have little impact long term.
It's easy to say just have more servers and scale back when you're working on the assumption you're not going to be one of the players that are getting relocated. When you consider the impact on players who will have to transfer or be left on dead/closing servers, the decision of how to manage the release spike in both players and the amount of hours they play suddenly becomes more difficult.
There isn't a "right" or "wrong" way to play, if you want to use a screwdriver to put nails into wood, have at it, simply don't complain when the guy next to you with the hammer is doing it much better and easier. - Allein
"Graphics are often supplied by Engines that (some) MMORPG's are built in" - Spuffyre
"Always" is a greatly over used word.
Every single mmo in history eh?
You stay sassy!
Good point. I guess it's not so cut-and-dry when servers are linked to discrete communities in the game ("worlds", "shards", what have you). But innovating on this point would probably help devs manage the game long after launch, too. If you had a good system for merging servers or moving players around without disrupting their experience it could help down the line when numbers inevitably fluctuate.
Of course, I'm not sure how to do that without having one massive world, which obviously brings its own set of challenges.
The software development company I work for faces the exact same problem. It has to due with budget. Most companies don't want to spend the extra money purchasing servers that they will only need for 1-3 months for the initial surge.
Just some bullet points for people who may not work in IT, when you buy additional servers you will also need to buy a lot more stuff.
However, most game companies seem to not yet have solution to work around this problem.
We work around this by buying physical servers for the projected growth and create virtual servers in amazon's cloud for the initial explosion. That way you aren't paying for sometimes up to x3 : physical servers, networking equipment, san space, support contracts, and crap loads of additional licensing (i'm looking at you, oracle).
MMO companies could get around this by having two types of servers, long term servers and regular servers (this would work even better if it is a subscription game)
Want to play on a regular server? 15 dollars a month
Want to play on a long term server? 15 dollars a month, however you are locked into a 1/2 year contract.
the regular servers will be more susceptible to huge booms / shrinks and since they are all in amazon's cloud its no biggie as servers can easily be spun up or consolidated.
because theyre not always disasters...
dummy.
Thank you for the informative post. Virtual servers in the cloud are exactly what I had in mind. So you've confirmed that it's possible (at least in theory). Isn't a week enough time to get something like that going?
(And no, this isn't about me wanting to play FFXIV. I'm not even sure I'll buy it, I'm just kind of shocked that sales are still suspended.)
And to dovetail with the previous point, more server capacity wouldn't have to mean more server choices in the game. You could simply have more resources devoted to "Shard X" for the first 1-3 months, then scale them back without making anybody change their in-game server.
It's all so simple in my mind, which is why I know I'm wrong, which is why I want more info!
You aren't wrong, it is a very simple process last week we had a client tell us their reported active user sessions would be around 2000, I wake up next morning to see 300-400 alerts from our monitoring system, oh look they have 6000 users on and all our servers are cpu capped. (thanks for letting us know client
) We spun up x3 more servers and had their environment stable right around COB the same day.
Technically speaking we can get a new server up and running in a few hours, however with our change management system that becomes around 5-6 hours (when people are giving you lots of money you need to make sure you get it right the first time)
I think the issue is taking stuff to the cloud is still kinda new to some people, so they may not realize how efficient it is yet.
"What is truly revealing is his implication that believing something to be true is the same as it being true. [continue]" -John Oliver
MMO launches are very complex, making it difficult for their Lead Engineers to get their heads around what needs to be done and do all the proper contingency planning.
Even with experienced+smart engineers its easy to miss every single point of failure.
Points of failure typically are:
1. Database capacity/performance. Consider clustering to mitigate risk
2. Network capacity/performance. Use a proven server farm vendor with high peek bandwidth
3. Account creation/Billing. One could outsource this to proven Account management and billing vendors who have their own load balancing design.
4. Game Server Performance. Consider Load balancing or lots of stress testing using bots (easier said than done
)
5. Web servers. Citrix load balancers (ex. Netscaler) in front of multiple web servers are pretty effective.
6. DOS Attacks. Again, work with a server farm vendor who has proven peek capacity(and strategy) for these types of situations.
Ultimately, its a big task to plan for any company. When in doubt, I would stick to the old Army motto for critical mission success:
"Rehearse, Rehearse, Rehearse!"
-WL
Werewolf Online(R) - Lead Developer
Here's the thing.
While all this information on server tech etc is interesting as information...as a consumer I don't care.It's not my job to provide the product and service promised it's the company producing these products and services.
Just like I don't care how a car is assembled only that it works as advertised and works safely.Now things happen and go wrong which is why we have factory recalls but these things should be the exception not the rule or the company deserves to go out of business.
Drakynn, I bet you might be surprised by how many businesses have a successful launch purely by luck, vs. all the preparation others companies might do.
Why did FF launch fail and continue to fail?
I bet mostly due to bad luck multiplied by bad reactive decisions exaggerated by players not logging out... ever.
-WL
Werewolf Online(R) - Lead Developer
Solutions could be:
Less hype pre launch. If the game is any good you will still get many players. And many more of them will not end up being disappointed about how the game turned out. They may in fact love it for its merits instead of for its hype.
Staged launch, perhaps increase the price for early access instead of discounting it. Already here with paid beta.
Could also have limited play time at launch, with accounts limited to being able to be logged in at certain hours of the day, spreading the load, but giving everyone a chance to play at launch.
Could use some editing.. but I'll fix that in a patch or expansion.
I agree.
The marketing departments have too much control over the games these days.
They are selling the hype and not a game. So it doesnt really matter if the product is any good. Or if they can deliver. Enough people will pay and not bother with a refund.
Well the only thing that comes to mind to stay on the FFXIV subject is that the company's communications might be to far split, meaning Sales might not communicate much with the Tech. team.
I am unaware of pre-sales ending so don't know the time between end of pre-sale and pre-sale early acces release...
But it could have been simple math.
But disaster as in bugs or other ingame issue's I already know these things happen with early acces or even release as there are so many differnet computer configurations where many might encounter the same or similar issue, there is also a large portion of players that don't encounter them which could be many reason they don't experiance those issue's
Main reason I avoid pre-orders or release date, well atleast the last couple of years, while I never pre-order a MMORPG I use to be that on release day ingame player. I just accepted and understand that things might go wrong in one of the most complicated game genre's to develop.
So it sounds to me like somebody who owns a cloud could make a few extra $ by creating a system that quickly grows and subtracts with the life span of a game. Develop a nice system that sees user level nearing 100% and builds a new server to handle the load. Once the server load decreases then the servers turn off and go to some other use as the users logoff.
Seems to me like a business opp there.
HomePage/Gaming Blog - http://dalewj.com . MMORPGer - Current game: http://AfterWorld.ru .
Author of Diaries of Afterworld- http://www.jconsult.com/afterworld and the Outside Sci-Fi series- http://www.jconsult.com/outside
Two solutions that seem to work are 1) channels (ex:Runescape) and the single-server approach (ex: ATITD, EVE). They remove the issue of server transfers and dead servers, but they also come with their own unique technical challenges.
There isn't a "right" or "wrong" way to play, if you want to use a screwdriver to put nails into wood, have at it, simply don't complain when the guy next to you with the hammer is doing it much better and easier. - Allein
"Graphics are often supplied by Engines that (some) MMORPG's are built in" - Spuffyre
As a developer I can tell you that it is very hard to make things happen reliably overnight. Especially if your initial architecture doesn't support what you are trying to do. Yes today we have AWS and EC2 (Amazon Web Services) which allow you to allocate a lot of bandwidth dynamically. That said if you are really dealing with something massive you still need to contact them ahead of time and let them know that you will be requesting massive number of boxes.
All that assumes that your back end architecture plays nicely with external virtual machines. AWS become really promintent and reliable only in the past 1 - 2 years, which is a lot less than a game dev cycle. Dev studio can't stop and change their achitecture just because a better tech became avaliable 1/2 way through it costs too much money.
So when people write something like - well you have a lot of people can't you just write something really fast to add more boxes I laugh. Because writing something really fast means minimal amount of testing and usually means you are hacking existing networking infrastructure.
Truth be told any type of launch weather it is a MMO or any other online system is a very difficult process. You can stress test your system all you want there will always be bugs that weren't caught, hard drives will fail, networking cards will stop responding, some providers will route traffic in a wierd way, i.e. 6 people will complain because things are slow, chrome users will complain why things are not in html5, mobile users will complain that on a blackberry XX version z things don't render correctly, etc.
Building things are like MMOs is not trivial, there is a reason why engineers get paid big $. Even then we can't test everything in every way possible. So we do the most time and budget allows and then it is off into the wild. There are some launches when you luck out and things go well - and then there are other 90% of the launches when you are in the office 24/7 trying to get everything to work.
If you think that developers are happily asleep at home while people are trying to log into the game you are dead wrong. Most likely developer are at their desk trying their hardest to fix things while their manage yells at them that they are not working fast enough.
For Tera's launch they allowed you to freely server transfer once a day. So, if you're server was overloaded you could just transfer to a server that wasnt. Yeah, you might not be able to play with your friends and you might have to rejoin your guild later on but atleast you could play Tera in the first weeks.
However, this would not fix FF14s server issues. SE has 1 server cluster in Montreal, Canada for all NA and European servers and only 1 instance server for both regions.
So, FF14 ARRs launch was doomed from the start because SE lacking forward thinking in concern to servers.
Customers who are docile and non combative when it comes to their money = bad product.
It's quite simple. You have a large segment of sheeple who think the status quo of borked product release is okay and to be expected. They think expecting some semblance of quality control is "entitlement". They think getting the most for your dollar means you're poor. Then they turn around and spend hundreds of dollars on bad product and defend the multi-millionaires profiting.
Bottom line is in general, consumers are idiots, the companies know it and get away with it and it won't change because people are only getting dumber.
¯\_(ツ)_/¯
Some things can't meaningfully be tested until a game goes live. In beta, people aren't trying to maximally exploit your systems in order to power-level or farm with maximum efficiency, as they know it's all going to be wiped soon. Come launch day, a lot of people are doing that, so players behaving differently breaks some things that worked fine in the beta and in earlier testing.
As for server load in particular, one problem is that you don't really know how many players you're going to have to host concurrently. You don't know how many people will buy the game, and you don't know how much they'll each play. Making things scale up flawlessly so that you just add more hardware and everything works is not a trivial thing to do.
Furthermore, games tend to have a rush of people at launch that quickly subsides after launch. You absolutely do want to make sure that you'll have plenty of hardware for your players a month after launch. But if that only takes half as much hardware as it would take to handle the launch day rush, do you really want to shell out a bunch of money for hardware and licenses for it that you'll only briefly use?
No one really cares that much if your game is a mess at launch. People who don't want to play games that are a mess won't play your game near launch anyway, so they won't care if you had a flawless launch. People who play a ton of games at launch are often the sort of people who tend to rapidly jump from one game to the next, so they'd probably have soon left even if your game had been flawless at launch. Even someone who really wants to play your game in particular and might well hang around for a long time might be upset if the game is a mess on launch day, but if it's fixed by the end of a month, is he really going to quit a game that is now good because it used to be broken?
That, of course, assumes that you can quickly fix the things that go awry on launch day. If your game is still a complete mess six months after launch, you've got big problems.
I want to also add something that I don't think has been mentioned so far is that not too many companies or really any company gets THAT many chances at having to manage an MMO launch in the first place. Look at major companies like Blizzard who only had to manage 1 major MMO launch followed by expansions which are similar in terms of traffic but I believe MMO launch provides more unexpected values where expansions you are kind of already relying on the current subscriber base already and already expect/know that your servers can handle a great portion of concurrent players by then.
You can do your homework all you want about it, but like going to college and finding out your bachelor degree doesn't directly translate into your work like you hoped it would, you actually have to do it and be there to acquire that experience and make those mistakes yourself. MMO launches aren't a common thing and it isn't an easy matter for many companies to figure out compiled on the other reasons already listed here i.e. resource management, computer technology, beta testing vs live etc. Just be glad after the first couple weeks, generally with many MMO launches it smooths itself out. I never played an MMO for a prolonged period of time (many months/years) that truly had persisting server issues.