Originally posted by BadSpock I suppose that is true.
It's got to be hard to build an infrastructure that can support a million + players trying to hit your servers at the same time. It's pretty much a DDoS attack.
It just boggles my mind that after 13+ years of MMOs, no one seems to have "the answer" on how to have a smooth launch.
Don't they test exactly where the bottlenecks will be and figure this stuff out in beta?
It's probably a money thing. How much sense does it make to spend the money on something for opening day when a month or two later you're going to have some fraction of that traffic to deal with? Do you scale for the long term traffic or the opening day traffic?
That is a very good point.
What kind of solution could MMO devs implement?
Renting or leasing servers for a month or two for launch? Utilizing something like IaaS (infrastructure as a service) to have additional resources hosted by a 3rd party provider (cloud) for the initial launch rush?
"3,936GB of RAM, 2,574GHZ worth of processing power"
But can it run Crysis?
Sorry, had to...
That's a serious investment.
2,574 GHz at about 2.6GHz per processor core would be 990 cores, with monster Enterprise blade servers, you can get up to 4 processors per blade, 8 cores per processor - so about 30 high end servers.
3936 / 30 = ~132 GB RAM per server.
Conveniently, HP has Enterprise level blade servers with those exact (ish) stats for about $30,000 a piece.
Add in hard drives (don't need much because you likely have SANs) and other tech like redundant NICs, you are probably about a million USD for servers, SANs get very expensive depending on the amount of storage and speed you need, but storage as a whole is cheap - you could probably pull off 30 TB of storage on 15K SAS drives for your DB for 150-175k.
10G capable (either fiber or 10G copper) switch gear wouldn't be more than another 75-100K for than many servers and SANs.
Add in UPS, Firewall...
Easily 1.25 million or so. (Very rough math of course, but just a fun estimate.)
"This is the persistence layer of EVE Online. There is only one database server running Microsoft SQL Server 2008. This is backed up using online backups to the 'old' TQ DB hardware, a fiber channel RAID array. The current database resides entirely on solid state disk drives, two RAMSAN400 and 2 RAMSAN500 units from Texas Memory Systems. They are not SSD RAID disks, but rather single disk drives capable of high data throughput but crucially fully random requests. On traditional hard drives this incurs a lag due to disk seek operations while read heads are placed in the correct position on the platter and the data is read out.
At peak hours as of 2007 the database handles around 2-2.5k transactions per second, which generate roughly 40,000 input/output operations per second on the disks.
The server used is an IBM x3850 M2 server with 128 GB RAM and two 2.6 GHz six core Xeons (Dunnington) running Windows server 2008 x64 Enterprise Edition and Microsoft SQL Server 2008."
So as of 2007, 2-2.5k transactions per second, 40,000 IOPs.
That is a really, really high IOPs count, but using SSD is the kicker.
You can get double those IOPs now on SSD, but I wonder what the life span is?
I'd expect it to vary wildly from game to game. If you can never have more than 50 players in a zone at a time without the players being forced to sit and wait at a loading screen for however long it takes to do what you need to do, then you can take a lot of shortcuts that aren't available if you want to have tens of thousands of players in a single world and able to travel around within it in a way that looks seamless to players.
Furthermore, there aren't firmly established standards of, here's how things are supposed to be done in an MMORPG. It's not a fluke that a large fraction of good MMORPGs build their own game engine for their particular game.
Now, I wouldn't propose that any MMO use this actual service, but the technology behind it would very neatly fit into the scalable instance server type category, where you could just spin up instances as needed (on some scalable resource).
I remember the original Everquest used to place certain combinations of zones on specific hardware, and each server had it's own dedicated set of hardware. And the chat channels were a loosely modified IRC server (and often would run/crash/function independantly of the game servers).
Comments
That is a very good point.
What kind of solution could MMO devs implement?
Renting or leasing servers for a month or two for launch? Utilizing something like IaaS (infrastructure as a service) to have additional resources hosted by a 3rd party provider (cloud) for the initial launch rush?
but everyone knows eve-online runs on Excel
serious comment --> there's a fairly recent dev-blog re the hardware infrastructure that ccp use for hosting in their London datacentre
some light reading -->
http://www.pcgamer.com/2013/06/15/eve-online/
http://www.datacenterknowledge.com/archives/2010/06/16/closer-look-eve-onlines-new-server-cluster/
https://wiki.eveonline.com/en/wiki/Tranquility (old but has a recent(ish) update)
"3,936GB of RAM, 2,574GHZ worth of processing power"
But can it run Crysis?
Sorry, had to...
That's a serious investment.
2,574 GHz at about 2.6GHz per processor core would be 990 cores, with monster Enterprise blade servers, you can get up to 4 processors per blade, 8 cores per processor - so about 30 high end servers.
3936 / 30 = ~132 GB RAM per server.
Conveniently, HP has Enterprise level blade servers with those exact (ish) stats for about $30,000 a piece.
Add in hard drives (don't need much because you likely have SANs) and other tech like redundant NICs, you are probably about a million USD for servers, SANs get very expensive depending on the amount of storage and speed you need, but storage as a whole is cheap - you could probably pull off 30 TB of storage on 15K SAS drives for your DB for 150-175k.
10G capable (either fiber or 10G copper) switch gear wouldn't be more than another 75-100K for than many servers and SANs.
Add in UPS, Firewall...
Easily 1.25 million or so. (Very rough math of course, but just a fun estimate.)
At peak hours as of 2007 the database handles around 2-2.5k transactions per second, which generate roughly 40,000 input/output operations per second on the disks.
The server used is an IBM x3850 M2 server with 128 GB RAM and two 2.6 GHz six core Xeons (Dunnington) running Windows server 2008 x64 Enterprise Edition and Microsoft SQL Server 2008."
So as of 2007, 2-2.5k transactions per second, 40,000 IOPs.
That is a really, really high IOPs count, but using SSD is the kicker.
You can get double those IOPs now on SSD, but I wonder what the life span is?
I'd expect it to vary wildly from game to game. If you can never have more than 50 players in a zone at a time without the players being forced to sit and wait at a loading screen for however long it takes to do what you need to do, then you can take a lot of shortcuts that aren't available if you want to have tens of thousands of players in a single world and able to travel around within it in a way that looks seamless to players.
Furthermore, there aren't firmly established standards of, here's how things are supposed to be done in an MMORPG. It's not a fluke that a large fraction of good MMORPGs build their own game engine for their particular game.
Now, I wouldn't propose that any MMO use this actual service, but the technology behind it would very neatly fit into the scalable instance server type category, where you could just spin up instances as needed (on some scalable resource).
http://aws.amazon.com/ec2/
I remember the original Everquest used to place certain combinations of zones on specific hardware, and each server had it's own dedicated set of hardware. And the chat channels were a loosely modified IRC server (and often would run/crash/function independantly of the game servers).
Eve is using FlashSystem from IBM now. Expected life is 5 years.