Howdy, Stranger!
It looks like you're new here. If you want to get involved, click one of these buttons!
Quick Links
MMO Server Tech - does anyone know?
So a recent discussion on the FFXIV:ARR boards about what kind of server tech MMO companies run has popped up - and makes me realize I have NO idea what their datacenter environments actually look like?!
I remember seeing a picture ones of EVE Online's datacenter, and it was just a few pics of a crap ton of blade servers.
But how is the infrastructure set up?
I"m an IT Project Manager, I was a tech on the network/admin/engineer/desktop/server/you name it etc. side for over 12 years before.
I assume they have a domain for internal networking - only certain users have access to certain resources, internal e-mail, your basic domain environment with Group Policy etc.
Based on the size of the development studio, you can assume a mix of Domain Controller(s) and Exchange, I can't imagine too many dev studios would go Cloud Hosted e-mail.
For the actual developers, I'm sure they've got powerful machines for the graphical work and sound etc. with access to shared network resources, so some kind of local SAN or file server, Group Policy based access, etc.
But what about the actual game environment?
The game itself I don't care about - it's software, whatever base be in Linux or what it's in theory a very large Remote Desktop Services Server - end-user logs in to an interface and gets access to the app - the game. SQL servers and massive databases I imagine.
But that app itself, the game, lots of database tie ins of course, so hopefully a massive SAN infrastructure with redundancy build into the RAID setup and probably multiple heads and hopefully a series of LUNs connected to your network via hopefully redundant switches... whole thing firewalled and hopefully redundant mega-fast internet connections.
The client is what renders the game and the world graphically, just pulling information from the databases continuously on where objects are and what they are and also some kind of processing core and transaction logging etc.etc.
But are we talking fully virtualized environment? Where servers are merely a pool of physical NICs, processors, and RAM resources assigned to various applications which call data from DBs on a SAN?
I know it used to be that a server in game - a world - was its own physical server as hardware problems would sometimes bring down a single shard or world. I HOPE they've all moved away from that and instead use a farm environment with shared resources.
But is the hardware custom? The software certainly is, but do they use blade servers kind of "off the shelf" from Dell, HP, etc. and do they use VMWare or some other hypervisor or....
Questions!
Anyone have answers?
I ask because in FFXIV they are having massive infrastructure problems. The servers can't handle the load. Do they drop hardware in the farm quickly to expand available hardware resources?
Or is the hardware custom built and adding new hardware a difficult and involved process, etc.?
Comments
http://lotrovault.ign.com/View.php?view=CommunityArticles.Detail&id=11#Files
This is an old thread about when LOTRO was in its early days, think some of your questions will be answered, as for the hardware specific unless your 'in the know' i dont think we will find out
but i will follow thread as a hardware / software installer i am curious 
you've piqued my curiosity for that Sir I thank you lol
also found this
http://gdcvault.com/play/1016640/Guild-Wars-2-Programming-the
i haven't watched it myself (football kicks of very soon) but it has some interesting chapter headings
This post is all my opinion, but I welcome debate on anything i have put, however, personal slander / name calling belongs in game
where of course you're welcome to call me names im often found lounging about in EvE online.
Use this code for 21days trial in eve online https://secure.eveonline.com/trial/?invc=d385aff2-794a-44a4-96f1-3967ccf6d720&action=buddy
HomePage/Gaming Blog - http://dalewj.com . MMORPGer - Current game: http://AfterWorld.ru .
Author of Diaries of Afterworld- http://www.jconsult.com/afterworld and the Outside Sci-Fi series- http://www.jconsult.com/outside
No not at all.
As I said, I'm sure the code base for the game itself is all custom.
But the network itself, the infrastructure - no game dev studio is going to write their own OS for their workstations, or their internal email and domain, or their own virtualization environment...
That first helps a lot, confirms for the most part they are working in virtualized environments with servers farms and tons of DB servers and ftp/web servers as auth and patching etc.
I'd still love to know what kind of hardware they use, and what platforms they run.
As I said, obviously the code for the game itself and probably the whole login/patch process is going to be custom, but what is the DB architecture, what VM service do they use? etc. etc.
If it is a virtualized farm, it SHOULD be pretty easy and quick for devs to rack up and ram in more physical resources (RAM, CPUs, NICs, storage, etc.) and add it to the farm.
Unless as I said they are using very custom hardware. I wonder if it makes sense for them to use off the shelf for that very reason, or if the hardware HAS to be custom for the type and quantity of workload?
Now THAT'S what I'm talking about.
Tons and tons of SQL. Makes sense. Much lower total bandwidth than I was anticipating, however total amount of memory, processing, and storage was about what I had figured.
http://www.wildstar-online.com/en/news/lets_talk_about_servers.php
"Our storage is MS SQL Server. We have a “few” realms per SQL server, this is one of our scalable components."
So yes, game companies certainly do use some "M$" technology.
That's actually probably pretty old - the EvE thing.
That's about 40GB of RAM per server, and about 2 CPU per server at 2.3 Ghz.
A "modern" blade server could have hundreds of GB of RAM, dual processors with hyper threading, so you could have two 8 core CPUs per blade w/ Hyperthreading for 32 threads per blade...
Hell, I could price out that infrastructure using HP blades, Cisco switches, HP SANs, and all the enterprise VMWare and SQL server licensing you could ever want.
I wouldn't be that insanely expensive to build or maintain either.
A game server is not SQL, that's a different part of the network, and can run by itself without any/many other M$ servers. But building a game server on Windows server would cause endless networking problems. And most gaming company are replacing their SQL installs with other databases now. 10 years ago SQL was the biggest game in town. Now not so much.
so I still stand by building a game server on M$ tech, not so much.
HomePage/Gaming Blog - http://dalewj.com . MMORPGer - Current game: http://AfterWorld.ru .
Author of Diaries of Afterworld- http://www.jconsult.com/afterworld and the Outside Sci-Fi series- http://www.jconsult.com/outside
That's pretty old tech, dual processor for sure but based on that pic only 8 RAM slots.
Modern blade has a lot more than that. Like 24 RAM slots for a higher end blade.
Actually I think most companies have engines, including the entire network engine, written themselves. Turbine uses their own engine that they have evolved over the years.
Also the internal stuff in a company general will not be on the blade servers where the game is hosted. The blade servers will often be rented out of a facility somewhere from a major provider. Their internal stuff for e-mail and work as you mentioned would be on their internal network on site.
It used to be that each blade was a server instance (not a game world) and each game world was made up of several server instances. Now a days they tend to go with virtualization so that resources can be moved around more easily.
Your source?
EvE used (uses) SQL, Wildstar is using SQL...
Seems every bit of technical info I can find from actual game devs is that DB bottlenecking is their biggest concern when it comes to infrastructure.
Who is a bigger name in DB than SQL?
When I attempted to design an MMORPG (in 2002-2003), I was flying blind on server side hardware. My initial feeling was that a server farm built on Intel hardware with Microsoft operating systems just wasn't going to cut it. So, I was looking at several medium sized Unix servers as a base plane. I thought there were several advantages to this approach.
Logic, my dear, merely enables one to be wrong with great authority.
That makes sense of course, data center houses the actual game, internal office probably separate from the hosting site - some sort of high speed VPN tunnel or MPLS etc. between datacenter and production offices.
Who do they use for virtualization though? VMWare and HyperV are the biggest players.
What about the DBs? Seems every reference I've seen so far (only a couple) use SQL.
And their web / ftp servers for auth and patching? I don't even know who the big names are in that space, but they have to connect to a DB too right so again SQL or?
I mean, biggest bottleneck on the DB would be either hardware limitations - total IOPS?
or on the software side of things - how many simultaneous call your clients can make from the DBs and how fast the DB software can process them?
I mean, you'd obviously want a SHIT ton of RAM in your farm to keep as much data in faster memory than sitting on your drives.
Processing power is probably not negligible of course, but no where close to a bottleneck.
With proper iSCSI and multi pathing load balancing LUNs etc. you shouldn't have any bottlenecks on 1G-10G copper/fiber on bandwidth between servers and SAN...
EvE used (uses) SQL, Wildstar is using SQL...
Seems every bit of technical info I can find from actual game devs is that DB bottlenecking is their biggest concern when it comes to infrastructure.
Who is a bigger name in DB than SQL?
There are a few names that are as big or bigger than MS Sql Server. Oracle for one, IBM for another. In terms of performance, reliability and scalability I wouldn't go with Sql Server. It's very accessible, and good for smaller things, but I would think the amount of data being managed in a full blown MMORPG would be more than you'd want to put in a MS Sql Server. This is based on personal experience more than anything. Sql Server just doesn't scale that well.
Though, it might make sense to go with something that isn't even a sql server for speed. I'm not that familiar with those databases, but Google manages 40% of the world's internet traffic without a sql server, so whatever they are using should scale well. It probably requires a lot more work to use, but once it was setup it would fly.
I can not remember winning or losing a single debate on the internet.
Not sure which magical database solutions suddenly came out but from the main ones I know of, SQL makes the most sense.
Oracle - Too expensive
IBM DB2 - Maybe
Pervasive - Not as good as SQL
Access - LOL
SQL is still the biggest player.
I would wager most companies create their own interfaces between billing, web, and game although I know at least some F2P games use a third party vendor for their item stores which then have to link back in to the company's billing/game.
You can see that many of the websites themselves are constructed with common programs such as VBulletin (before their latest forum update the Asheron's Call forum icon was actually that of VBulletin).
So theirs is based on Oracle, not SQL.
Kind of wish a major MMO besides SWTOR used Hero Engine as that's not the best benchmark of performance.
article from 8 years ago
Engineering Everquest
http://spectrum.ieee.org/consumer-electronics/gaming/engineering-everquest
You'Re About To Go into the Death Star," Joffe says. A balding, goateed guy with narrow, rectangular glasses, Joffe places his palm in a biometric scanner in the first-floor lobby of a building near Sony Online Entertainment's main offices, and a gray door unlocks noiselessly.
The Death Star is a huge, warm, windowless room containing the rows and rows of servers that run Sony's online games. The whooshing of a massive air-conditioning system is so loud that conversation is almost impossible. A large steel cage surrounds more than 500 servers stacked 32 high in towering racks--and this is just one battalion, albeit the largest, in Sony's 1500-machine army of servers.
EQ2 fan sites
Still, this is drifting away from the primary point of the thread - we don't need to debate which DB engine is best.
I'm really talking hardware infrastructure.
Server farms with shared resources are almost certainly the norm.
So that means processing servers (RAM, CPUs) and storage (SANs) with associated network gear I don't care about (switches, firewall, internet, etc.)
Do we think/know if they use more off the shelf virtualization tools to build their farm, such as VMWare or HyperV, or if the entire thing is custom?
And the hardware from the WoW blade that was for sale was an off the shelf from Dell I think it was?
Is it them assumed that each blade in a "modern" MMO datacenter is off the shelf too for better price and faster turn around on hardware upgrades and replacements/expansions?
So then if the bottleneck is in the databases, I mean you can add more heads and/or drives and increase your IOPS to the SANs and I'm sure there is a LOT of software/code optimization that can be done in the engine to speed up client access to the DBs...
but is that it?
I suppose that is true.
It's got to be hard to build an infrastructure that can support a million + players trying to hit your servers at the same time. It's pretty much a DDoS attack.
It just boggles my mind that after 13+ years of MMOs, no one seems to have "the answer" on how to have a smooth launch.
Don't they test exactly where the bottlenecks will be and figure this stuff out in beta?
I suppose that is true.
It's got to be hard to build an infrastructure that can support a million + players trying to hit your servers at the same time. It's pretty much a DDoS attack.
It just boggles my mind that after 13+ years of MMOs, no one seems to have "the answer" on how to have a smooth launch.
Don't they test exactly where the bottlenecks will be and figure this stuff out in beta?
It's probably a money thing. How much sense does it make to spend the money on something for opening day when a month or two later you're going to have some fraction of that traffic to deal with? Do you scale for the long term traffic or the opening day traffic?
I can not remember winning or losing a single debate on the internet.
I also think that a lot more is happening on the client side these days when compared to older MMO technology.
That shifts a lot of the heavy lifting away from the servers which improves performance for the end user while lowering the bandwidth and processing requirements on the server side.