Good article and I agree across the board. Beta is now a useless PR tool. Smoke and mirrors by and large. I won't even bother to sign up for betas anymore.
I certainly understand your pain. I see it all the time as I flit about looking for something interesting to test with my time. I think that Dominus had the right idea in the beginning, offering the beta up to everybody and then after that small initial period only inviting back the meaningful contributors. I think that several of the comments have the right idea of having a thorough application process to get in initially, but then that leaves developers with the potential of having to sift through a lot of applications to find the ones that are worthwhile. I think that a good idea would be to have a small test client created, and populated with a few known bugs. Some simple bugs that are easy to reproduce and somewhat obvious, and some more complex ones that are more difficult to find and / or reproduce. During the application process ask the applicant to download the little client and report any bugs they find. Perhaps presenting it as something that is currently being looked at and could use a lot more hands on testing. From this I imagine you would be able to weed out a large percentage who either did not download the client knowing that it wasn't the full game and had no interest, or did not report a single bug despite you knowing that had they played it they encountered a minimal of X number of bugs. You could effectively auto close applications that were submitted without receiving a bug report after X days. Based upon my interactions with the MMO world I would guesstimate this would allow you to be done with 50% or more of the applicants. From the rest that are left over you can judge them based upon their bug reports. Did they outline steps to reproduce the bugs? How was their documentation of the bugs? How many bugs did they find?
The potential problem of course is if it leaks out that this is how you're choosing your testers it will eventually lead to people documenting where to find all of the bugs.
Beyond the initial application process I think that more developer communication would be ideal. Dominus seemed to have a bit of it going on, but the more the better. Perhaps a forum post where the developers keep a running tally of known issues, and ideally with a small snippet about the status of their progress towards resolving the bugs or the bugs priority. It could even be something as simple as providing some slimmed down viewer access to the internal bug tracking software, allowing people to see that a bug has shown up there, has been assigned to somebody, and is being worked on. Maybe even just a weekly hour long chat in IRC, or an equivalent, going over what has been accomplished, what's on deck for being fixed, what the progress is of fixing specific things. One of the most disheartening things about being a tester can be feeling like your reports are falling upon deaf ears due to a lack of communication. Understandably it can take a bit of time out of a developers busy schedule to provide such feedback, but I think the pay offs would be worth it in the end.
OP forgot one salient point about beta testing. Those who do beta test and submit bugs, glaring problems like memory leaks and comment on bad design and are ignored.
The problem is that new MMOs don't have demo or trial versions. So of course people will look to beta for that, instead. Give us a proper chance to see if the game is worth buying, and we'll stop trying to use beta tests that way.
Ahem?
So, if some company gives you a demo or trial of an untested unfinished game, what are you going to do? Your post totally makes no sense to me...
You're still looking for the free trial...kinda proving my point...
That Guild Wars 2 login screen knocked up my wife. Must be the second coming!
It seems to me that it has to be presented as a beta. From my experience in betas, here are a few things that scream "Not Really a Beta"
1. Key giveaways. MMORPG, Massively, Fan Sites, etc. If you are giving away keys to people happen to be there to click, not a beta. That's marketing
2. Priviledged Key giveaways. IGN, Fileplanet, etc. People receiving keys probably believe they earned it and/or paid for it and therefore are getting a sneak preview.
3. No pre-screening. Register an e-mail address, get a chance at an invite. Again, screams marketing.
Things that do make it appear as if it's a legitimate beta:
1. Questionaires. Dxdiag, gaming history, etc. Feeler questsions such as what aspect of the game interests them.
2. Focused testing. Depending on the engine/coding, limit tests to limited aspects, level ranges, pvp instances, etc.
3. Small wave invites. If everyone and their brother gets in, it appears to be a stress test and/or marketing.
4. Forced feedback. Complete a quest, fill out a questionaire. Play for 30 minutes complete a questionaire. Craft something, fill out a questionaire.
While playing a game should be fun, beta should also feel like a job.
The more I age, the more I understand, and the more I wonder about the "current state of affairs," as you so eloquently pointed out in a strong active voice. What happened to the gaming industry? You are right, no one takes chances any more. Rift is WoW, COD is Battlefield, everything is the same, and freaking fantasy to boot. Ten years ago I wouldn’t have even read website things like this because frankly I was gaming, and everything else was a waste of time; of course that was before twitting face booking and self entitled thirteen year olds that occupy the masses of gamers I believe now. Beta has become a tool for advertisement, I agree. In this latest Tribes beta I recalled something about paying for something for some reason or another, I didn’t stick around long enough to find out, the game just wasn’t for me. Now you pre-order to get into so called beta. However, I think what is going on here is you forgot to relate beta to that of free-to-play, because everything is tailored to thirteen year old kids who want things for free because they have no credit card to put it on. I've waited for over 8 years for another real man game like Neocron (indie- youtube "neocron pvp"), or Planetside as a substitute. Where conflict happened for a reason, and outpost wars, territory, and resources made it happen... However, to answer your question, when you say beta, I still think beta. I take it on a case by case basis.
It seems to me that it has to be presented as a beta. From my experience in betas, here are a few things that scream "Not Really a Beta"
1. Key giveaways. MMORPG, Massively, Fan Sites, etc. If you are giving away keys to people happen to be there to click, not a beta. That's marketing
2. Priviledged Key giveaways. IGN, Fileplanet, etc. People receiving keys probably believe they earned it and/or paid for it and therefore are getting a sneak preview.
3. No pre-screening. Register an e-mail address, get a chance at an invite. Again, screams marketing.
Things that do make it appear as if it's a legitimate beta:
1. Questionaires. Dxdiag, gaming history, etc. Feeler questsions such as what aspect of the game interests them.
2. Focused testing. Depending on the engine/coding, limit tests to limited aspects, level ranges, pvp instances, etc.
3. Small wave invites. If everyone and their brother gets in, it appears to be a stress test and/or marketing.
4. Forced feedback. Complete a quest, fill out a questionaire. Play for 30 minutes complete a questionaire. Craft something, fill out a questionaire.
While playing a game should be fun, beta should also feel like a job.
they dont really do that since they are pretty upfront with the state of the game and they let oyu know its a beta not a free trial
I remember when Beta's were more work than play and rather miss those days honestly. Can they be brought back?
Yes, certainly.
The problem though, is that it's probably more trouble than it's worth at this point.
The amount of vetting to get a pool of viable testers can burn up time.
The name of the game is "bug hunt", but most Beta Testers think it's " Is it fun?"
It was at one point an inroad into the game business and really isn't now. Adults , by and large, don't have the tme to devote to it and the current generation of teens has sever entitlement issues.
On to entitlement issues... AC and AC2's beta tests (specifically AC) proved that those that do get the responsibility have an impression of vested ownership that Game Devs have had to disabuse them of in order to avoid legal liability.
I think a smaller game could make it work though.. They have a more pronounced need for the bodies as opposed to these mega-groups that can throw money around more freely.
It seems to me that it has to be presented as a beta. From my experience in betas, here are a few things that scream "Not Really a Beta"
1. Key giveaways. MMORPG, Massively, Fan Sites, etc. If you are giving away keys to people happen to be there to click, not a beta. That's marketing
2. Priviledged Key giveaways. IGN, Fileplanet, etc. People receiving keys probably believe they earned it and/or paid for it and therefore are getting a sneak preview.
3. No pre-screening. Register an e-mail address, get a chance at an invite. Again, screams marketing.
Things that do make it appear as if it's a legitimate beta:
1. Questionaires. Dxdiag, gaming history, etc. Feeler questsions such as what aspect of the game interests them.
2. Focused testing. Depending on the engine/coding, limit tests to limited aspects, level ranges, pvp instances, etc.
3. Small wave invites. If everyone and their brother gets in, it appears to be a stress test and/or marketing.
4. Forced feedback. Complete a quest, fill out a questionaire. Play for 30 minutes complete a questionaire. Craft something, fill out a questionaire.
While playing a game should be fun, beta should also feel like a job.
Pretty much mirrors my own thoughts. Lord knows I've sent in bug reports til my fingers bled on a few games in the past, only see a complete lack of acknowledgment or fixes occurring, even on simple things. Kind of given up these days.
A Modest Proposal for MMORPGs: That the means of progression would not be mutually exclusive from the means of enjoyment.
I have been in countless beta's since my first, which was EQ. I can tell you on reason why beta testers do not do the job anymore. It comes right down to auto-responses to bug reports. In a recent beta I was involeved in, I had to report the same bug 8 times before I got an actual response, all the rest were computer generated, hello I hear you are having an issue, did you try this crap.. Needless to say, I had no interest in doing that repeatedly, so my bug reports were much less often.
So, as a developer, I ask you to do the same thing I ask my bank, my cable company and everyone else...stop being so automated. While it might get rid of the flow of bug reports that truly are garbage...it also deflates those that are dedicated to the task. Give us IN GAME petitions again with guides/advisors or whatever...that we can TALK to about an issue.
Keep a list of GOOD testers and call upon them again, in the next game. That in itself is a perk to do a good job. Make a rating system for your developers to RATE the bugs reported and assign scores to the testers. Give perks for good reports and an ongoing relationship in future games. Things like bug squasher as a title, would be a valuable perk in the long haul of the game.
I agree that a beta should be a testing ground not a sneek peek. But here is the problem the developers bring on themselves. Most if not all games do not launch with a trial period. I mean a free trial period that they all eventually add in there later.
No one wants to pay for and play a game that ends up tanking (this means you STO and WAR). But most people that will end up being the dedicated type want to be able to be there for day one.
I have been burned by enough games that ended up not delivering that I now don't preorder anything. That solves the problem of getting burned but its not the best solution. I WANT to try before I buy. I am the one paying the cash for the product so that should mean something right? I don't mean its owed to me or they HAVE to do it or anything like that I am just saying that would make me a satisfied customer. As it is I am not a customer at all until the game is established and I hear enough good about it to make an informed decision. So a good game will eventually get my business but it could have it so much sooner with a trial.
Since none of them have free trials at launch I will do everything I can to get into a beta these days. Don't get me wrong I test. I have submitted hundreds of bug reports and participated in untold dozens of stress tests in my time. But the real driving force behind me and I would bet most other people to include those that don't test anything at all durring a beta is the desire to see if the game will have any appeal at all. Does it have any potential or is it all hype?
Anyone else out there see my point? If I know a game is going to have a trial at launch I wont bother with the beta if I dont have the time to put into it.
Good article, and touches on why I have not beta-tested any MMORPG yet. There is no way for a game company to release a polished product without beta-testing it on a goodish number of players. It seems like closed betas are now the only beta, and "open betas" are really a demo anymore. I got the Blizzard yearly deal, so will be beta-testing MoP.
I have beta-tested other games. I made sure to report bugs or game-breaking annoyances and realized that I was there to help the devs and programmers rather than playing a full product.
It is a shame, and a loss of time to the developers and programmers when players use betas and tests to basically just get ahead. It does take time to be a good beta-tester though: bug reports, participating in the forums, number crunching, etc. The bug reports have to be detailed enough so that the bugs can be found and fixed... that isn't always so easy.
Good luck to debuggers... it is not a job I would actually like doing (looking at someone else's code... eep!).
I agree that a beta should be a testing ground not a sneek peek. But here is the problem the developers bring on themselves. Most if not all games do not launch with a trial period. I mean a free trial period that they all eventually add in there later.
No one wants to pay for and play a game that ends up tanking (this means you STO and WAR). But most people that will end up being the dedicated type want to be able to be there for day one.
I have been burned by enough games that ended up not delivering that I now don't preorder anything. That solves the problem of getting burned but its not the best solution. I WANT to try before I buy. I am the one paying the cash for the product so that should mean something right? I don't mean its owed to me or they HAVE to do it or anything like that I am just saying that would make me a satisfied customer. As it is I am not a customer at all until the game is established and I hear enough good about it to make an informed decision. So a good game will eventually get my business but it could have it so much sooner with a trial.
Since none of them have free trials at launch I will do everything I can to get into a beta these days. Don't get me wrong I test. I have submitted hundreds of bug reports and participated in untold dozens of stress tests in my time. But the real driving force behind me and I would bet most other people to include those that don't test anything at all durring a beta is the desire to see if the game will have any appeal at all. Does it have any potential or is it all hype?
Anyone else out there see my point? If I know a game is going to have a trial at launch I wont bother with the beta if I dont have the time to put into it.
This post makes valid points, but also makes me think that the perfect beta-tester would be one who is not hyped up about a game. I know that fans tend to test, but that is sort of a bad thing in the sense that they get too emotionally wrapped up in the game. It is probably better overall to be a fairly neutral tester who does not necessarily go into the beta all hyped up.
I guess the industry's thought process on why there are no demos or trials of most MMOs is that with a long-term game the game publishers probably expect $50-60 to be a relatively cheap entry fee, seeing as some of that initial money usually goes into the first month of subscription (makes it $35-45 which is on average pretty cheap for an AAA title that cost millions to develope). Perhaps the "free month" industry standard is seen as the "demo", because even though you had to pay for entry, the real money they make is from subscriptions and keeping people interested in the game (and more and more often through cash shops too).
They hope and wait for gamers to hype up their game once it launches so that it will attract new players and it usually takes several months before a free trial is offered (usually when the game's box starts going down in price in offline stores). I agree so very much with the above poster that free trials are necessary from launch on.
It would avoid some rather unpleasant scenarios of some pre-order "beta-testers" getting to max level really quickly and then whining about not having anything to do within a couple days of launch, but also would slow down the rate of exploiters cropping up in the game. It would also avoid the somewhat risky gamble of waiting for word-of-mouth to kick in. The latter worked in the old, pre-WoW days because the MMO community was smaller and there were fewer games to choose from, but I think these days, the industry cannot rely on that anymore.
Personaly I like playing in Betas and trying to break the game. Anything from a spelling mistake and bad grammer all the way to game breaking bugs.
Even once released I am still posting bug reports. Unfortunatly I have sent in more bug reports for SWTOR than any other release since it went live, and most of these bugs could of been sorted in closed beta.
Interessting whine. But I'm afraid it's one sided.
If you leave hype aside beta does serve a good purpose if enough people join. It gives devs an opportunity to check the following:
- Performance: A small "elite" group of testers can only focus on functionality of the game. Never on performance and load. Never! Just ask yourself what good is a perfectly polished game that crashes after 100 people log in. And lets not forget ping (lag). A selected group of testers would probably be local, maybe even hired by the company. But if you pass enough beta keys around and you get people loging in from various locations and countries using differens isps then you have a good chance catching problems you never dream existed. This is what performance and load are about. There are also subsections like login server performance, or authentication server performance (usually separate from the main game server) but this is more or less all in same direction. The more people you get the more data you can collect about how your game behaves in extreme cases.
- Functional bugs: Take a good look at your calculator and then try to think about number of options one can use to enter a simple equation. Amazing how many "test cases" can you get out of a simple calculator, right? Now take a look at any game and think about nuber of options a player has to go through, starting with login, avatar creation screen, naming, tutorial... Bazillion gazillion combinations in an average game. This can not be covered by a test team or by small group of 1337 beta testers. The more users you get on your server the bigger the probability is that one of them will discover a way to break the game. It is a probability game, nothing more, nothing less.
- "Idiot users": Let's be honest. All devs can focus on is the positives! After more then a decade in software development this is an axiom in my opinion. A fact! No developer ever assumes that some users might even think about using his application (a game in this case) to do something that was never planned with it. It's like trying to water the plants with a chainsaw or trying to light a fire with a hammer. Maybe it can be done but results would probably be devastating for plants and trees. Same with "idiot users", who can behave "in the wrong way" simply because their assumptions are wrong or their knowledge of gaming and technology is very poor. It's like the story of a lady that used CD tray as a cup holder or of another user that used a mouse by trying to step on it. Small group of beta testers would NEVER have anyone in that group. And if game's error handling routines are poor and an "idiot" user keeps doing something silly that constantly crashes the server then EVERYONE will suffer and not just him.
- "1337 users": Opposite of "idiots". Those guys can hack into your game and cause havoc for anyone around. Botting, scripting, even using APIs to make your own apps and screw the game in your advantage. Good luck trying to find all security holes by letting 100 loyal testers do the testing. And how many games you know that died due to hacking and boting?
There are other things that could be said in gavor of current beta practices. But the way current gaming industry is set up there is NO WAY around how beta is done. Pass quantillion keys and hope for the best. Yes there is also greed involved when companies release unfinished products since all of the guys in the industry want to make money (I would be worried if any of them dont). Greed also plays a big role when you don't want to have proper test teams, which is not problem since you can always find million souls just waiting to log in and try something new. So public gets something in return for early access.
All of the above is "imho". Yet my "ho" is based on a decade of software testing experience and more then 2 decades of gaming experience so I think I know what I'm talking about.:)
No fate but what we make, so make me a ham sandwich please.
After reading that, noone can say you are wrong ... not a word about "improving the game" or " finding bugs" ... what has the emphasis is "Eager to get your hands on TERA?"
Spiider there are many phases of beta I do not think Sanya meant stress test beta which is basically the only form of beta these days. Most of the other testing is done with a group of developer friends and insiders. That use to not be the case. Now beta is used more as a marketing tool that an actual beta test, the game at the stage most people get into beta is almost completely done and no major design changes can be made.
Sanya how would you convince me? Here you go in 4 easy steps…..
Never use beta as a promotion tool splashed all over the web page, pre-order and get a spot in the “beta test”
Do not do a twitter/facebook contest to get into “beta”. Matter of fact; do not use Twitter or Facebook at all for this. If you are looking for real beta testers those are the LAST places you should going.
Be very clear what stage the game is in, what the beta tester can expect from the game in its current state and very specifically what testing you are doing and feed back you are requiring from them. Yes I said it REQUIRING from them. This is not a play test make sure that is completely understood.
Once you do let us in, do not then turn around and tell us we do not know what we are talking when we give specific feedback you do not want to hear. If a very large majority of the real beta testers tells you something in very specific detail without trolling is VERY broken in your game, listen to them do not threaten them or out right dismiss them.
I will use my last experience beta testing in a game called WAR. Just about every real beta tester told Mythic that bright wizards were very over powered and badly broken in pvp and we were basically told to shut up, we have metrics, completely dismissed, and then threatened with expulsion from beta when we did not let it go(some of us got kicked).
They proceeded to launch the game with the class completely unchanged from beta and we all saw how that went, really frustrating to see something that should have been addressed in beta go live, when we told them exactly what would happen in detail if they did not adjust bright wizards, never mind the stuck on every rock in the game issue.……sorry I digress……
After reading that, noone can say you are wrong ... not a word about "improving the game" or " finding bugs" ... what has the emphasis is "Eager to get your hands on TERA?"
And still servers went down which only confirms my theory that this is the right way to go. They let bazillion players try to log in which gave them perfect load test. Beta test successful. The facts that no one could play is irrelevant for devs. They will sell it to you as "early sneak peak" when in fact it's "early load test".
No fate but what we make, so make me a ham sandwich please.
I'm not really sure a basic language port sneak peak really constitutes a beta.
Each company runs betas differently. The server load/mass invite beta is generally a final stage of beta. Recent betas like Rift and SWToR were sneak peaks. Hell in SWToR they even disabled the ingame bug reporting.
After reading that, noone can say you are wrong ... not a word about "improving the game" or " finding bugs" ... what has the emphasis is "Eager to get your hands on TERA?"
And still servers went down which only confirms my theory that this is the right way to go. They let bazillion players try to log in which gave them perfect load test. Beta test successful. The facts that no one could play is irrelevant for devs. They will sell it to you as "early sneak peak" when in fact it's "early load test".
I can follow that logic ... but this is not one test .. it's for all (5) closed beta tests.
I decided to come back to this thread to add something.
Beyond the promontional use of beta and the various points made in the article. There is another common thing that pops up in betas now.
Early on in the TOR general test when the population was still quite small... A very vocal group of testers spent a lot of time trying to totally redesign the game. I've seen this in other tests of course but this was the most recent.
Somehow it seems people miss the point that they want us to test the game they are making. As opposed to trying to tell them the game we (or some of us) want them to make. In fact I would say there were more threads on the beta forums arguing about the game that wasn't being made (any apsect of MMO you could come up with that were not going to be in TOR) than threads discussing issues with the game we actually were testing.
You can always invite a large group of people and toss in the promotional crap when you need to load test your servers. Perhaps developers should be more active with their test community in removing testers who are only devoting time to ... the game that isn't instead of testing the one there is. It would also clean up their beta forums so people could more easily find relevant threads to their testing issues etc
Somehow it seems people miss the point that they want us to test the game they are making. As opposed to trying to tell them the game we (or some of us) want them to make.
*hangs head in shame*
But in defence of all the armchair developers out there, is this really a bad thing?
I realize that critiquing the core design in a fully realized beta is a little pointless and so is appointing yourself excutive producer. But if you have sincere high-level feedback and you haven't been told what features are written in stone, what's the point of keeping it to yourself?
To me the problem is all the previous games that were beta tested came out with the same exact bugs reported in beta. IE. Vanguard, AOC, WAR, the long list of games that need to be patched after retail and others. This long storied tradition has made people lose faith in beta testing, and see it for what it really is. A marketing ploy where feedback and bug reports are ignored or shelved for after release.
Now, whenever a dev team wants to actually get bug reports they have to deal with what the industry has created - Gamers that don't believe bug reports are taken seriously. People can only be fooled so many times before they catch on and adapt.
How do I, as a developer, convince you that beta means BETA?
Do not use the word beta at all. When asked, deny you are looking for beta testers but say that you are looking for people to really test your game. The word beta should be considered to be contaminated in the gaming industry.
Do not use the word beta at all. When asked, deny you are looking for beta testers but say that you are looking for people to really test your game. The word beta should be considered to be contaminated in the gaming industry.
Or how about players go with what the devs are looking for and actually test the product rather than just use it for a sneak preview? The players are just as complicite... at least from my experience of watching the Rift and ToR boards on here during their open betas...
Comments
Good article and I agree across the board. Beta is now a useless PR tool. Smoke and mirrors by and large. I won't even bother to sign up for betas anymore.
I certainly understand your pain. I see it all the time as I flit about looking for something interesting to test with my time. I think that Dominus had the right idea in the beginning, offering the beta up to everybody and then after that small initial period only inviting back the meaningful contributors. I think that several of the comments have the right idea of having a thorough application process to get in initially, but then that leaves developers with the potential of having to sift through a lot of applications to find the ones that are worthwhile. I think that a good idea would be to have a small test client created, and populated with a few known bugs. Some simple bugs that are easy to reproduce and somewhat obvious, and some more complex ones that are more difficult to find and / or reproduce. During the application process ask the applicant to download the little client and report any bugs they find. Perhaps presenting it as something that is currently being looked at and could use a lot more hands on testing. From this I imagine you would be able to weed out a large percentage who either did not download the client knowing that it wasn't the full game and had no interest, or did not report a single bug despite you knowing that had they played it they encountered a minimal of X number of bugs. You could effectively auto close applications that were submitted without receiving a bug report after X days. Based upon my interactions with the MMO world I would guesstimate this would allow you to be done with 50% or more of the applicants. From the rest that are left over you can judge them based upon their bug reports. Did they outline steps to reproduce the bugs? How was their documentation of the bugs? How many bugs did they find?
The potential problem of course is if it leaks out that this is how you're choosing your testers it will eventually lead to people documenting where to find all of the bugs.
Beyond the initial application process I think that more developer communication would be ideal. Dominus seemed to have a bit of it going on, but the more the better. Perhaps a forum post where the developers keep a running tally of known issues, and ideally with a small snippet about the status of their progress towards resolving the bugs or the bugs priority. It could even be something as simple as providing some slimmed down viewer access to the internal bug tracking software, allowing people to see that a bug has shown up there, has been assigned to somebody, and is being worked on. Maybe even just a weekly hour long chat in IRC, or an equivalent, going over what has been accomplished, what's on deck for being fixed, what the progress is of fixing specific things. One of the most disheartening things about being a tester can be feeling like your reports are falling upon deaf ears due to a lack of communication. Understandably it can take a bit of time out of a developers busy schedule to provide such feedback, but I think the pay offs would be worth it in the end.
OP forgot one salient point about beta testing. Those who do beta test and submit bugs, glaring problems like memory leaks and comment on bad design and are ignored.
TRUST THE COMPUTER! THE COMPUTER IS YOUR FRIEND!
Stay Alert! Trust No One! Keep Your Laser Handy!
Yellow Clearance Black Box Blues!
Ahem?
So, if some company gives you a demo or trial of an untested unfinished game, what are you going to do? Your post totally makes no sense to me...
You're still looking for the free trial...kinda proving my point...
That Guild Wars 2 login screen knocked up my wife. Must be the second coming!
It seems to me that it has to be presented as a beta. From my experience in betas, here are a few things that scream "Not Really a Beta"
1. Key giveaways. MMORPG, Massively, Fan Sites, etc. If you are giving away keys to people happen to be there to click, not a beta. That's marketing
2. Priviledged Key giveaways. IGN, Fileplanet, etc. People receiving keys probably believe they earned it and/or paid for it and therefore are getting a sneak preview.
3. No pre-screening. Register an e-mail address, get a chance at an invite. Again, screams marketing.
Things that do make it appear as if it's a legitimate beta:
1. Questionaires. Dxdiag, gaming history, etc. Feeler questsions such as what aspect of the game interests them.
2. Focused testing. Depending on the engine/coding, limit tests to limited aspects, level ranges, pvp instances, etc.
3. Small wave invites. If everyone and their brother gets in, it appears to be a stress test and/or marketing.
4. Forced feedback. Complete a quest, fill out a questionaire. Play for 30 minutes complete a questionaire. Craft something, fill out a questionaire.
While playing a game should be fun, beta should also feel like a job.
The more I age, the more I understand, and the more I wonder about the "current state of affairs," as you so eloquently pointed out in a strong active voice. What happened to the gaming industry? You are right, no one takes chances any more. Rift is WoW, COD is Battlefield, everything is the same, and freaking fantasy to boot. Ten years ago I wouldn’t have even read website things like this because frankly I was gaming, and everything else was a waste of time; of course that was before twitting face booking and self entitled thirteen year olds that occupy the masses of gamers I believe now. Beta has become a tool for advertisement, I agree. In this latest Tribes beta I recalled something about paying for something for some reason or another, I didn’t stick around long enough to find out, the game just wasn’t for me. Now you pre-order to get into so called beta. However, I think what is going on here is you forgot to relate beta to that of free-to-play, because everything is tailored to thirteen year old kids who want things for free because they have no credit card to put it on. I've waited for over 8 years for another real man game like Neocron (indie- youtube "neocron pvp"), or Planetside as a substitute. Where conflict happened for a reason, and outpost wars, territory, and resources made it happen... However, to answer your question, when you say beta, I still think beta. I take it on a case by case basis.
they dont really do that since they are pretty upfront with the state of the game and they let oyu know its a beta not a free trial
I remember when Beta's were more work than play and rather miss those days honestly. Can they be brought back?
Yes, certainly.
The problem though, is that it's probably more trouble than it's worth at this point.
The amount of vetting to get a pool of viable testers can burn up time.
The name of the game is "bug hunt", but most Beta Testers think it's " Is it fun?"
It was at one point an inroad into the game business and really isn't now. Adults , by and large, don't have the tme to devote to it and the current generation of teens has sever entitlement issues.
On to entitlement issues... AC and AC2's beta tests (specifically AC) proved that those that do get the responsibility have an impression of vested ownership that Game Devs have had to disabuse them of in order to avoid legal liability.
I think a smaller game could make it work though.. They have a more pronounced need for the bodies as opposed to these mega-groups that can throw money around more freely.
Pretty much mirrors my own thoughts. Lord knows I've sent in bug reports til my fingers bled on a few games in the past, only see a complete lack of acknowledgment or fixes occurring, even on simple things. Kind of given up these days.
A Modest Proposal for MMORPGs:
That the means of progression would not be mutually exclusive from the means of enjoyment.
I have been in countless beta's since my first, which was EQ. I can tell you on reason why beta testers do not do the job anymore. It comes right down to auto-responses to bug reports. In a recent beta I was involeved in, I had to report the same bug 8 times before I got an actual response, all the rest were computer generated, hello I hear you are having an issue, did you try this crap.. Needless to say, I had no interest in doing that repeatedly, so my bug reports were much less often.
So, as a developer, I ask you to do the same thing I ask my bank, my cable company and everyone else...stop being so automated. While it might get rid of the flow of bug reports that truly are garbage...it also deflates those that are dedicated to the task. Give us IN GAME petitions again with guides/advisors or whatever...that we can TALK to about an issue.
Keep a list of GOOD testers and call upon them again, in the next game. That in itself is a perk to do a good job. Make a rating system for your developers to RATE the bugs reported and assign scores to the testers. Give perks for good reports and an ongoing relationship in future games. Things like bug squasher as a title, would be a valuable perk in the long haul of the game.
I agree that a beta should be a testing ground not a sneek peek. But here is the problem the developers bring on themselves. Most if not all games do not launch with a trial period. I mean a free trial period that they all eventually add in there later.
No one wants to pay for and play a game that ends up tanking (this means you STO and WAR). But most people that will end up being the dedicated type want to be able to be there for day one.
I have been burned by enough games that ended up not delivering that I now don't preorder anything. That solves the problem of getting burned but its not the best solution. I WANT to try before I buy. I am the one paying the cash for the product so that should mean something right? I don't mean its owed to me or they HAVE to do it or anything like that I am just saying that would make me a satisfied customer. As it is I am not a customer at all until the game is established and I hear enough good about it to make an informed decision. So a good game will eventually get my business but it could have it so much sooner with a trial.
Since none of them have free trials at launch I will do everything I can to get into a beta these days. Don't get me wrong I test. I have submitted hundreds of bug reports and participated in untold dozens of stress tests in my time. But the real driving force behind me and I would bet most other people to include those that don't test anything at all durring a beta is the desire to see if the game will have any appeal at all. Does it have any potential or is it all hype?
Anyone else out there see my point? If I know a game is going to have a trial at launch I wont bother with the beta if I dont have the time to put into it.
Good article, and touches on why I have not beta-tested any MMORPG yet. There is no way for a game company to release a polished product without beta-testing it on a goodish number of players. It seems like closed betas are now the only beta, and "open betas" are really a demo anymore. I got the Blizzard yearly deal, so will be beta-testing MoP.
I have beta-tested other games. I made sure to report bugs or game-breaking annoyances and realized that I was there to help the devs and programmers rather than playing a full product.
It is a shame, and a loss of time to the developers and programmers when players use betas and tests to basically just get ahead. It does take time to be a good beta-tester though: bug reports, participating in the forums, number crunching, etc. The bug reports have to be detailed enough so that the bugs can be found and fixed... that isn't always so easy.
Good luck to debuggers... it is not a job I would actually like doing (looking at someone else's code... eep!).
Playing MUDs and MMOs since 1994.
This post makes valid points, but also makes me think that the perfect beta-tester would be one who is not hyped up about a game. I know that fans tend to test, but that is sort of a bad thing in the sense that they get too emotionally wrapped up in the game. It is probably better overall to be a fairly neutral tester who does not necessarily go into the beta all hyped up.
I guess the industry's thought process on why there are no demos or trials of most MMOs is that with a long-term game the game publishers probably expect $50-60 to be a relatively cheap entry fee, seeing as some of that initial money usually goes into the first month of subscription (makes it $35-45 which is on average pretty cheap for an AAA title that cost millions to develope). Perhaps the "free month" industry standard is seen as the "demo", because even though you had to pay for entry, the real money they make is from subscriptions and keeping people interested in the game (and more and more often through cash shops too).
They hope and wait for gamers to hype up their game once it launches so that it will attract new players and it usually takes several months before a free trial is offered (usually when the game's box starts going down in price in offline stores). I agree so very much with the above poster that free trials are necessary from launch on.
It would avoid some rather unpleasant scenarios of some pre-order "beta-testers" getting to max level really quickly and then whining about not having anything to do within a couple days of launch, but also would slow down the rate of exploiters cropping up in the game. It would also avoid the somewhat risky gamble of waiting for word-of-mouth to kick in. The latter worked in the old, pre-WoW days because the MMO community was smaller and there were fewer games to choose from, but I think these days, the industry cannot rely on that anymore.
Playing MUDs and MMOs since 1994.
Good article.
Personaly I like playing in Betas and trying to break the game. Anything from a spelling mistake and bad grammer all the way to game breaking bugs.
Even once released I am still posting bug reports. Unfortunatly I have sent in more bug reports for SWTOR than any other release since it went live, and most of these bugs could of been sorted in closed beta.
Interessting whine. But I'm afraid it's one sided.
If you leave hype aside beta does serve a good purpose if enough people join. It gives devs an opportunity to check the following:
- Performance: A small "elite" group of testers can only focus on functionality of the game. Never on performance and load. Never! Just ask yourself what good is a perfectly polished game that crashes after 100 people log in. And lets not forget ping (lag). A selected group of testers would probably be local, maybe even hired by the company. But if you pass enough beta keys around and you get people loging in from various locations and countries using differens isps then you have a good chance catching problems you never dream existed. This is what performance and load are about. There are also subsections like login server performance, or authentication server performance (usually separate from the main game server) but this is more or less all in same direction. The more people you get the more data you can collect about how your game behaves in extreme cases.
- Functional bugs: Take a good look at your calculator and then try to think about number of options one can use to enter a simple equation. Amazing how many "test cases" can you get out of a simple calculator, right? Now take a look at any game and think about nuber of options a player has to go through, starting with login, avatar creation screen, naming, tutorial... Bazillion gazillion combinations in an average game. This can not be covered by a test team or by small group of 1337 beta testers. The more users you get on your server the bigger the probability is that one of them will discover a way to break the game. It is a probability game, nothing more, nothing less.
- "Idiot users": Let's be honest. All devs can focus on is the positives! After more then a decade in software development this is an axiom in my opinion. A fact! No developer ever assumes that some users might even think about using his application (a game in this case) to do something that was never planned with it. It's like trying to water the plants with a chainsaw or trying to light a fire with a hammer. Maybe it can be done but results would probably be devastating for plants and trees. Same with "idiot users", who can behave "in the wrong way" simply because their assumptions are wrong or their knowledge of gaming and technology is very poor. It's like the story of a lady that used CD tray as a cup holder or of another user that used a mouse by trying to step on it. Small group of beta testers would NEVER have anyone in that group. And if game's error handling routines are poor and an "idiot" user keeps doing something silly that constantly crashes the server then EVERYONE will suffer and not just him.
- "1337 users": Opposite of "idiots". Those guys can hack into your game and cause havoc for anyone around. Botting, scripting, even using APIs to make your own apps and screw the game in your advantage. Good luck trying to find all security holes by letting 100 loyal testers do the testing. And how many games you know that died due to hacking and boting?
There are other things that could be said in gavor of current beta practices. But the way current gaming industry is set up there is NO WAY around how beta is done. Pass quantillion keys and hope for the best. Yes there is also greed involved when companies release unfinished products since all of the guys in the industry want to make money (I would be worried if any of them dont). Greed also plays a big role when you don't want to have proper test teams, which is not problem since you can always find million souls just waiting to log in and try something new. So public gets something in return for early access.
All of the above is "imho". Yet my "ho" is based on a decade of software testing experience and more then 2 decades of gaming experience so I think I know what I'm talking about.:)
No fate but what we make, so make me a ham sandwich please.
Right after reading this i ended up on the TERA beta signup page.
http://tera.enmasse.com/news/posts/beta-test-schedule
After reading that, noone can say you are wrong
... not a word about "improving the game" or " finding bugs" ... what has the emphasis is "Eager to get your hands on TERA?"
Spiider there are many phases of beta I do not think Sanya meant stress test beta which is basically the only form of beta these days. Most of the other testing is done with a group of developer friends and insiders. That use to not be the case. Now beta is used more as a marketing tool that an actual beta test, the game at the stage most people get into beta is almost completely done and no major design changes can be made.
Sanya how would you convince me? Here you go in 4 easy steps…..
Never use beta as a promotion tool splashed all over the web page, pre-order and get a spot in the “beta test”
Do not do a twitter/facebook contest to get into “beta”. Matter of fact; do not use Twitter or Facebook at all for this. If you are looking for real beta testers those are the LAST places you should going.
Be very clear what stage the game is in, what the beta tester can expect from the game in its current state and very specifically what testing you are doing and feed back you are requiring from them. Yes I said it REQUIRING from them. This is not a play test make sure that is completely understood.
Once you do let us in, do not then turn around and tell us we do not know what we are talking when we give specific feedback you do not want to hear. If a very large majority of the real beta testers tells you something in very specific detail without trolling is VERY broken in your game, listen to them do not threaten them or out right dismiss them.
I will use my last experience beta testing in a game called WAR. Just about every real beta tester told Mythic that bright wizards were very over powered and badly broken in pvp and we were basically told to shut up, we have metrics, completely dismissed, and then threatened with expulsion from beta when we did not let it go(some of us got kicked).
They proceeded to launch the game with the class completely unchanged from beta and we all saw how that went, really frustrating to see something that should have been addressed in beta go live, when we told them exactly what would happen in detail if they did not adjust bright wizards, never mind the stuck on every rock in the game issue.……sorry I digress……
And still servers went down which only confirms my theory that this is the right way to go. They let bazillion players try to log in which gave them perfect load test. Beta test successful. The facts that no one could play is irrelevant for devs. They will sell it to you as "early sneak peak" when in fact it's "early load test".
No fate but what we make, so make me a ham sandwich please.
I'm not really sure a basic language port sneak peak really constitutes a beta.
Each company runs betas differently. The server load/mass invite beta is generally a final stage of beta. Recent betas like Rift and SWToR were sneak peaks. Hell in SWToR they even disabled the ingame bug reporting.
I can follow that logic ... but this is not one test .. it's for all (5) closed beta tests.
I decided to come back to this thread to add something.
Beyond the promontional use of beta and the various points made in the article. There is another common thing that pops up in betas now.
Early on in the TOR general test when the population was still quite small... A very vocal group of testers spent a lot of time trying to totally redesign the game. I've seen this in other tests of course but this was the most recent.
Somehow it seems people miss the point that they want us to test the game they are making. As opposed to trying to tell them the game we (or some of us) want them to make. In fact I would say there were more threads on the beta forums arguing about the game that wasn't being made (any apsect of MMO you could come up with that were not going to be in TOR) than threads discussing issues with the game we actually were testing.
You can always invite a large group of people and toss in the promotional crap when you need to load test your servers. Perhaps developers should be more active with their test community in removing testers who are only devoting time to ... the game that isn't instead of testing the one there is. It would also clean up their beta forums so people could more easily find relevant threads to their testing issues etc
*hangs head in shame*
But in defence of all the armchair developers out there, is this really a bad thing?
I realize that critiquing the core design in a fully realized beta is a little pointless and so is appointing yourself excutive producer. But if you have sincere high-level feedback and you haven't been told what features are written in stone, what's the point of keeping it to yourself?
To me the problem is all the previous games that were beta tested came out with the same exact bugs reported in beta. IE. Vanguard, AOC, WAR, the long list of games that need to be patched after retail and others. This long storied tradition has made people lose faith in beta testing, and see it for what it really is. A marketing ploy where feedback and bug reports are ignored or shelved for after release.
Now, whenever a dev team wants to actually get bug reports they have to deal with what the industry has created - Gamers that don't believe bug reports are taken seriously. People can only be fooled so many times before they catch on and adapt.
Do not use the word beta at all. When asked, deny you are looking for beta testers but say that you are looking for people to really test your game. The word beta should be considered to be contaminated in the gaming industry.
It takes one to know one.
Or how about players go with what the devs are looking for and actually test the product rather than just use it for a sneak preview? The players are just as complicite... at least from my experience of watching the Rift and ToR boards on here during their open betas...
Playing MUDs and MMOs since 1994.