How do I, as a developer, convince you that beta means BETA?
Do not use the word beta at all. When asked, deny you are looking for beta testers but say that you are looking for people to really test your game. The word beta should be considered to be contaminated in the gaming industry.
Or how about players go with what the devs are looking for and actually test the product rather than just use it for a sneak preview? The players are just as complicite... at least from my experience of watching the Rift and ToR boards on here during their open betas...
That sort of behaviour was clearly triggered by those companies themselves that made it very clear people were not in beta for actual testing. People really interested in testing got burned/disappointed, and guess who is left to visit the test servers now: people that only want sneak peaks of what is to come. There are still alot of people out there that actually want to test, but not so much when they are not really needed, but still invited into beta or to live test servers or whatever attracts the sneak peak crowd.
you could call it alpha instead of beta, have a stiff application/selection process, and make clear that continued participation is contingent upon not only time spent testing but also constructive feedback (choose your medium: email, forums, surveys, bug reports, irc chats, a combination of all of those etc.).
And then enforce that rule.
The whine from people being kicked would also generate free publicity.
I honestly think the majority of testers who want to help test a game (in the old-fashioned way, knowing that things will be borked and not work properly, being willing to repeat action x thirty times for a solid repro, roaming the world looking for spots to glitch or fall through, etc.) would not have an issue with any of that.
They'd especially not have an issue with a dev team that are engaged and responsive with their testers in return. Indie developers usually rock in that respect.
Marketing betas have hurt the testing process terribly, and have hurt the quality of released products even more.
Trouble is, with them as the standard it's hard as heck to find old-school testers wiling to put in the donkey-work to help improve a game. Everyone's a tester nowadays, and the word itself seems to mean different things depending on who you ask.
I guess it's down to companies to outline what testing means with regard to testing their game, and then stick by their own definition.
- Support your testers in test methods and improving the quality of bug reports
- Have different testing groups to focus on areas (combat/crafts/pvp/hacking/exploiting/griefing/etc..)
- Evaluate the effectiveness of your testers and replace them where needed.
- Reward your valuable testers.
I think there are many ways to get good use out of your testers, but the key is to keep the group manageable and actually manage them like they were your own employees.
Regarding the load performance test ... i understand their need, but in godsname also call it a load test then.
Regarding the use of beta as marketing tool ... why don't you just release a demo 2 weeks before release ... that always worked best for me.
When a Preview Tester crashes the game, they curse and rant on the boards, generating bad buzz.
When a Beta Tester crashes the game, they cheer and attempt duplication, possibly making the game better in the process.
If the Corporate string pullers want the "Beta Test" to be a Preview Test, then they need a new name for the real bug-squashing test that makes it sound unappealing to the average gamer, like the Crash Test, then wait until Crash Test is done, and the only things left are server stress and such. Then announce the "Beta Test".
Well the definition of "beta" has changed a bit. Beta now means sneak-peek, or pre-order bonuses or at the very least a head start with your character so you can out level the competition.
I'm not going to rant against "The Man" or any of that crap. Companies are built to make money.
I guess being selective about not only who is invited into initial testing but who to keep. If you have a tester playing the game for three weeks and they haven't uttered an ounce of constructive criticism then give their spot to some one who will.
Make the testers feel as though they are a part of something. That this game in some small way is their baby to. It's their game to.
Today's beta crowd is vastly different. Most of them expect a sneak a peak. Gosh knows when I was in the stress test for swtor all you would read in the forums was folks going I cant log in, and they obviously did not understand the meaning of a stress test.
When I run in beta I try to break things and send in reports, but most folks now days expect to play the sneak peak.
I have to say most beta's nowdays are an advertisment at best not an actual beta.
Being in the firefall beta I suppose its all about how you view it. There's no denying the moment you load the program , that it, thats your first impression. Its human nature. now as somone who been beta testing since the late 90s I have to say one major mistake I always see. People think Beta's are 100% Demo's Like I said above the moment the game start yes you will have a first impression but like all first impressions its not represeantive to the big picture.
People think of I'll just silently play through this beta and assume they know whats broken. That is the WORST way to be in any gamebeta. Not only because your not contributing to making the game better, but you have the gall to assume the Dev;s know every little issue. Developers are ALOT different than players. They can get tunnel vision quickly and lost sight of other things. Thats why as a player you are there to remind them of the little things. A texture missing on a map somewhere. a gltich with mob ai. All the stuf that would immidiately stand out to you A dev may miss.
Yes you can have a first impression of a beta. dont let anyone tell you that you cant. However DO NOT view beta's as final Demos. You are there to help the devs make the game solid before it even hits the shelves.
Totally agree with the article.. a Beta test is just that a test. Its there to find as many bugs as possible before they game is released. Also then towards then end of beta testing you usually get at on of people invited for the test to make sure the servers can handle the load..
Over the last year I have tested qutie a few games and usually straight away you get people demanding compensation because they cant play..
For example the recent Tera EU sneak peak had issues from the start. basically straight away you had someone demanding to be compensated..
That sort of behaviour was clearly triggered by those companies themselves that made it very clear people were not in beta for actual testing. People really interested in testing got burned/disappointed, and guess who is left to visit the test servers now: people that only want sneak peaks of what is to come. There are still alot of people out there that actually want to test, but not so much when they are not really needed, but still invited into beta or to live test servers or whatever attracts the sneak peak crowd.
I guess I am old-fashioned then. To me a beta is a test period (and this is not exclusive to the gaming industry). If I am invited into a beta, I am there to test it. It is not exclusively on the shoulders of the devs that this preview tendancy has happened.
If the devs only treat their betas like a trial preview, then it tells me their quality department has serious flaws and it is best not to end up investing in the final product because they do not employ the industry standard of testing whatever they release.
Sanya you said "an MMO can be flawlessly designed with intricacies like a mechanical Swiss watch and still suck the chrome off a bumper when those intricacies collide with a large number of human beings. "
You seem to be saying that something that "sucks the chrome off a bumper" is a bad thing when it is actually a very good thing.
Part of the problem isn't just the players, it is the way beta is managed by the development team as well. Most of the betas I have been invited to over the last few years have said, "Go ahead and play the game and see what doesn't work." This is very open-ended, non-focused testing and it is very easy for players to assume systems and mechanics are largely complete.
Years prior, I remember entering betas where they were conducted in phases which focused on functional testing of certain, specific aspects of the game. Sometimes this limited individuals to a certain race/class combination, sometimes a specific level range. This gave the feeling that the development team understood where certain systems were but needed dedicated feedback on other areas. This created a much more concentrated experience, and not just of the first 1-20 levels (which seems to be the common beta restriction today).
Improve the structure of your beta, and you will improve the feedback and buy-in of the players involved in it.
More than a few games have had successful small betas and then fallen on their face at launch because they never did any stress testing. A proper stress test takes thousands and possibly tens of thousands of players online simultaneously. That is really only achieved in large scale open beta.
Good article and I agree across the board. Beta is now a useless PR tool. Smoke and mirrors by and large. I won't even bother to sign up for betas anymore.
Most of the games now are so simplistic they in reality have no complex systems that really need testing. The genre is pretty much dead for anyone with more than an IQ of 1.
________________________________________________________ Sorcery must persist, the future is the Citadel
My simple response to this is that you don't invite people randomly if you want testers.
I've been in MMO betas since as far back as UO, but these days because of the random invites I rarely get in until a few weeks before launch. I'd happily test and accept the beta for what it is, if I got invited.
Well, that's all they are today, no more real testing taking place by time they let folks come in. (stress test is probably a more accurate term).
I mean look at all the open beta testing SWTOR launched with and yet it managed to launch with some issues regardless. (though I blame that mostly on a rushed launch)
Just trying to live long enough to play a new, released MMORPG, playing New Worlds atm
Fools find no pleasure in understanding but delight in airing their own opinions. Pvbs 18:2, NIV
Don't just play games, inhabit virtual worlds™
"This is the most intelligent, well qualified and articulate response to a post I have ever seen on these forums. It's a shame most people here won't have the attention span to read past the second line." - Anon
Here is how I would execute a beta test for a game I'm developing. I'd canvas for beta testers as normal, but I wouldn't have beta forums. At all. Beta Testers would be sorted into teams, or cells. Some times they'd be given no instruction, and allowed just to play. Other times each cell would be given a task to test.
This would help mitigate the echo chamber effect that beta forums suffer from. You get the most vocal minority posting at times, thus influencing decisions of others.
Also, the cell approach would reduce the effect of those in the beta who are playing for a peek. You would assign cells only partial access to the game, but by having many cells, you could cover the entire expanse of the game.
OP forgot one salient point about beta testing. Those who do beta test and submit bugs, glaring problems like memory leaks and comment on bad design and are ignored.
Me being one of them .. whenever I can i'll beta test a game and i'm used to bugs that are reported making there way into the release but the saddest thing is the players in beta's these days :-(
You get a good deal of them being just there to try the game out, the amount of questions like 'will my character be wiped' and then the moaning and complaining when told ofcourse it would be.
Funnily they always know an mmo that they beta'd and their toon wasn't wiped .. out of every one i've done (and this is going back a bit .. lol) I haven't known one to do that.
As someone above said, people are using beta's now as free trial's because they don't want to pay for the game to find out if they would like it or wait for the free trials to be released.
Instead they'll just pose as testers and some will end up slagging the game off even though they were testing an unfinished product (ok .. mmo's are basically never finished but you get what i mean)
I was in the beta for vanguard for an entire YEAR... I also rose to being the tailoring crafter liasion.
Personally I think beta should be earned not bought. They should call those late beta megatests what they really are "Stress and Tuning Pre-Play."
A real beta gives you actual access to the devs or at least one layer removed from the devs where your feedback is actually worthwhile. They play with the people -- I remember Aradunes thestwian wanger... This kidding around takes some of the edge off...
A real beta test should REQUIRE a commitment to finding bugs. If you are just playing the game as a preview and do not find anything you should be bumped for people who are more willing to spend the time to actually make the game better.
I am one of the people who actually find real bugs even in a tuning pre-play -- I know I found my share in rift and that was thought of as a pretty bug-free experience by most. You have to mess with the games a little sometimes to make them give up their mistakes. Try doing things most people wont do but some people could do.
With the lengthening of the development cycles for games, I do not see why they couldnt have medium sized (say 3 servers) beta tests significantly earlier in the development cycle than we see today. Let people see the game at a point where they can actually make a difference and find more bugs than geometry issues and typos. Encourage people to report on things that are unbalancing as well..
These days a lot of people will actually HIDE an exploit they find during beta rather than reporting it, hoping to exploit it later. Finding an exploit class bug and not reporting it should be worth a ban from not only that beta but any other beta run by the company in the future. Looking for and reporting exploit-class bugs is one of the most important thing you can do as a beta-tester. I know I found a few in games.
I also think there should be rewards for early beta testing. Or even late beta testing if you manage to distinguish yourself. I mean if you find and turn in an exploit-class bug you should at the very least get a free copy of the game and an attaboy. They should keep lists of people who have found and report even things like typos etc so they get into future beta tests earlier rather than later.
If when the game released you got some in game benefit for finding that typo in the second to last beta phase that somehow made it through that far and then was fixed before release, people would treat the betas with more respect.
I also think there should be rewards for early beta testing. Or even late beta testing if you manage to distinguish yourself. I mean if you find and turn in an exploit-class bug you should at the very least get a free copy of the game and an attaboy. They should keep lists of people who have found and report even things like typos etc so they get into future beta tests earlier rather than later.
If when the game released you got some in game benefit for finding that typo in the second to last beta phase that somehow made it through that far and then was fixed before release, people would treat the betas with more respect.
This is actually a pretty good idea. Atm there are rewards for players who beta "test" and find an exploit and don't report it... they can use that exploit at release and gain xp/money from it.
Perhaps giving free copies of the game away is abit much though, maybe giving ingame items that scale up depending on the amount or severity of the bug/s found would be incentive enough. Nothing that would give a massive advvantage to a player, but maybe smth like:
LVL1: title
LVL2: LVL1 + pet
LVL3: LVL2 + mount
LVL4: LVL3 + ingame item to boost xp for set duration
That being said, its a sad reflection of the current state of many gamers that rewards for finding and reporting bugs are even being mentioned
I also think there should be rewards for early beta testing. Or even late beta testing if you manage to distinguish yourself. I mean if you find and turn in an exploit-class bug you should at the very least get a free copy of the game and an attaboy. They should keep lists of people who have found and report even things like typos etc so they get into future beta tests earlier rather than later.
If when the game released you got some in game benefit for finding that typo in the second to last beta phase that somehow made it through that far and then was fixed before release, people would treat the betas with more respect.
This is actually a pretty good idea. Atm there are rewards for players who beta "test" and find an exploit and don't report it... they can use that exploit at release and gain xp/money from it.
Perhaps giving free copies of the game away is abit much though, maybe giving ingame items that scale up depending on the amount or severity of the bug/s found would be incentive enough. Nothing that would give a massive advvantage to a player, but maybe smth like:
LVL1: title
LVL2: LVL1 + pet
LVL3: LVL2 + mount
LVL4: LVL3 + ingame item to boost xp for set duration
That being said, its a sad reflection of the current state of many gamers that rewards for finding and reporting bugs are even being mentioned
Considering that people shouldnt be finding tons of exploitable level bugs, giving a free copy of the game is not a large issue -- especially because a free digital copy really doesnt cost the company anything. I say this because lets say you played a game in beta and you found a major issue in the game and you were rewarded with a free copy.... The odds that you are going to stay with that game for a few months is MUCH higher than if you reported a major bug and got the typical "We appreciate your blah blah" form letter.
It is also a form of viral publicity. Instead of potentially having someone out there saying "Yeah the game is a buggy mess," you have someone out there saying "This company cares enough about fixing their game that they gave me a free copy." Lastly that the possibility is out there would make more bugs be reported which is a very good thing.
>>How do I, as a developer, convince you that beta means BETA?
Make Beta Testing unpleasent.
Drive away the hordes that are just there for a 'free game' or a 'head start'.
Do this by limiting the options of the testers.
Give your 'testers' limitied scope within the game.
For example:
Some are limited to just character generation. They can generate a character...and another...and another...but they cannot actually play.
Some can play the tutorial. They are assigned a character (pre created) and they can play the tutorial levels and zones... then they can 're-roll' and do that again.... and again... and again...
Have some that can play only level 1-10...some that can play 11-20... some that can only play certain zones... oh...and did I mention the weekly character wipes?
The reward is that for those who stick around... find and report bugs... test the boring stuff...actually contribute in a meaningful way on the (beta) forums... after a while they get greater 'freedom'. They get to Create their own characters and level them up all the way. Maybe they are still locked to a class or faction (that you want tested) but it's better.
And the ones that really stick it out and are of value - get free run in the end. But it takes time. It takes HARD WORK.
Maybe keep a register of the best testers - maybe it's something the industry as a whole needs to do?
Perhaps a forum run by developers or even a sub forum on a well known site like MMORPG.com where members can only access it if they can produce a log in code supplied by appreciative Devs?
And membership lapses after a few months.
That way you have a pool of real testers available that you know have been of real value to other devs. Not fool proof - but better than the mouth breathers that you get by giving away 'keys'.
If you want to run 'pre release promotions' then call it that. NOT a "Beta Test"
If you want to run a 'stress test to failure' then call it that. NOT a "Beta Test"
And at the end of a Beta Test...wipe all characters without exception. Make it clear this will happen. Those that choose to leave as a result are not 'testers' anyway.
Note: I have 'Beta Tested' several games now. I actually report bugs and set out to break things. In one MMO I managed to break the Tutorial less than a fortnight prior to retail release. I was left wondering exacly what the other 'testers' had done for the previous months?
I have been personally invited back (they sent me an email) by one company to test an expansion. In that game I had been extremely critical of the game in the Beta forum but I had always given a reason and proposed solutions. I was attacked by many Fanbois for my comments and most of my comments resulted in no change to the game - BUT - apparently the Devs liked having someone around who was not a 'yes man'?
Oh No Sanya, you are starting to sound like me! -)
I can still remember when Devs required actual formatted reports to be sent via email when testing. Or when characters were wiped every 3 days. Or the server(s) were reset every 25 minutes. And us testers loved it all.
Testing DAoC was like being included in a large family. Everyone listened to each other and testers helped the Devs and each other. The bonding community made you care about the project. I found this to be true when testing F&F for LOTRO:SoA... especially with Tiggs and her Extra Crispy guild involved. Testers felt like they were assets to the projects...not just another warm body to run around in the game while background-game-logging programs tracked our playstyles.
Testing has gone from individual input to spreadsheet input. We are now ants keeping the hive humming while our actions are being tracked and quantified in automated reporting tools. And testers are no longer given the tools to help report locations and issues. Recent games I've tested (and one I am currently testing) don't have a /loc command, no way to take screenshot in-game (other than using Fraps), no interaction with the Devs/Mods/CSRs, bug reports only receive auto-generated replies (Tabula Rasa had a great bug reporting system!), and in-game help tickets are ignored & deleted by GMs with no reasons given.
LOTRO has an invite-only test server where new content is tested prior to release to the public test server. Turbine Devs interact directly with these testers to break new systems, flesh out new content, listen to ideas, and focus-test team-based areas. I am sure other game publishers & developers have servers like Turbine's but just don't publish the fact.
I sincerely hope game companies return to the idea that testers are there to help you create a stable enjoyable enviroment and are assets who want to invest their time FREE OF CHARGE to help the projects.... and not just another person who might buy the game if they get pre-release exposure.
TQQdles™,
Dolnor Numbwit
aka Noldor Numbwit
Eternal Newbie
P.s. I still wear the DAoC Road Trip T-shirt now and then! -)
1) Define exactly what you are looking to do, both for yourselves and your audience.
2) Don't call a stress test or prerelease test drive a 'beta'.
3) Pick your first rounds of beta testers carefully. It isn't that hard... just look for the folks on your forums who take the time to wrote out thoughtful, logical topics or replies. Invite them. Those people are in turn likely to know people who wouldn't be a waste of time. Let your proven testers invite 5 more people. Simple.
4) Make certain that the bug handling process is as transparent as possible. Feedback to the community is MANDATORY. As others have explained earlier in this thread, nothing discredits developers and a company more than having issues reported, ignored, then left unresolved for long periods accompanied by a communications blackout. That type of behavior is common, and sends a tremendously negative set of messages to the community.
Comments
That sort of behaviour was clearly triggered by those companies themselves that made it very clear people were not in beta for actual testing. People really interested in testing got burned/disappointed, and guess who is left to visit the test servers now: people that only want sneak peaks of what is to come. There are still alot of people out there that actually want to test, but not so much when they are not really needed, but still invited into beta or to live test servers or whatever attracts the sneak peak crowd.
It takes one to know one.
Hmmm,
you could call it alpha instead of beta, have a stiff application/selection process, and make clear that continued participation is contingent upon not only time spent testing but also constructive feedback (choose your medium: email, forums, surveys, bug reports, irc chats, a combination of all of those etc.).
And then enforce that rule.
The whine from people being kicked would also generate free publicity.
I honestly think the majority of testers who want to help test a game (in the old-fashioned way, knowing that things will be borked and not work properly, being willing to repeat action x thirty times for a solid repro, roaming the world looking for spots to glitch or fall through, etc.) would not have an issue with any of that.
They'd especially not have an issue with a dev team that are engaged and responsive with their testers in return. Indie developers usually rock in that respect.
Marketing betas have hurt the testing process terribly, and have hurt the quality of released products even more.
Trouble is, with them as the standard it's hard as heck to find old-school testers wiling to put in the donkey-work to help improve a game. Everyone's a tester nowadays, and the word itself seems to mean different things depending on who you ask.
I guess it's down to companies to outline what testing means with regard to testing their game, and then stick by their own definition.
You actually need to MANAGE your testers.
- Define use and abuse cases
- Track activity of testers
- Support your testers in test methods and improving the quality of bug reports
- Have different testing groups to focus on areas (combat/crafts/pvp/hacking/exploiting/griefing/etc..)
- Evaluate the effectiveness of your testers and replace them where needed.
- Reward your valuable testers.
I think there are many ways to get good use out of your testers, but the key is to keep the group manageable and actually manage them like they were your own employees.
Regarding the load performance test ... i understand their need, but in godsname also call it a load test then.
Regarding the use of beta as marketing tool ... why don't you just release a demo 2 weeks before release ... that always worked best for me.
When a Preview Tester crashes the game, they curse and rant on the boards, generating bad buzz.
When a Beta Tester crashes the game, they cheer and attempt duplication, possibly making the game better in the process.
If the Corporate string pullers want the "Beta Test" to be a Preview Test, then they need a new name for the real bug-squashing test that makes it sound unappealing to the average gamer, like the Crash Test, then wait until Crash Test is done, and the only things left are server stress and such. Then announce the "Beta Test".
Well the definition of "beta" has changed a bit. Beta now means sneak-peek, or pre-order bonuses or at the very least a head start with your character so you can out level the competition.
I'm not going to rant against "The Man" or any of that crap. Companies are built to make money.
I guess being selective about not only who is invited into initial testing but who to keep. If you have a tester playing the game for three weeks and they haven't uttered an ounce of constructive criticism then give their spot to some one who will.
Make the testers feel as though they are a part of something. That this game in some small way is their baby to. It's their game to.
It takes a village to raise a child.
Today's beta crowd is vastly different. Most of them expect a sneak a peak. Gosh knows when I was in the stress test for swtor all you would read in the forums was folks going I cant log in, and they obviously did not understand the meaning of a stress test.
When I run in beta I try to break things and send in reports, but most folks now days expect to play the sneak peak.
I have to say most beta's nowdays are an advertisment at best not an actual beta.
Being in the firefall beta I suppose its all about how you view it. There's no denying the moment you load the program , that it, thats your first impression. Its human nature. now as somone who been beta testing since the late 90s I have to say one major mistake I always see. People think Beta's are 100% Demo's Like I said above the moment the game start yes you will have a first impression but like all first impressions its not represeantive to the big picture.
People think of I'll just silently play through this beta and assume they know whats broken. That is the WORST way to be in any gamebeta. Not only because your not contributing to making the game better, but you have the gall to assume the Dev;s know every little issue. Developers are ALOT different than players. They can get tunnel vision quickly and lost sight of other things. Thats why as a player you are there to remind them of the little things. A texture missing on a map somewhere. a gltich with mob ai. All the stuf that would immidiately stand out to you A dev may miss.
Yes you can have a first impression of a beta. dont let anyone tell you that you cant. However DO NOT view beta's as final Demos. You are there to help the devs make the game solid before it even hits the shelves.
It..Burns..
Totally agree with the article.. a Beta test is just that a test. Its there to find as many bugs as possible before they game is released. Also then towards then end of beta testing you usually get at on of people invited for the test to make sure the servers can handle the load..
Over the last year I have tested qutie a few games and usually straight away you get people demanding compensation because they cant play..
For example the recent Tera EU sneak peak had issues from the start. basically straight away you had someone demanding to be compensated..
https://forum.tera-europe.com/showthread.php?t=28873
I dont know why he thought he had the right to anythign but there we go..
I guess I am old-fashioned then. To me a beta is a test period (and this is not exclusive to the gaming industry). If I am invited into a beta, I am there to test it. It is not exclusively on the shoulders of the devs that this preview tendancy has happened.
If the devs only treat their betas like a trial preview, then it tells me their quality department has serious flaws and it is best not to end up investing in the final product because they do not employ the industry standard of testing whatever they release.
Playing MUDs and MMOs since 1994.
Sanya you said "an MMO can be flawlessly designed with intricacies like a mechanical Swiss watch and still suck the chrome off a bumper when those intricacies collide with a large number of human beings. "
You seem to be saying that something that "sucks the chrome off a bumper" is a bad thing when it is actually a very good thing.
Part of the problem isn't just the players, it is the way beta is managed by the development team as well. Most of the betas I have been invited to over the last few years have said, "Go ahead and play the game and see what doesn't work." This is very open-ended, non-focused testing and it is very easy for players to assume systems and mechanics are largely complete.
Years prior, I remember entering betas where they were conducted in phases which focused on functional testing of certain, specific aspects of the game. Sometimes this limited individuals to a certain race/class combination, sometimes a specific level range. This gave the feeling that the development team understood where certain systems were but needed dedicated feedback on other areas. This created a much more concentrated experience, and not just of the first 1-20 levels (which seems to be the common beta restriction today).
Improve the structure of your beta, and you will improve the feedback and buy-in of the players involved in it.
More than a few games have had successful small betas and then fallen on their face at launch because they never did any stress testing. A proper stress test takes thousands and possibly tens of thousands of players online simultaneously. That is really only achieved in large scale open beta.
Most of the games now are so simplistic they in reality have no complex systems that really need testing. The genre is pretty much dead for anyone with more than an IQ of 1.
________________________________________________________
Sorcery must persist, the future is the Citadel
My simple response to this is that you don't invite people randomly if you want testers.
I've been in MMO betas since as far back as UO, but these days because of the random invites I rarely get in until a few weeks before launch. I'd happily test and accept the beta for what it is, if I got invited.
But hey, random invites...
Open beta = Free trial right?
Well, that's all they are today, no more real testing taking place by time they let folks come in. (stress test is probably a more accurate term).
I mean look at all the open beta testing SWTOR launched with and yet it managed to launch with some issues regardless. (though I blame that mostly on a rushed launch)
"True friends stab you in the front." | Oscar Wilde
"I need to finish" - Christian Wolff: The Accountant
Just trying to live long enough to play a new, released MMORPG, playing New Worlds atm
Fools find no pleasure in understanding but delight in airing their own opinions. Pvbs 18:2, NIV
Don't just play games, inhabit virtual worlds™
"This is the most intelligent, well qualified and articulate response to a post I have ever seen on these forums. It's a shame most people here won't have the attention span to read past the second line." - Anon
Here is how I would execute a beta test for a game I'm developing. I'd canvas for beta testers as normal, but I wouldn't have beta forums. At all. Beta Testers would be sorted into teams, or cells. Some times they'd be given no instruction, and allowed just to play. Other times each cell would be given a task to test.
This would help mitigate the echo chamber effect that beta forums suffer from. You get the most vocal minority posting at times, thus influencing decisions of others.
Also, the cell approach would reduce the effect of those in the beta who are playing for a peek. You would assign cells only partial access to the game, but by having many cells, you could cover the entire expanse of the game.
That link was...pathetic.
One of the biggest fail threads I have seen in awhile.
Wow.
I was in the beta for vanguard for an entire YEAR... I also rose to being the tailoring crafter liasion.
Personally I think beta should be earned not bought. They should call those late beta megatests what they really are "Stress and Tuning Pre-Play."
A real beta gives you actual access to the devs or at least one layer removed from the devs where your feedback is actually worthwhile. They play with the people -- I remember Aradunes thestwian wanger... This kidding around takes some of the edge off...
A real beta test should REQUIRE a commitment to finding bugs. If you are just playing the game as a preview and do not find anything you should be bumped for people who are more willing to spend the time to actually make the game better.
I am one of the people who actually find real bugs even in a tuning pre-play -- I know I found my share in rift and that was thought of as a pretty bug-free experience by most. You have to mess with the games a little sometimes to make them give up their mistakes. Try doing things most people wont do but some people could do.
With the lengthening of the development cycles for games, I do not see why they couldnt have medium sized (say 3 servers) beta tests significantly earlier in the development cycle than we see today. Let people see the game at a point where they can actually make a difference and find more bugs than geometry issues and typos. Encourage people to report on things that are unbalancing as well..
These days a lot of people will actually HIDE an exploit they find during beta rather than reporting it, hoping to exploit it later. Finding an exploit class bug and not reporting it should be worth a ban from not only that beta but any other beta run by the company in the future. Looking for and reporting exploit-class bugs is one of the most important thing you can do as a beta-tester. I know I found a few in games.
I also think there should be rewards for early beta testing. Or even late beta testing if you manage to distinguish yourself. I mean if you find and turn in an exploit-class bug you should at the very least get a free copy of the game and an attaboy. They should keep lists of people who have found and report even things like typos etc so they get into future beta tests earlier rather than later.
If when the game released you got some in game benefit for finding that typo in the second to last beta phase that somehow made it through that far and then was fixed before release, people would treat the betas with more respect.
This is actually a pretty good idea. Atm there are rewards for players who beta "test" and find an exploit and don't report it... they can use that exploit at release and gain xp/money from it.
Perhaps giving free copies of the game away is abit much though, maybe giving ingame items that scale up depending on the amount or severity of the bug/s found would be incentive enough. Nothing that would give a massive advvantage to a player, but maybe smth like:
LVL1: title
LVL2: LVL1 + pet
LVL3: LVL2 + mount
LVL4: LVL3 + ingame item to boost xp for set duration
That being said, its a sad reflection of the current state of many gamers that rewards for finding and reporting bugs are even being mentioned
Considering that people shouldnt be finding tons of exploitable level bugs, giving a free copy of the game is not a large issue -- especially because a free digital copy really doesnt cost the company anything. I say this because lets say you played a game in beta and you found a major issue in the game and you were rewarded with a free copy.... The odds that you are going to stay with that game for a few months is MUCH higher than if you reported a major bug and got the typical "We appreciate your blah blah" form letter.
It is also a form of viral publicity. Instead of potentially having someone out there saying "Yeah the game is a buggy mess," you have someone out there saying "This company cares enough about fixing their game that they gave me a free copy." Lastly that the possibility is out there would make more bugs be reported which is a very good thing.
>>How do I, as a developer, convince you that beta means BETA?
Make Beta Testing unpleasent.
Drive away the hordes that are just there for a 'free game' or a 'head start'.
Do this by limiting the options of the testers.
Give your 'testers' limitied scope within the game.
For example:
Some are limited to just character generation. They can generate a character...and another...and another...but they cannot actually play.
Some can play the tutorial. They are assigned a character (pre created) and they can play the tutorial levels and zones... then they can 're-roll' and do that again.... and again... and again...
Have some that can play only level 1-10...some that can play 11-20... some that can only play certain zones... oh...and did I mention the weekly character wipes?
The reward is that for those who stick around... find and report bugs... test the boring stuff...actually contribute in a meaningful way on the (beta) forums... after a while they get greater 'freedom'. They get to Create their own characters and level them up all the way. Maybe they are still locked to a class or faction (that you want tested) but it's better.
And the ones that really stick it out and are of value - get free run in the end. But it takes time. It takes HARD WORK.
Maybe keep a register of the best testers - maybe it's something the industry as a whole needs to do?
Perhaps a forum run by developers or even a sub forum on a well known site like MMORPG.com where members can only access it if they can produce a log in code supplied by appreciative Devs?
And membership lapses after a few months.
That way you have a pool of real testers available that you know have been of real value to other devs. Not fool proof - but better than the mouth breathers that you get by giving away 'keys'.
This has been a problem for years:
http://www.mmorpg.com/discussion2.cfm/thread/209376/page/1
If you want to run 'pre release promotions' then call it that. NOT a "Beta Test"
If you want to run a 'stress test to failure' then call it that. NOT a "Beta Test"
And at the end of a Beta Test...wipe all characters without exception. Make it clear this will happen. Those that choose to leave as a result are not 'testers' anyway.
Note: I have 'Beta Tested' several games now. I actually report bugs and set out to break things. In one MMO I managed to break the Tutorial less than a fortnight prior to retail release. I was left wondering exacly what the other 'testers' had done for the previous months?
I have been personally invited back (they sent me an email) by one company to test an expansion. In that game I had been extremely critical of the game in the Beta forum but I had always given a reason and proposed solutions. I was attacked by many Fanbois for my comments and most of my comments resulted in no change to the game - BUT - apparently the Devs liked having someone around who was not a 'yes man'?
Nothing says irony like spelling ideot wrong.
Oh No Sanya, you are starting to sound like me! -)
I can still remember when Devs required actual formatted reports to be sent via email when testing. Or when characters were wiped every 3 days. Or the server(s) were reset every 25 minutes. And us testers loved it all.
Testing DAoC was like being included in a large family. Everyone listened to each other and testers helped the Devs and each other. The bonding community made you care about the project. I found this to be true when testing F&F for LOTRO:SoA... especially with Tiggs and her Extra Crispy guild involved. Testers felt like they were assets to the projects...not just another warm body to run around in the game while background-game-logging programs tracked our playstyles.
Testing has gone from individual input to spreadsheet input. We are now ants keeping the hive humming while our actions are being tracked and quantified in automated reporting tools. And testers are no longer given the tools to help report locations and issues. Recent games I've tested (and one I am currently testing) don't have a /loc command, no way to take screenshot in-game (other than using Fraps), no interaction with the Devs/Mods/CSRs, bug reports only receive auto-generated replies (Tabula Rasa had a great bug reporting system!), and in-game help tickets are ignored & deleted by GMs with no reasons given.
LOTRO has an invite-only test server where new content is tested prior to release to the public test server. Turbine Devs interact directly with these testers to break new systems, flesh out new content, listen to ideas, and focus-test team-based areas. I am sure other game publishers & developers have servers like Turbine's but just don't publish the fact.
I sincerely hope game companies return to the idea that testers are there to help you create a stable enjoyable enviroment and are assets who want to invest their time FREE OF CHARGE to help the projects.... and not just another person who might buy the game if they get pre-release exposure.
TQQdles™,
Dolnor Numbwit
aka Noldor Numbwit
Eternal Newbie
P.s. I still wear the DAoC Road Trip T-shirt now and then! -)
1) Define exactly what you are looking to do, both for yourselves and your audience.
2) Don't call a stress test or prerelease test drive a 'beta'.
3) Pick your first rounds of beta testers carefully. It isn't that hard... just look for the folks on your forums who take the time to wrote out thoughtful, logical topics or replies. Invite them. Those people are in turn likely to know people who wouldn't be a waste of time. Let your proven testers invite 5 more people. Simple.
4) Make certain that the bug handling process is as transparent as possible. Feedback to the community is MANDATORY. As others have explained earlier in this thread, nothing discredits developers and a company more than having issues reported, ignored, then left unresolved for long periods accompanied by a communications blackout. That type of behavior is common, and sends a tremendously negative set of messages to the community.