Howdy, Stranger!
It looks like you're new here. If you want to get involved, click one of these buttons!
Quick Links
General: Free Zone: Are Betas Useful?
Stradden
Managing EditorMember CommonPosts: 6,696
MMORPG.com Free Zone columnist Richard Aihoshi tackles the idea of beta testing and asks whether or not the development phase is as useful as has always been implied.
Does it strike you as kind of curious that by the time most MMOGs reach the closed beta stage where members of the public can apply to see and try them, there doesn't seem to be much true testing left to do? In this respect, I'm referring to elements, features, balance, etc. that are still subject to decisions or meaningful changes depending on the feedback the participants provide as well as related metrics.
Given the fact a substantial number of free to play offerings are imports that have already been operating for some time in other regions, I understand their betas aren't clean slates. The extent to which these games can be modified is always going to be limited. Accordingly, I don't expect that anything and everything can be changed. However, I can't help but wonder if there's any real point to "testing" when everything important is already decided or locked down.
Read Are Betas Useful?
Cheers,
Jon Wood
Managing Editor
MMORPG.com
Comments
There is one other function that an open beta serves beyond anything to do with the content of the game: the stability of the servers. It is difficult for a company to stress-test their server with 10,000+ users unless they open it up to the hordes.
That said, "closed" betas seem to be less useful. On the one hand, it might make a small subset of gamers that get in on them very happy, but it doesn't do much for those that are rejected. Could the overall result be to alienate people who are left out?
I know beta testing is a necessary evil, but recently they've become a form of RMT in my opinion. You pay extra money for the collector's edition or preorder and you get to play the game before the majority which have to wait for the release, thus gaining a slight advantage. It's not a huge deal, but it's not really fair either.
Vault-Tec analysts have concluded that the odds of worldwide nuclear armaggeddon this decade are 17,143,762... to 1.
IMO they need to stop allowing Pre-Order customers into closed betas. Closed Beta testers should be the type of players that will actually test, and submit bugs, suggestions, etc. I also feel that testers who haven't logged in for a certain amount of time, or have never submitted feedback should be removed from the test. I feel it should be important to devs to weed out people who just want a preview.
The whole thing in a nutshell as I see it when it comes to beta testing, it is only as useful as those doing the testing.
If you have a bunch of testers that have no clue about gaming as a whole and what they are looking for or at or a bunch of kids that just want to play before anyone else then no it is not useful and is a waste of time for the devs and staff.
If it is people that realize it is beta and don't put the game down and report their findings and even their suggestions then by all means it is going to be useful.
Then if the team and devs working on the game listen and do what is needed and even take suggestions into consideration then the whole process can be useful. Realistically all suggestions and ideas may not be able to be put in but maybe if the team is listening it might at a later time. I myself have done some beta testing and I go in with the thought this is beta and I expect to find problems and bugs. As a tester I report what I find and if it is something I am not sure of then I put it down to find out. It is really annoying to hear peeps in general chat complaining because things don't work right and comment the game is not any good.. of course they don't work right that is why it is called BETA. /shakes head /face palm
Gikku
Love these betas that already have their item mall active, features events and no wipes, from some foreign developer 3 year old game.
Call them stress tests if you need, there are so many more efficient ways to get a stress test than calling it a beta, opening up the item mall and leaving the server online 24/7.
In my opinion that's just an excuse for some media spotlight, the announcement of closed beta #1, #2, #3, etc and open beta #1, #2, #3,...
Well, I don't know which betas you refer to in your article but in 3 of the last 4 Betas I have been involved in (1 not an MMO) I managed to 'break' the tutorials. So in my case - yeah still lots of testing left to do.
And sadly, in the case of the three with easily breakable tutorials all those games were less than a month from final release.
I agree that anything called a "test" should be a test and those 'testers' that fail to provide feedback should be kicked.
I also think that companies that want to boast about beta tester numbers should release metrics - what percentage actually 'test'? (submit feedback / bug reports)
If testers are leaving - how long were they in game - what level did they reach?
And if they don't submit feedback / reports...kick them.
You want to have a "pre-release-preview" call it that.
Nothing says irony like spelling ideot wrong.
The beta phase in software development is as useful as the developers want to make it. The "Beta" phase for most MMO's we see have very little to nothing in common with the similary named software development beta. (honestly i think the mass abuse of the word was started by Google).
What can developers get from a beta?
What do players get from the beta:
Some product beta's definatly lean more towards actual testing (Wrath comes to mind, long semi-closed beta), but the problem is the group that are using the beta for a free demo have the opposite goals to those that really want to beta test. You show them a bug ridden product (i.e. a beta) and they just leave saying the game is bug ridden garbage. And to be fair a beta product shouldn't have many obvious bugs, that however I have yet to see.
Long beta's seldom do a product any favours (Tabula Rasa, Auto Assault), but I think that's because those games were lacking in many area's not just bugs. That and the fact all the beta feedback about the issues was point blank ignored. What do they know, they're only customers?
In my ideal terminology there are three stages:
alpha test: playable, but still in active development, major features are still being debated/implemented
beta test: all major features implemented (content/features can be pulled but no longer added), balance tweaking is still open to debate
release candidate: all content is locked in, only changes at this point are dealing with bugs, exploits and polish.
Depends on what you mean it should be useful for.
Is it useful in the sense that major features might change as a result of beta? Likely not.
Is it useful in the sense that it finds bugs and exploits that was not found during closed beta? Definetely yes. I have been part of several open betas were hard to find game breaking bugs/exdploits was found and dealt with during the open beta.
And that goes for any major piece of software. There are always bugs to be found when you release it to thousands of people instead of tens/houndred.
My gaming blog
Balancing is something that happens through the lifetime of an MMORPG. I dont think I have ever seen an MMORPG that did not suffer through balancing after release.
Reason is simple. Overpowered builds and/or FOTM classes are rarely all found during beta. However releasing a game with major exploits/bugs is a very bad idea indeed as it will instantly make you lose alot of live customers. Just look as VSOH and AoC releases.
My gaming blog
Closed betas are not supposed to benefit the player. It's a stage of testing that's supposed to benefit the game, so the developers can see if their concepts actually work the way they're supposed to when exposed to a greater number of people than their alpha test, and to see if this greater number of people use their devices in ways they didn't think about so they still have time to change some things. Open betas are typically nothing more publicity for the game, and a chance to stress their servers to something that they're likely to see when the game goes live.
Unfortunately - betas are used as a publicity stunt all too often. I've participated in several betas of games that were releasing their english-speaking version and even years later have not bothered to fix the glaring translation errors. As a beta tester for these games, I'm upset with the lack of response and refuse to participate any further in the game. There have also been exploits and bugs (such as dropping through the walls) that were never properly addressed or even attempted.
Do I expect that every small error will be fixed during a beta test period? No! I know that some issues involve serious re-coding of the game which is near to impossible. Other issues may be too far down on the task list to worry about right now due to other problems. And I know that it just plain takes time to fix things. But my request as a beta tester - and one who does try to determine how to recreate a bug or states exactly where the translation error is and how it SHOULD read - is that they at least try to show they care.
Give us good bug reporting tools - don't leave it all up to discussion in a forum that is only moderated by other players and not read by developers. If a bug isn't fixable, and it's been reported multiple times, shoot an email to a forum moderator and have them TELL us that - otherwise it's easy to assume that you just don't care about fixing it. And if you really want beta testers - vet them - put a few obvious things early in the game, let the testers know that there are x number of bugs in the tutorial - if they don't report a percentage of those - they won't be going any further. That will eliminate the glory hounds and ensure that beta testers are TESTING, not just playing through as fast as they can.
Perhaps the last seems harsh, but I do get annoyed when I'm doing bug reports or tracking how many times a difficult quest item drops, and others are just crowing about how they hit level 30 in two days. If they did that, you know most didn't even read the quest prompts, so if an NPC has given them bad directions they have no way of knowing that. (And yes, some players DO still read the NPC prompts, so it does matter in beta testing.)
OKay - rant over, perhaps somewhere there's a game designer that wants to use beta testing to actually determine a few things about the game. I'll keep looking.
Always a Lady - even when fighting life's demons.
Hope this helps:
(Interview with Jeff Anderson, former CEO at Turbine)
John: That answers a couple of questions for me. Just one more for you. In the closing months of developing an online game, with the game starting to head towards open beta, what are the aspects that are being worked on, polished, or finalized?
Jeff: What are the aspects?
John: Yeah, at this point in the game, nothing specific, just in general, what are the different things that get worked on at this stage?
Jeff: That's a great question. I don't think anyone's ever asked me that question, but I really like it. You know, there's a variety of stuff that you'd expect people to polish, but we're still doing tweaks to classes and we’re still doing tweaks to combat, and we're still doing tweaks to animations... we're obviously fixing bugs... and corrections to where quests point the player to make sure they are giving you clear enough direction so that you can find the objective. We're making tweaks to the placement of content so that areas feel full and so that there is nice flow from one location to another location. Maybe some of the things that people aren't really aware that we spend so much time on is integrating user feedback. That's a big chunk of what we are doing right now. From the beta players, from our own focus groups and from the test groups, we're evaluating how those people are reacting to the product, and spending a lot of time making changes to address those concerns.
If I can give you one example, maybe not a great example but an easy one: When you first start the game, you get into it and you'd be running around as a new player. First you try to find your way and you pull up the map, look at the world and say, "Oh, I'm in the Shire and I'm this little triangle here and I'm going to the north or east or whatever direction I am heading." When you are doing that, it's quite fine once you are comfortable with the game to navigate using those maps, but a lot of the players we have testing felt that when we put them into the game for the first time, they were getting lost in the starter areas. So as a Hobbit when they first come in, or as an Elf, Dwarf, or Human, when they first came into the area, they were unfamiliar with the game in general and kind of unfamiliar with the layout and with a game world that is so big - we're talking about 50 million square meters – and you would have your little triangle moving around and on the main map, the region map, say the Shire for example, the triangle moved very little, but yet it was covering huge spaces of content and you may have been running around in this acre of space and your marker was moving so little that the new player was finding it a little hard to navigate. As a result of that, based on user feedback, we created the next layer down of maps for some of the key places like the city of Bree. You'll actually be able to zoom one layer closer into the map so you can actually see the streets of Bree. It makes the player's life so much easier when he or she can effectively navigate around the place.
- RPG Quiz - can you get all 25 right?
- FPS Quiz - how well do you know your shooters?
What most devs call "beta" is nothing more than a free trial and server stress test. Players see this as a chance to get a head start on other players, as well as a chance to try out a new game to see if they want to buy the box.
As a small Indie dev who really NEEDS a real beta, this redefinition of what beta means is a real pain.
When I get BETA feedback like
"This game isn't finished and it has bugs. The starter tutorial is missing!!! FAIL"
... I can't do much other than facepalm. The situation is a mess.
Ken
Currently in open Wide-Beta
www.ActionMMORPG.com
One man, a small pile of money, and the screwball idea of a DIY Indie MMORPG? Yep, that's him. ~sigh~
Personally i'm more then happy to play betas for most games that are not fully released. For me, it saves the trouble of downloading a F2P in a few months to come. If the game doesn't live up to expectations, you can simply stop the beta. i'm currently on the European version on Allods online testing the game and so far there are no major problems with the game (except translations). I feel like betas are a chance to experience new games, more than an actual test themselves. It gets you buzzed and hyped for a game and for most players you don't even take into account that your suppose to be "testing" the game. Many players simply do the quests, customize their characters as far as they can and reach the end of the beta without taking into account any of the issues (if any). the only slight issue i have is that these games have been released in their mother countries, and tested therefore making our betas basically bug less (for most games). In this situation you really just wan't the full game so you can gain all you spells, levels, raid, PvP and fully experience the game instead of feeling endlessly stuck at certain sections. If it's tested, just advertise the full game.
Waiting for the next big thing.
Yes I think beta is a usefull tool. It give me a chance to see how that game works. How well the handle the bug report. How well they go about fixing bugs. How well the implement patches. How well they listen to the player base.
If I see a company who does not even try to fix bugs or even attempt to listen to what the beta testers has to say then I stay clear of it. Knowing that I will be banking my head on the wall.
It is sad that, and I can say this with all honstly. I have been in a recent beta I cant say who yet the NDA is not lifted. I have not see any of the things I sent bug reports on fixed. Not only that but none of the concers in the beta forums are have even been touched on. This for a game thats going to push this live very soon. All I can say this beta has ben a joke, and a stunt. It was designed to help retain the current customer bleed off, I think it has had the oposite affect. It has made more folks run away than stay.
Beta is free viral advertising if the game is well done. If the game is crap beta can sink your game before its released.
I participated in guildwars, world of warcraft, & Lord of the Rings Online Beta. Guildwars & WOW were quite enjoyable. At the time I was playing DAOC and I'd go into DAOC and talk about how great those games were. My friends grew interested and joined beta, they told more people and so on...
On the other hand you try a game lets say, Alganon which came across like a poor copy of WOW without the polish or gameplay it just turns everyone off. The game has such a negative impression I doubt it will do well. Its kind of like when the movie Waterworld was being made. You just kept hearing all this negativity about the movie and how much it was overbudget that it was deemed a failure before it was released.
In regards to beta being useless I'd have to say that LOTRO's beta fit that part.
I was banned for pointing out aspects of the game that should be included. Since those aspects weren't in the books people decried their addition based on lore alone. My point was that if their main competition provides the aspects of the game they'd be foolish not to do the same. Who wants to start over in another mmo that doesn't provide the same types of gameplay ? When I stuck to my point they ended up banning me. I wasn't rude, I didn't swear I was suggesting what I thought would make the game better. The aspect of the game I was talking about was PVP, which they ignored all suggestions and in turn MVP was a failure. Since that time they've introduced miniscule changes to improve that aspect of the game, these changes were things I suggested in beta that got me banned.
Yea I was wondering what happened. The original beta for that game lasted for over a year. And they did fix a lot of things.However I have to say the beta for MOM they fixed nothing. We pointd out tons of stuff, and when we went live none of the bugs, and huge glarring ones at that that had contained several threads into the hundreds of pages were completely ignored. So it was more of a publicity stunt for MOM.
Beta tests, BOTH open and closed, are really just marketing opportunities. Turbine realized this, and had LOTRO 90% ready to play during the closed Beta (heck, even the Alpha version I was in was pretty close). Recently BioWare immediately clarified a position that current testing phases WEREN'T beta, but were testing phases. They were careful not to use the Term Beta. Why? Because they too realize that Betas are marketing tools more than game fixing tools.
People cling to this notion that the Beta tests are this exclusive testing ground, but the reality is different. The reality is that you're letting in the public who have demonstrated time and again that they will break NDAs to serve their own selfish egos. There is no fool-proof way to police who gets into the beta either -- people will say anything on their applications. So you have a random sampling of the public who come to the Beta Test for their own purposes -- some to get a free look at the game, some to get a leg up on live players, and yes, some to test. But because of this very nature of 'the public', games need to be near-ready for launch at the Beta Stage. If it's not, you'll have NDA breakers spreading negative marketing messages, you'll have potential players give up in frustration at bugs, etc.
Betas are no longer testing grounds...plain and simple.
_____________________________
Currently Playing: LOTRO; DDO
Played: AC2, AO, Auto Assault, CoX, DAoC, DDO, Earth&Beyond, EQ1, EQ2, EVE, Fallen Earth, Jumpgate, Roma Victor, Second Life, SWG, V:SoH, WoW, World War II Online.
Games I'm watching: Infinity: The Quest for Earth, Force of Arms.
Find the Truth: http://www.factcheck.org/
i have never heard of one single game that let pre oders into CLOSED beta for that game, pre order often get a head start which is like 3 days be fore launce AFTER OPEN Beta.
as for closed beta i was in the CO one and their were a number of changes faciliated by us testers.
The death penently was complety changed to the star system we have now, the something fishy mission started as open world, and you could asign power from the open world (no power house) i fact i faught black talon using 5 powers and flight in closed beta...
but what happens with those betas where noone is listened to? feedback goes unheard and all it really is is to acquire some more money beta should be open to all for free its suppose to be a period with which they test the game not try to bilk money out of players
I wouldn't say that with 100% certainty. I am currently in a beta for an Indie Company and we are testing and providing feedback and suggestions. The message boards for this game is wide open to the public(No NDA) and you can see the interaction between the beta testers and the developers quite freely. If it were to test servers only there would be way more beta testers and there isn't. They seem to be very selective in who they let into their beta. They must have done well in choosing because the beta testers that are currently testing the game come down hard if they suspect someone is just there to play a free game.
No question a BETA "SHOULD" be useful ,but what we have seen from them is a compelte joke.
Look at Aio na game that claimed they were releasing a superb polished product,yuet right away long Que ue's,no water physics and MANY players can't get past Game Guard,so what the heck did they learn from BETA...NOTHING?
Most Beta as we seem them know are nothing more than GIMMICKS to bribe players into their game and websites are joining in the gimmick to attract viewers/subscribers to their website.example ...register to get your free BETA code.
You will see the most obvious bugs and mistakes in Beta and yet the game will release with the exact same mistakes,sop Beta has been a waste of time for usefulness.Both the players and the developers are to blame for making Beta into a joke,neither one treats it properly, to make sure the game is released playable for all.
There is a NEW trend from BETA's i am witnessing witch is even more lame than what we have been seeing from Beta.This NEW trend is to take a game that is 2 months from being finished and they CLAIM to start a BETA on a game that is not really in Beta stage ,because it is not even finished.However they try to fool the public by claiming it is in Beta stage and just use that time, togive the APPERANCE they are ready and in testing,but they are not.
Never forget 3 mile Island and never trust a government official or company spokesman.
I haven't gotten the sense that any beta has been done well at this point. The reason is simple. When I played on AO Test Server, they were continually improving the feedback process for committed testers. And those testers were respected by the developers for their professionalism and commitment to discovering bugs and reporting them thoroughly. They came up with a system that rewarded based on the content of the bug report, and that reward system was tied into the reporting system, which was an Internet-based application that you logged into separately, and that was closely monitored by the development team. The in-game bug reporting tools are a joke. Not a single one, in my opinion, has managed to capture the information needed to resolve issues. This is unfortunate, as a better tool in-game would probably allow developers to pinpoint problems almost instantly. However, we're discussing betas here, and I'd have to agree with others who see them more as a marketing tool than a true effort to resolve bugs before the game goes live. It's a shame, too, because I feel that a lot of games out there had real potential, if they'd only lived up to it by listening to the players and their suggestions before going live.
/agree
Just because you grease the pockets of Game Stop doesn't mean you're a better beta tester then I am.