Over the last couple of weeks I’ve been participating in the beta test of upcoming game Starcraft II. In an interview with Gamasutra magazine, the lead Starcraft II designer, Dustin Browder, had this to say about the data they have collected from the beta test:
The danger with a lot of this data is that you have to be very careful how you use it. With unit stats, I can tell you that, for example, in a Protoss versus Terran game, 12 percent of the time the Protoss build carriers. And when they build carriers, they win 70 percent of the time. You could say, “That must mean carriers are overpowered!”
That’s not really true, though. It could just be that as you get towards the end of the game, if the Protoss have the extra resources to waste on a bunch of carriers, they’re probably going to win anyway.
…
Of course, it doesn’t mean the carriers aren’t overpowered either. That stat alone actually tells you nothing. It’s a very dangerous stat. If you listen to that stat, you can make all kinds of mistakes
…
If we look at the stats and we say, “This doesn’t actually back anything we’re experiencing online,” I’m very suspicious of that number. We get information from a lot of different sources, and then we use the other sources to refute or corroborate. We look at another source and say, “You know what? What they’re saying online matches my play experience, and it matches the stats. This seems real. Let’s talk about what some possible fixes can be.”
So very, very relevant to Test Pilot and Firefox!
Not The User's Fault 
June 8, 2010 at 10:01 am
It should be a well known fact to game designers that balancing game stats with more stats is a huge mistake. You need to balance the the game’s stats by analyzing them directly, not their results.
In StarCraft’s case (the original), Carriers were a bit overpowered, and they were expensive as hell to produce. The problem is, mixing those two stats (power and cost), you get an incredibly powerful campaign unit (since you’d mostly carrier your way through), but a not so powerful multiplayer unit. That’s the main problem with balancing a single set of stats for two completely different types of game: single player and multiplayer. The objectives just change a lot, and while in single player you can wait until you have 20 carriers with 8 interceptors each, you can’t really do that in multiplayer, and his reading of the stats is, I must say, very down to the point.
In the original games, the amount of power matters above all else in the campaign. The protoss were the most powerful with the Carriers (if you had 12 carriers and an Arbinger, there’s absolutely nothing that would kill you), second were the Terrain with the Battle Cruisers (they were more frail and less powerful, but their yamatoguns dealt with the AA defenses pretty quickly and everything else would die very quickly) and third were the Zerg with no real best-of-the-best. That’s why the Zergs, for me, are the most powerful online: their cheap!
That unbalancing was really by biggest problem with starcraft, but it’s very well balanced anyway, it’s a load of fun and it’s artistic direct was fantastic. Now with cartoon style things, I don’t really know anymore…
June 8, 2010 at 10:12 pm
What do Test Pilot and Starcraft 2 not have in common?
Region locking >.<
(from a grumpy NZ SC2 beta tester)
June 26, 2010 at 9:26 pm
cant wait for starcraft 2… im still playing broadwar nowdays..hehe
December 21, 2010 at 8:07 am
The game was quite well balanced for a beta. However the experience was limited to the multiplayer content only and there was no teasers for the campaigns.