Over the last couple of weeks I’ve been participating in the beta test of upcoming game Starcraft II. In an interview with Gamasutra magazine, the lead Starcraft II designer, Dustin Browder, had this to say about the data they have collected from the beta test:

The danger with a lot of this data is that you have to be very careful how you use it. With unit stats, I can tell you that, for example, in a Protoss versus Terran game, 12 percent of the time the Protoss build carriers. And when they build carriers, they win 70 percent of the time. You could say, “That must mean carriers are overpowered!”

That’s not really true, though. It could just be that as you get towards the end of the game, if the Protoss have the extra resources to waste on a bunch of carriers, they’re probably going to win anyway.

Of course, it doesn’t mean the carriers aren’t overpowered either. That stat alone actually tells you nothing. It’s a very dangerous stat. If you listen to that stat, you can make all kinds of mistakes

If we look at the stats and we say, “This doesn’t actually back anything we’re experiencing online,” I’m very suspicious of that number. We get information from a lot of different sources, and then we use the other sources to refute or corroborate. We look at another source and say, “You know what? What they’re saying online matches my play experience, and it matches the stats. This seems real. Let’s talk about what some possible fixes can be.”

So very, very relevant to Test Pilot and Firefox!