“An undefined problem has an infinite number of solutions” – Robert A Humphrey

I found that quote on the underside of a tea bottlecap. I think it’s germane to what we’re doing with Test Pilot.

In the months since we started the Test Pilot program, we’ve run three studies, which have collected massive piles of data.

But as the bottlecap warns, those massive piles won’t get us anywhere unless we know what problem we’re trying to solve with them.

There’s a temptation, when looking at a pile of data, to leap to design conclusions.

For example: Suppose studies show that almost nobody is using the profile manager. Great, that means we can get rid of the profile manager! Right?

Or, to use a non-hypothetical example:

Minimum, average, maximum tabs open per session

(Thanks to Blake Cutler for generating all the graphs used in this post).

The most common Firefox session is one with three tabs open? Well hot dog, we should optimize Firefox for the three-tab use case! Right?

Careful, there. This kind of assumption is dangerous. Maybe lots of people would start using the profile manager if they knew it existed. Maybe people would love to have sessions with more tabs open, if the interface was better for managing lots of tabs.

(more…)