Blake (who generated all the cool graphs I’ve been using to present Test Pilot results) has used the session data from the Week-in-the-Life study to form an interesting hypothesis: That Firefox crashes per user follow a power-law distribution. If true, the power-law distribution means that “mean crashes” and “typical experience” are two very different things.
I should emphasize that the Week-in-the-Life study was not really designed to look at crashes, so I’d rather do a follow-up study specifically targeting this hypothesis before I support it with any confidence. But, as Blake says:
If our crash data follows a similar distribution, the average crash per user metric tells us little about the experience of a typical Firefox user.
Anecdotal evidence supports this hypothesis. While we all know people who swear by Firefox’s stability, we also know people who complain of frequent failures.
With this in mind, I suggest we use Test Pilot to run a longitudinal study of true Firefox crashes.
Agreed! And it’s great to see more hypotheses coming out of the Test Pilot data!