I get a lot of people contacting me by email, IRC, forums, or blog comments to say that they’re “worried that if I join Test Pilot I’ll skew the data” because “I’m sure that my tab usage is atypical”.
People! Don’t worry about being an atypical user!
First of all, we have already had almost 5,000 Test Pilot data submissions. One outlier isn’t going to do much to “skew” a data set of that size.
But more importantly, you shouldn’t assume that you’re abnormal. We don’t know what “normal” tab usage is! That’s why we’re doing this experiment, to find that out. If we started out with an idea of what normal tab usage looked like, and threw out things that didn’t match our preconceived notions, that would be a clear case of experimenter bias. Then we’d really be skewing the data.
For instance, I was surprised to find out that there are users who have over 500 tabs open at a time. Over 500! They’re surely outliers, but they’re not abnormal users &emdash; they’re just users. That number isn’t skewing the data &emdash; it is the data. Thanks to those users’ participation, we now know that having 500 tabs open is something that people do with Firefox, something we might not have known otherwise.
As I said in a previous post, I do believe we have a major oversampling of the power-user / early-adopter demographic in our current Test Pilot user base, and that we need to work on fixing this by reaching out to a wider sample of users. But note that word: wider. Excluding yourself because you think you’re atypical isn’t helpful. If you really want to help our sample — and I’m touched that so many of you do want to help " the best thing you can do is to let your less-techie friends know about Test Pilot.