I get a lot of people contacting me by email, IRC, forums, or blog comments to say that they’re “worried that if I join Test Pilot I’ll skew the data” because “I’m sure that my tab usage is atypical”.
People! Don’t worry about being an atypical user!
First of all, we have already had almost 5,000 Test Pilot data submissions. One outlier isn’t going to do much to “skew” a data set of that size.
But more importantly, you shouldn’t assume that you’re abnormal. We don’t know what “normal” tab usage is! That’s why we’re doing this experiment, to find that out. If we started out with an idea of what normal tab usage looked like, and threw out things that didn’t match our preconceived notions, that would be a clear case of experimenter bias. Then we’d really be skewing the data.
For instance, I was surprised to find out that there are users who have over 500 tabs open at a time. Over 500! They’re surely outliers, but they’re not abnormal users &emdash; they’re just users. That number isn’t skewing the data &emdash; it is the data. Thanks to those users’ participation, we now know that having 500 tabs open is something that people do with Firefox, something we might not have known otherwise.
As I said in a previous post, I do believe we have a major oversampling of the power-user / early-adopter demographic in our current Test Pilot user base, and that we need to work on fixing this by reaching out to a wider sample of users. But note that word: wider. Excluding yourself because you think you’re atypical isn’t helpful. If you really want to help our sample — and I’m touched that so many of you do want to help " the best thing you can do is to let your less-techie friends know about Test Pilot.
Not The User's Fault 
September 26, 2009 at 2:54 am
A friend of mine is known to quite regularly have several hundred tabs open in Firefox. Since he leaves his computer running 24/7, he never closes Firefox. Whenever he wants to check anything on the internet, he just opens a new tab.
I myself find that I may have up to 80 tabs open at a time when doing research and the like, looking through every reference source I can think of and following through on every link I deem relevant. So as you were saying, having many tabs open is something people do with Firefox, and it is certainly information to be considered when looking for ways to improve Firefox.
September 28, 2009 at 3:56 am
Amen.
September 28, 2009 at 3:29 pm
Silly nitpick.
Since wordpress apparently is escaping your entities, since it is serving it as UTF-8 anyway, maybe you should just use “—” (U+2014 or UTF-8 0xE28094) instead of the emdash entity.
September 28, 2009 at 5:43 pm
Also, it is super useful for us to look not just at distribution of usage patterns, but clusters of behavior. If a type of tab usage is a bit overrepresented because of the demographics of the Test Pilot user base, that’s fine, because it’s the general archetypes we are looking at.
September 29, 2009 at 5:28 pm
Certainly if you were trying to make your determinations based on a typical user testing set of a dozen or so then you’d need to be careful not to include too many power users. But if you’ve got 5,000 submissions then I’d have to say “amen” too. Even the outliers aren’t going to mess up a data set -that- size!