Not The User's Fault

I get a lot of people contacting me by email, IRC, forums, or blog comments to say that they’re “worried that if I join Test Pilot I’ll skew the data” because “I’m sure that my tab usage is atypical”.

People! Don’t worry about being an atypical user!

First of all, we have already had almost 5,000 Test Pilot data submissions. One outlier isn’t going to do much to “skew” a data set of that size.

But more importantly, you shouldn’t assume that you’re abnormal. We don’t know what “normal” tab usage is! That’s why we’re doing this experiment, to find that out. If we started out with an idea of what normal tab usage looked like, and threw out things that didn’t match our preconceived notions, that would be a clear case of experimenter bias. Then we’d really be skewing the data.

For instance, I was surprised to find out that there are users who have over 500 tabs open at a time. Over 500! They’re surely outliers, but they’re not abnormal users &emdash; they’re just users. That number isn’t skewing the data &emdash; it is the data. Thanks to those users’ participation, we now know that having 500 tabs open is something that people do with Firefox, something we might not have known otherwise.

As I said in a previous post, I do believe we have a major oversampling of the power-user / early-adopter demographic in our current Test Pilot user base, and that we need to work on fixing this by reaching out to a wider sample of users. But note that word: wider. Excluding yourself because you think you’re atypical isn’t helpful. If you really want to help our sample — and I’m touched that so many of you do want to help " the best thing you can do is to let your less-techie friends know about Test Pilot.

Alexander Limi has posted a video, from the Design Lunch three weeks ago, wherein the UX leads answered questions about their provisional redesign of the Firefox UI.

Alex wants to warn you that the contents of this video are very rough, very much a work-in-progress, and very much liable to change. You shouldn’t watch it thinking "this is how the next version of Firefox is definitely going to be". You should think of it as a behind-the-scenes look at "how the sausage gets made", in Alex’s words. He also warns:

There’s swearing, there’s mumbling, there’s ranting, there’s hand-waving, there’s political incorrectness.

I’m also in the video, playing MC in my overalls.

Today we recorded Part 2, which goes into detail about the notifications interface and the downloads manager; it should be posted fairly soon.

For my contribution to Mozilla Service Week, I’m going to be helping the local Stanford Chinese School improve their student/teacher database. Should be an interesting learning experience. Among other things, they have a bugs where Chinese characters in certain places are getting mangled. Figuring this out will be good practice in the practical aspects of internationalizing the Web.

OK, let’s say I’m an amateur game developer. I’ve got an idea for a computer game where players build and manage a space colony and terraform an alien planet. I’d like to write it as a hobby project and play it against my friends.

First thing I gotta do is pick a platform. What are my options?

In theory, I could develop for a video game console. But that means super-expensive software development kits, and hardware-specific programming techniques, and besides, the company that controls the console will get to decide whether to allow my game or not. Console games are not practical for a hobbyist developer. I’d better write a computer game instead.

I could write my game for Mac OS X, using Cocoa or whatever. I’d have some pretty snazzy developer tools for building my user-interface, but only Mac users would be able to play it; that’s a pretty big drawback.

I could write my game for Windows. Same deal; snazzy tools, but only people with Windows could play it. As a one-man project, I don’t have the resources to maintain parallel Mac and Windows versions. And what about Linux users?

(more…)

Here at Mozilla we talk about “The Open Web” a lot. We talk about it all the time. It’s one of our causes. The word “open” appears seven times in our (very short) Manifesto. We have a proposal for a project called Drumbeat which aims to get people acting as “stewards of the open web”.

But what does “open web” actually mean?

Recently my coworker Jinghua asked me what the Open Web means in concrete terms. “Don’t give me abstractions or generalities”, she said, “tell me one specific reason that users should care.”

Daaaang. That’s a good question!

Is “The Open Web” a thing that exists now, or is it an ideal that we are trying to bring into existence? Is the openness or closedness of the web a binary distinction, or a sliding scale? How is it related to open-source software? Is it just a theoretical conceit of us web purists? Above all why is it something that users should care about?

There is not a single answer to this question; there are many answers.

Here’s one: the open web is a world where kids can teach themselves how to hack.

Here’s another: the open web is a world where people can write a subversive-but-legal tool to make U.S. Judicial records freely searchable. (Both blog posts by Atul Varma.)

I’ve been ruminating over some answers of my own; I’ll explain them in future blog posts.

Test Pilot is a delicate balancing act.

On one side is user privacy. I can’t overstate how committed we are to protecting the privacy and anonymity of our users, keeping them fully informed, and getting their consent before sending any data about them back to Mozilla. We practice full disclosure, collect nothing without an explicit opt-in, and let you review exactly what will be uploaded, in human-readable form, before you upload it. We never associate any of your uploaded data with your name, email address, or any other personal data about you. We’ve been significantly more conservative about data collection than most other organizations that do this kind of research.

On the other side are the needs of scientific research. We want to collect the most detailed and accurate data possible and share it with the researcher and UI designer communities, in order to try to improve the state of not just the Firefox interface but of web interface design in general. There are certain burning usability questions which would be much easier to answer if we were to collect certain kinds of personal information about our users.

(more…)

For Test Pilot to be a success, we need to recruit a group of volunteer test subjects who accurately represent Web users as a whole. In order to figure out what sort of people are downloading and installing Test Pilot, we included a survey with the first version. (Note: In order to preserve user anonymity, user’s answers to this survey are not correlated with any data submissions they make through Test Pilot.)

Here are the survey results.

Hmmm. 96% male, 4% female. Google Chrome as popular as Internet Explorer. More Linux users than Mac users.

What we have here seems to be a slice of the “early adopter” demographic. It doesn’t look like a representative sample of Firefox users as a whole, and certainly not of Internet users as a whole.

That’s to be expected, of course. Test Pilot was announced on the Mozilla Labs blog, so the initial Test Pilot user base is probably drawn heavily from Labs blog readers and other people who closely follow what Mozilla is doing. Those tend to be people who are interested in the cutting edge of technology.

So there’s nothing wrong with having a lot of early adopters in the Test Pilot pool. But I daresay that if we make Firefox UI decisions based on a sample like this one, we’d be doing a disservice to the majority of our users who don’t fit the early-adopter profile.

How can we reach out to a wider and more representative sample of users? One that includes more women, more internet newbies, and more people who are used to IE? This is really two questions: First, how can we let these people know that Test Pilot exists; and second, how do we make participation appealing to them? I don’t know the answers to these questions yet. Here are a few ideas that we’ve been kicking around:

• Promote Test Pilot through Facebook (duh).
• Have a checkbox on the main Firefox download page that someone can check to opt-in to Test Pilot — it would bundle the extension with the download.
• Offer cool features that are only accessible to users with Test Pilot installed, as an incentive to join. (But these would probably be experimental features, thus appealing mainly to the early-adopters.)
• Appeal to people’s natural curiosity about themselves and people they know, by positioning Test Pilot as a way for users to learn about their own internet habits, and those of their friends, with an option to send that data to Mozilla.

How else can we help Test Pilot reach beyond the early-adopters and find a wider, more accurate sample of humanity? I would love to hear your ideas on this question.

The reason I haven’t been updating this blog lately is because I have been hard at work on Test Pilot, Mozilla’s platform for distributed usability research.

One week ago, we released Test Pilot 0.2, which contained the code for the first Test Pilot study. This is a study on how users in the real world interact with Firefox tabs. We’re looking to answer questions like:

• When users close a tab, do they stay on the next default tab, or do they quickly switch elsewhere? (In other words: how good or bad is our default of switching back to the previously open tab?)
• What does the distribution of open tabs per user over time look like? Does a typical user leave certain tabs open all the time? How many people out there use only a single tab? How many use so many tabs at a time that they need to use the tab scroll bars?
• When a user drags a tab to a different window, how often do they immediately undo that action (indicating that it was probably a mistake)?

We’re storing the data in a very raw form, which should make it possible to use it to answer not only these questions, but other questions that we haven’t even thought of yet.

The tabs study was set to finish seven days after installation. The first users installed 0.2 seven days ago, so last night and this morning, the first users began to submit their data. Everything seems to be going smoothly, and the data is flowing in right now. On Monday we’ll start analyzing the data and sharing what we find.

It’s been a long road to get here, but now that the first Test Pilot study is finally in user’s hands, I have a little time to step back and talk about what it all means. Personally, I’m super excited about Test Pilot. I’ve long believed that we need data like this in order to approach user-interface design in a more scientific way. Now that the experiment is actually running on user machines, I feel like we’ve finally made the first concrete step to making that a reality.

Ultimately, the data that we collect through Test Pilot is not just for Mozilla: it’s a shared resource for the wider scientific and UI designer communities. Just as Mozilla is all about sharing source code in order to enable community development, we should be all about sharing research in order to enable community design.

There’s a lot we still have to solve in order to realize that ideal. Over the next few blog posts, I’m going to talk about questions like:

• How do we balance the need to protect user privacy with the need to get the most detailed and accurate data for the sake of science?
• How can we encourage normal, i.e. non-technical, non-early-adopter, users to participate in Test Pilot, in order to get a more representative sample of Web users as a whole?
• How do we start to build a design community around Test Pilot, and how do we help them design practical and scientifically sound studies in order to answer their design questions?