The reason I haven’t been updating this blog lately is because I have been hard at work on Test Pilot, Mozilla’s platform for distributed usability research.

One week ago, we released Test Pilot 0.2, which contained the code for the first Test Pilot study. This is a study on how users in the real world interact with Firefox tabs. We’re looking to answer questions like:

  • When users close a tab, do they stay on the next default tab, or do they quickly switch elsewhere? (In other words: how good or bad is our default of switching back to the previously open tab?)
  • What does the distribution of open tabs per user over time look like? Does a typical user leave certain tabs open all the time? How many people out there use only a single tab? How many use so many tabs at a time that they need to use the tab scroll bars?
  • When a user drags a tab to a different window, how often do they immediately undo that action (indicating that it was probably a mistake)?

We’re storing the data in a very raw form, which should make it possible to use it to answer not only these questions, but other questions that we haven’t even thought of yet.

The tabs study was set to finish seven days after installation. The first users installed 0.2 seven days ago, so last night and this morning, the first users began to submit their data. Everything seems to be going smoothly, and the data is flowing in right now. On Monday we’ll start analyzing the data and sharing what we find.

It’s been a long road to get here, but now that the first Test Pilot study is finally in user’s hands, I have a little time to step back and talk about what it all means. Personally, I’m super excited about Test Pilot. I’ve long believed that we need data like this in order to approach user-interface design in a more scientific way. Now that the experiment is actually running on user machines, I feel like we’ve finally made the first concrete step to making that a reality.

Ultimately, the data that we collect through Test Pilot is not just for Mozilla: it’s a shared resource for the wider scientific and UI designer communities. Just as Mozilla is all about sharing source code in order to enable community development, we should be all about sharing research in order to enable community design.

There’s a lot we still have to solve in order to realize that ideal. Over the next few blog posts, I’m going to talk about questions like:

  • How do we balance the need to protect user privacy with the need to get the most detailed and accurate data for the sake of science?
  • How can we encourage normal, i.e. non-technical, non-early-adopter, users to participate in Test Pilot, in order to get a more representative sample of Web users as a whole?
  • How do we start to build a design community around Test Pilot, and how do we help them design practical and scientifically sound studies in order to answer their design questions?