In light of the idea that openness is hard work and that improving the discovery path for new contributors is a constant struggle, I’d like to do something to help make Test Pilot a little more discoverable, and possibly gain some useful feedback as well.
Why not publicly post the code for upcoming Test Pilot studies? The Mozilla community can check it out and verify for themselves that we’re upholding our promise not to collect any sensitive or personally identifiable data. They can also point out anything that’s potentially wrong with the code, things that we’re overlooking that might make the collected data less accurate, etc. That sort of feedback would be very useful for us developers.
Of course, it’s not like we’ve been keeping the Test Pilot study code secret up until now, or anything. They’ve always been, and will always be, available through our public Mercurial repository. (Look in the /testcases/ directory). But that’s not exactly easily discoverable.
So, I’m going to start announcing upcoming studies on this blog and linking directly to the code, to make it easier for interested parties to offer feedback.
The next study we’ll be releasing is called the Firefox 4 Beta Interface Study v2. It’s a revised version of the earlier Beta Interface study that we ran near the beginning of the Firefox 4 Beta program. It’s been updated to include instrumentation of the new features in the beta, such as Sync, Panorama, and App Tabs; we’d really like to know how many people are using these new features, and how they’re using them – e.g., how often do people sync? If they use app tabs, how many app tabs do they have? If they use Panorama, how many tab groups do they use? And so on. We’ll also continue tracking the same things that the first version tracked – frequency of use of menu items and toolbar buttons. That way, we can look at how people are using the toolbars and menus now,
compared to how they used them earlier on in the beta cycle (summed up in this heatmap), and see if there are any trends there.
Of course, as always, we’re not collecting any URLs, search terms, names of sites, or names of bookmarks. You don’t have to take my word for it; take a look at the code and see for yourself!
You can read the latest Firefox 4 Beta Interface Study code on the web via the Mercurial repository.
Or you can take a look at the Bugzilla tracking bug for the study; the code is attached to the bug.
Finally, the code for the study is also mirrored on GitHub, so feel free to look at it there if you prefer to use GitHub’s code review tools.
So pick your method and take a look at the code; comments, questions, and criticism welcome.