December 8, 2009
We’re trying something new with the “Week in the Life” Test Pilot study. Instead of running just once per user, it will automatically recur about once every two months (60 days, to be precise) and run for one week each time. The idea is to let it recur over the course of a year, and see whether we can detect any long-term trends in the data that would indicate user habits changing over time. For instance, a lot of us who work in web browsers have a hunch that our users are using bookmarks less, and tabs more, compared to a few years ago. But does the data actually support this?
Because the “Week in the Life” study recurs, and because we never submit any data without the user’s explicit permission, we’ve got a potential user experience problem: Test Pilot is going to ask you whether you want to submit the data from the study every time it recurs. Do you want to submit the data? Do you want to submit the data? How about now? How about now, huh? How about that data, do you want to submit it?
Isn’t it annoying being asked the same question over and over again?
To mitigate this problem, I’ve added a new UI widget to the Test Pilot status page:
The principle of “Don’t pester the user” is important, but so is the principle of “Make sure you have the user’s permission before doing anything with their data”. These principles are natural enemies. Finding a compromise between them is not easy! I know that my little drop-down menu is not a perfect solution. What do you think? Is it self-explanatory enough? Too wordy? Is there a better approach to this problem?
October 1, 2009
We’ve had over 5,000 users submit data from the Test Pilot tabs study!
Considering that people had to first hear about Test Pilot, then opt in by installing the extension, then opt in again by choosing to submit the data, 5,000 is a really good number. Better than we had any right to expect, certainly.
Over the past couple weeks, I’ve been sifting and analyzing the data, and working with Blake Cutler from the Mozilla Metrics team to generate graphs of interesting statistics about tab usage. I’ve just put up a results page showcasing several of these graphs.
We’ve also posted samples of the aggregated data which are free for anyone to download and use. There was some discussion on my previous post about how to aggregate the data in a way that was still useful to researchers. What we ended up doing was building files that include row-level data from a random subsample of the users that fit particular criteria. It’s stripped of any information on the language/locale, operating system, or installed extensions for any individual user in the sample.
Third-party researchers have already begun using the data to do their own analysis! Andy at Surfmind.com has a post containing some very cool-looking visualizations and has proposed an interesting theory about there being two classes of heavy tab users.
September 25, 2009
I get a lot of people contacting me by email, IRC, forums, or blog comments to say that they’re “worried that if I join Test Pilot I’ll skew the data” because “I’m sure that my tab usage is atypical”.
People! Don’t worry about being an atypical user!
First of all, we have already had almost 5,000 Test Pilot data submissions. One outlier isn’t going to do much to “skew” a data set of that size.
But more importantly, you shouldn’t assume that you’re abnormal. We don’t know what “normal” tab usage is! That’s why we’re doing this experiment, to find that out. If we started out with an idea of what normal tab usage looked like, and threw out things that didn’t match our preconceived notions, that would be a clear case of experimenter bias. Then we’d really be skewing the data.
For instance, I was surprised to find out that there are users who have over 500 tabs open at a time. Over 500! They’re surely outliers, but they’re not abnormal users &emdash; they’re just users. That number isn’t skewing the data &emdash; it is the data. Thanks to those users’ participation, we now know that having 500 tabs open is something that people do with Firefox, something we might not have known otherwise.
As I said in a previous post, I do believe we have a major oversampling of the power-user / early-adopter demographic in our current Test Pilot user base, and that we need to work on fixing this by reaching out to a wider sample of users. But note that word: wider. Excluding yourself because you think you’re atypical isn’t helpful. If you really want to help our sample — and I’m touched that so many of you do want to help " the best thing you can do is to let your less-techie friends know about Test Pilot.
March 5, 2009
Since the fall quarter of last year, HCI student and Ubiquity community contributor Zac Lym has been doing good work testing Ubiquity on new users. Although the study included only a small number of users, it’s an important one since it’s the first and only rigorously derived usability data we have on Ubiquity. The users in this study had no prior exposure to or preconceptions about the Ubiquity interface, and the experimenter gave them no help using it; so the problems and frustrations they encounter give us insight about the specific areas where Ubiquity needs improvement in discoverability and learnability.
The primary material resulting from the research is a series of videos of users in action. Zac has made these videos available on the Mozilla wiki along with write-ups of his methodology and his findings. These findings are well worth reading in full for anyone interested in improving the Ubiquity user experience.
January 28, 2009
Posted by jonoscript under Uncategorized
| Tags: research
I saw a great presentation yesterday by Patrick Dubroy, who presented some preliminary results from his research on how people use tabs in the real world.
[UPDATED: You can now read Patrick's whole presentation on his blog.]
The sample size was small (22 people) so we should be cautious about any conclusions we draw from this study, but it’s pretty exciting to have any real scientific data at all on this question (as opposed to anecdotes, personal observations, etc.) The study also points in some exciting directions for further research.
The big thing I took away from Patrick’s presentation was that among heavy web users, tabs are enabling new styles of browsing behavior that rely less on bookmarking and less on the back button. According to Patrick, the back button is getting used less and less as the years go by, and for all but two of his test subjects, switching tabs was a more common action than hitting “back”.
This change seems to coincide with the rise of web applications, where a user might spend a long time interacting with data on a single page. Web applications live comfortably in tabs (I always keep gmail open in my leftmost tab, for instance) but did tabbed browsers help popularize web applications, or did web applications help popularize tabbed browsing, or neither?
One anecdote from the presentation really stuck with me: apparently some users love opening multiple links in separate tabs, but they use a laborious manual workaround to do so, because they don’t know about command (or control) -clicking a link to open it in a new tab. This tells me that the feature needs to be more discoverable somehow.