Since the fall quarter of last year, HCI student and Ubiquity community contributor Zac Lym has been doing good work testing Ubiquity on new users. Although the study included only a small number of users, it’s an important one since it’s the first and only rigorously derived usability data we have on Ubiquity. The users in this study had no prior exposure to or preconceptions about the Ubiquity interface, and the experimenter gave them no help using it; so the problems and frustrations they encounter give us insight about the specific areas where Ubiquity needs improvement in discoverability and learnability.

The primary material resulting from the research is a series of videos of users in action. Zac has made these videos available on the Mozilla wiki along with write-ups of his methodology and his findings. These findings are well worth reading in full for anyone interested in improving the Ubiquity user experience.

A confession: The videos from the tests are painful for me to watch! If I had been there in person, you probably would have had to physically restrain me from correcting the test subjects. Even watching the videos is making me gnaw on my fingers.

angry1

Times like this I have to remind myself why I called this blog Not the User’s Fault. There’s a strong impulse to defend one’s own work, and therefore to blame the user for their mistakes. (What are these people, stupid?) This impulse is to be resisted. If we get defensive, we can’t learn anything and we can’t improve our UI. When so many people, even following the tutorial, can’t figure out what to hit to bring up the command line, and when they make basic conceptual mistakes like typing a ubiquity command into the Awesome bar, and when they give up in frustration before discovering anything useful, it’s because we haven’t presented the interface in a way that gives people the right idea about how to use it. That’s our fault.

To understand what went wrong, let’s examine the biggest barriers to learnability. By “biggest”, I mean the ones that tripped up the greatest number of people. According to Zac’s tests, these included:

  • Problems with the tutorial page: Some users confused by the layout or the heavy use of jargon.
  • Problems with the tutorial video: some of these were technical problems — the video won’t load, or loads but has no volume, etc. Among people who managed to see the video, many were confused by its jargon, bored by its length, or set up for disappointment by its grandiose claims.
  • Difficulty discovering the right combination of keys to press to bring up Ubiquity. Some users gave up here, never getting any further. Others confused themselves further by accidentally changing the hotkey settings on the about:ubiquity page. (It’s apparently easier to change this by accident than I thought).
  • Difficulty understanding the basic interaction model of the interface, i.e. “bring up this box and type a verb-noun command into it”. From Zac’s post: “Every tester interacted with the Awesome bar looking for Ubiquity, thinking it was Ubiquity, or for reasons other than putting in a URL.” And three users made the opposite mistake and typed URLs (e.g. “mail.yahoo.com”) into the Ubiquity command line.

Other recurring problems too were mostly related to specific commands. While command-specific UI failure is important, the problems listed above are problems at the most basic level, and will trip up users long before the users make it far enough to encounter command-specific problems. So I think it makes sense to focus first on the category above: what might be termed “first-run” problems.

Zac has blogged about the root causes of the usability problems that he found, and has also come up with a list of recommendations for fixing them. Of his suggestions, I especially like “Merge with the Awesomebar” (which we plan on doing), “use marketing to teach the interface” (which I blogged about here), and finally “Make a killer first-run page”. I think the first-run page can be a huge help, if we use it as a teaching tool. Where do we start?

Our current “Help” page — the first thing that users will see after installing the extension and restarting the browser — is nearly unusable if you’re not a Ubiquity developer. I’ve become so familiar with the page that I can look at it without really seeing it, so I had to make an effort to look through fresh eyes. Here’s what I saw:

ubiq-firstrun

The vast majority of the page (circled in red, with Xs) is developer-centric information or advanced features. Sure, we should have a home page for developers, but we shouldn’t send newbies there, nor should it be the result of typing “Help”. The three or four things on this page that are useful to new users (circled in green — the video, the tutorial link, the command-list link, and the place where you set the keystroke to use) should be the only things users see upon first install.

That would be a start. Then we could move on to improving the video and improving the text tutorial page. The video that’s currently there is more of an advertisement, or a PowerPoint presentation that Aza would give at a conference to promote Ubiquity; it’s not at all suitable as a training video. The text tutorial is full of information, but it’s in a form that makes it hard to digest: A giant wall of text, like an instruction manual.

What I’d really like to see is an interactive tutorial, like the training level in a real-time strategy game. Upon installation, you’d get a page which simply tells you to try hitting a certain key combination to bring up Ubiquity. It would do nothing else until you had successfully done so. Once the command line was on-screen, the tutorial would present further instructions about what to do next. After it had taken the user through the basics, it would teach them a few useful commands to try out, and tell them where to go when they need more information; then it would gracefully disappear. Surely, something like this would not be too hard to rig up with a little javascript and canvas, inside a chrome page, communicating with the extension.

Once we’ve done that, we can run another round of new-user testing. I hope that with a better first-run experience, fewer people will be confused or give up in frustration, and we’ll be able to observe users going further — the better to collect observations on the next set of usability problems, of course!

A parting thought: The great majority of our Ubiquity work has been developer-facing, i.e. adding features for use by command developers. This is only natural, since we’re still pre-version-1. However, we’re building this thing out in the open. It doesn’t matter what we call the version number: People are already using the software. So far Ubiquity has been adopted by a self-selecting audience of web geeks and self-described “power users”, people who are willing to invest some effort in learning a new interface. Ubiquity may be meeting the needs of these early adopters, but that’s not enough. I fear that if we continue too long in this developer-focused vein, without improving the user experience, we risk entrenching Ubiquity as “Power users only” and being forever stuck on the wrong side of “the chasm”.

We should look for ways to improve its learnability, without sacrificing its power, in order to make Ubiquity into something useful for a wider slice of the Firefox user base.