Here are my thoughts on things we did well in Ubiquity, and things we should have done differently, and areas for further exploration that we can get into when we start working on the project again. It’s all a bit long and rambly, and the parts aren’t in any particular order, but I want to put it out there to get discussion started.

What did we learn from a solid year of working on Ubiquity? Here goes.

1. This crazy thing can work.

A command line for the web? Why would anybody want that?

Ubiquity at one point had around half a million active users (according to daily server pings). This is amazing for an experiment-quality, proof-of-concept software that presents what is for most people a radically unfamiliar interface. Even in its incomplete, not-ready-for-prime-time state, Ubiquity was clearly providing something that large numbers of people found useful.

We proved that it is possible to square the circle: to create an interface that combines the speed and power of a command line with the learnability of a graphical interface. And we proved that this interface can be part of a new way of using the Web: a task-centric model of browsing where instead of going to a service, users bring the service to where they already are. Both the linguistic interface and the task-centric browsing model are compelling areas for future exploration.

I still hear from people who tell me that Ubiquity is the only reason they use Firefox.

There is an unanswered need out there, both for verb-centric Web use and for fast, smart, auto-completed language-based interfaces.

2. This crazy thing can be localized.

Difficulty of localization is sometimes raised as an objection to doing language-based interfaces. We proved that this difficulty can be overcome. Ubiquity is still to my knowledge the only attempt at an internationalized pseudo-natural-language input system, the first natural language or command-line interface to offer options besides the English-based default.

When we revisit Ubiquity, we need to get some native speakers to eat the dogfood in various languages and report back to us on how natural or unnatural it feels. Localization doesn’t just mean translation: it means making software usable for people in a different social context as well. E.g., for Chinese ubiquity, it would make more sense to offer default search commands for Baidu, Youke, and Taobao (services popular in China) instead of merely offering a Chinese-language version of Google, Youtube, and Ebay commands.

Localized Ubiquity needs to be something practical and useful for users in the target locale, not just an academic linguistics exercise.

3. Having a quick and dirty way to add browser features is hot stuff

Ubiquity’s command subscription model was pretty wacky when we first proposed it. What, it just downloads remote Javascript and runs it with full privileges? Isn’t that a terrible idea?

But it turned out to have many advantages. Unlike extensions, Ubiquity commands let you add features to the browser without having to restart. You can make changes to your code, test them, and deploy them to users without anyone ever having to restart the browser. Additionally, the simpler command-centric, Javascript-and-HTML-based API was able to expose most of the powerful features in the Mozilla platform without most of the complexity involved in learning that platform. All these things put together are a big win for developers: a much faster, easier, and more agile way to develop browser features.

The idea of the Ubiquity command subscription model was spun off into its own project: Jetpack. Jetpack isn’t just an important project in its own right; it’s a crucial part of Mozilla’s future add-on development strategy.

It turns out there are a lot of small, niche, highly specific features which aren’t worth the effort of doing as a full-on Firefox extension, but are worth doing as a Ubiquity command. Or a Jetpack feature.

When we revisit Ubiquity, we’ll be able to build it on top of Jetpack and take advantage of all the effort that has gone into making Jetpack a secure, stable platform. Ubiquity command sets can simply be specializedtypes of Jetpack features.

4. The “hit a hotkey to bring up an overlay” interface suffers from lack of visibility, non-monotony

Something I’ve commonly heard from Ubiquity users: “I keep forgetting that it’s there” / “I keep forgetting that I have a command for this thing” / “I have to stop and think about whether to do something the Ubiquity way or the Firefox way”.

“Monotony” is the quality an interface has when there is only a single way to do something. The name “monotony” sounds like a bad thing, but it’s actually a good quality. There’s only one “middle C” key on a piano. Super Mario Bros. only has one way to jump: the A button. It doesn’t have a “jump” menu item or a “jump” keyboard shortcut. Lack of monotony complicates an interface, as well as taxing your brain by making you decide which method to use each time you use it.

A radical redesign of the Ubiquity interface would be to have, not a hotkey-driven overlay, but a permanent, always-visible box into which you type commands. (There could still be a hotkey to focus input to this box, so using it would be just as fast.)

Once you start going down that road, of course, you have to realize that Firefox already has such a permanent box: the URL bar (and the ctrl-L shortcut to focus it).

An even more radical redesign would be to take away all the interface elements that do the same thing as Ubiquity commands. Remove everything redundant, so the Ubiquity interface is the only way to do things. This sounds scary, but I think it’s worth exploring. More screen space could be given to content instead of chrome, and the interface would be closer to monotonous.

Imagine that there’s a “go (url)” command. Also imagine that we have your history and bookmarks as a nountype, so that Ubiquity can recognize when you’re typing the name or url of a site, and suggest matching sites along with the “go” verb. Ubiquity could then replace, or rather subsume, the current URL bar.

4. Nountypes are very powerful… but hard to get people to use well

When we began Ubiquity, it had no comment of nountypes, nor did command have well-defined argument lists. Some commands grabbed the selection themselves; some pulled information from other sources; some had optional command line arguments which they parsed themselves. It was very ad-hoc, which led to lots of duplicated effort and inconsistent behavior.

Standardizing onto a system of arguments with defined types was a big win. Letting the parser handle assignment of arguments meant less work for command development. The magic word “this” allowed us to treat selections interchangeably with typed arguments. Recognizing potential types of a data input helps us generate better suggestions. And encoding knowledge (e.g. knowledge of how to recognize and parse dates) into a nountype saves us from having to re-implement that knowledge in every command that wants to use it.

As Ubiquity developed we found it advantageous to move more and more functionality out of individual commands into nountypes and/or the parser, in order to keep the code for individual commands as simple as possible. The upside is that our most recent version of the API gives commands a lot of power with very little complexity. The downside is that we had to go through what felt like dozens of iterations of the API (probably really more like 4) in order to get to this point. In our next round of Ubiquity experimentation, we’ll be able to build on this trial-and-error and present a solid, stable, flexible API from day one.

Despite their crucial role in the system, nountypes were mostly invisible to Ubiquity users. This was a chronic problem, because it meant that many command developers remained unaware of functionality available in nountypes, and continued to re-invent the wheel themselves. This also made it hard to focus volunteer developer attention on improving the nountypes, many of which languished in partial completion for long periods of time.

I think all these problems can be solved with better communication about the role and importance of noun types. There is much potential benefit there still waiting to be realized.

5. Naming standards are important… and hard to impose once people get used to just naming their commands whatever.

As I’ve said before, lack of standardization in the naming of commands makes the system as a whole inconsistent and difficult to learn. Technical limitations in the early versions of Ubiquity limited what names could be used, making the problem worse.

We made some progress toward cleaning up the anarchy in the Ubiquity command namespace but there remains more to be done.

Allowing spaces in command names was a big win, as was allowing synonyms, and standardizing on command names that start with a verb – at least in English. In other languages, the ideal naming standard is not yet so clear.

One of the major things that remains to be figured out is how to handle collisions in the namespace, as when two commands want to use the same verb. Several approaches have been discussed, but none implemented yet.

6. Inputs are not just strings

Input to a command can be many things. It can be text, obviously, but it can also be a selection of part of a web page, including HTML formatting and images. It could conceivably be the output of a previous command, or even some form of multimedia. Treating inputs as structured data instead of just strings is a big win because it allows commands to be more flexible in what they do with the data, and allows multiple formats or views to be passed around as a single object. The object can always return a textual representation when one is needed.

We still haven’t figured out a good way to treat completely non-textual data, like images. Image-processing commands remain a large question mark and a potentially fruitful area for exploration.

7. Switching between mouse and keyboard sucks

Not that this is a new problem. But it is definitely a pain to have to select text with the mouse, then switch to the keyboard to invoke a Ubiquity command, then switch back to the mouse to click on a link in a preview.

We don’t have a good solution to this yet. If the promise of Ubiquity is that “type what you want to do” can be a more efficient way to get things done on the web, then it should allow the user to focus on typing, not clicking or dragging. This means we need better keyboard methods for indicating selections on a page as well as for selecting a result (for instance, from a search command which displays multiple results in a preview). I’m a long-time Emacs user; I like never taking my hands off the keyboard. I want Ubiquity to let me use the Web the same way.

We can also go the opposite direction: Ditch the keyboard, focus on the mouse. We did develop a Ubiquity context menu, which enabled a user to get at their commonly used commands without taking a hand off of the mouse. This feature remained underdeveloped and little-known relative to the command-line based Ubiquity.

There was a lot of interest in mouse-based Ubiquity when I visited Mozilla China, because Chinese internet users are generally much more mouse-oriented and less typing-oriented than Americans.

But even outside of China, it’s worth investigating what a mouse-only Ubiquity would be like. It wouldn’t have the natural language aspect, obviously, but it could still offer the benefits of task-centric / verb-centric browsing.

8. Security is critical… or is it?

We put a big red scary warning page in front of the command subscription button, and left the rest up to users. Although we had many discussions about security models for Ubiquity, we never implemented any of them. We basically punted on security in the first round.

Jetpack has the same inherent security vulnerabilities as Ubiquity. The difference is that in Jetpack, they are being addressed. The Jetpack team is putting much effort into implementing innovative new security models that will make it safer to run arbitrary, remotely-hosted Javascript modules as part of your browser.

I’m not aware of a single malicious Ubiquity command actually being spotted in the wild thus far. They remain a theoretical threat. So it may be the case that putting a lot of work into security in the early stages wouldn’t have been the best use of our time.

When we do Ubiquity again, we will build it on top of Jetpack and be able to take advantage of the Jetpack security work. In retrospect, maybe punting on security was the right decision after all.

9. Looking at a preview of command execution can be more useful than actually doing it

When we started out, the command preview was meant simply as a way of showing a documentation string about the command the user was highlighting. As we went on, we started using it for more and more, rendering dynamic web content in the preview area to show the user exactly what would happen if they hit the enter key.

Eventually we realized that actual execution of the command was often not needed, unless the user wanted to open a new page or insert something into the current one. If their desire was only to see some information, the preview alone was enough.

This trend was most noticeable in the Map command. The Map command was really cool, but also totally inconsistent with how the rest of the commands worked. Using a lot of custom code, Map treated the preview itself as an interactive embedded web app, allowing the user to do things by clicking links and dragging images around in the preview. The concept of “execution” started to get a bit fuzzy, to the point where many users I talked to didn’t realize that the Map command even had an execution function.

Having one of the standout commands work so differently from everything else is confusing for new users. The simple answer would be to make the Map command work like everything else. But it would be interesting to explore what Ubiquity would look like if we took the interactive-preview approach to its logical conclusion and applied it to all commands.

If this approach is taken too far, we can end up with commands that look like whole applications implemented inside modal dialog boxes, which I don’t think anybody wants. For commands that want to offer interactivity in their preview, perhaps a better way would be to open a Jetpack-style sidebar. At any rate, we’ll need to figure out where to draw the line on complexity of previews, and on what should be in a preview vs. what should be part of command execution.

10. “Killer App” commands drive adoption

We spent so much time improving the Ubiquity core that we didn’t spend much time working on new commands, leaving that to the community. But now I think we might have been putting the cart before the horse. It’s not the system that is valuable to users, it’s the individual commands, and the time they can save. I hear again and again from users I talk to that there are just two or three Ubiquity commands that they use constantly, and the rest not at all. Of course, every user names a different two or three commands.

The lesson is that users are unlikely to be interested in Ubiquity if it’s seen as an alternative interface that’s different just for the sake of being different. They need a compelling reason to install it and learn it, and the most compelling reason is if there’s a few commands, or even one single command that, they find extremely useful for their personal workflow.

11. Users need to be able to find commands somehow

Many of the third-party commands that got written for Ubiquity duplicated the same functionality. The same things were getting written over and over again. I think it was easier for a devleoper to re-implement a command than to discover that there was already an existing implementation. There was a lot of duplicated effort going on simply because there was no good way to discover existing commands.

Most Ubiquity users won’t be command developers, only users of other people’s commands, and for them it will be even more essential to be able to find the specific commands that meet their needs (see “Killer App commands drive adoption”).

We had plans to write a command search engine, but they never quite came together. “The Herd” was a server-side process which collected information about the commands that people had subscribed to. One of its functions would have been to act as the backend for a command search engine. This worked as a proof-of-concept, but it didn’t scale up. The couchDB instance storing the data was not able to handle the load, and spent more time in a crashed state than it did running.

When we revisit Ubiquity, I’d like to make the command search engine a first-class feature, not an afterthought. I know that one of our community members has been working on a re-write of the Herd, which could be the crux of this effort.

12. There are too many directions for Ubiquity to grow in; we can’t follow up on all of them.

Voice recognition! Ubiquity in Thunderbird! Chaining commands together! Reaching out to the desktop to launch other applications!

Each one of these would be a cool project, but each one is also difficult and time-consuming to build. Which ones are highest priority?

Volunteer contributors will naturally hack on whichever feature they personally find most compelling. This is great! But the danger is that without an overall vision for the project clearly defining which features are important, we miss the forest for the trees. We could end up with a collection of amazing features bundled into an overall weak user experience.

Conclusion

There you have it: this post is a rough draft of the final Ubiquity retrospective report that I’m working on, which will eventually be published to the Labs website in its final form I would love to hear your thoughts on Ubiquity as well. I hope that others involved in the Ubiquity project will share their own thoughts about what we did well and what we should have done differently; if you write a post on the topic, please send me the link (by email or comment thread) so I can put together a collection of posts.

Thanks a lot!