Test Pilot is a delicate balancing act.

On one side is user privacy. I can’t overstate how committed we are to protecting the privacy and anonymity of our users, keeping them fully informed, and getting their consent before sending any data about them back to Mozilla. We practice full disclosure, collect nothing without an explicit opt-in, and let you review exactly what will be uploaded, in human-readable form, before you upload it. We never associate any of your uploaded data with your name, email address, or any other personal data about you. We’ve been significantly more conservative about data collection than most other organizations that do this kind of research.

On the other side are the needs of scientific research. We want to collect the most detailed and accurate data possible and share it with the researcher and UI designer communities, in order to try to improve the state of not just the Firefox interface but of web interface design in general. There are certain burning usability questions which would be much easier to answer if we were to collect certain kinds of personal information about our users.

For example, one of the questions that the Mozilla UX department is curious about is this: how often do people have duplicate tabs open? There’s anecdotal evidence that some web users end up opening so many tabs that they forget what they already have open. Instead of switching back to their Gmail tab, they might just open up a new Gmail tab. If this is widespread, it’s an indication of one way that the Firefox tabbed browsing interface is failing to meet users’ needs. Perhaps we ought to have some feature that detects when you’re about to open a duplicate tab, and redirects you to your already-open copy of the tab instead.

Do you know what would be a really easy way to do this research? We could include code in the (currently ongoing) Test Pilot tabs experiment that records the URL of each opened tab. We could look through that data on the server side and automatically identify duplicate tabs.

But whoa, whoa, whoa, hold on there! The URLs that somebody chooses to visit with their web browser are just about the most personal thing about them. It’s one of the most useful things for UI research, but it’s also one of the most private. I know I have URLs in my browser history that I would think twice about uploading to Mozilla, even if they were not associated with my name in any way.

I think that collecting every URL the user visits is out of the question. We made a decision that the current Tabs experiment would never collect URLs, and we repeat this statement over and over throughout our documentation in order to reassure people that we’re not trying to spy on them.

But that doesn’t mean we can’t detect duplicate tabs. We just need to be a little more clever about it. It’s the duplication that we care about, not the URLs, so let’s detect duplicates on the client side. When we send up the data, we’ll include a field that says whether or not an opened tab is a duplicate, but we won’t include the URLs.

This points the way to a general guiding principle: Whenever there’s sensitive data, process it on the client side, and upload only non-sensitive calculations derived from the data. By following this principle, I think we can resolve many, if not most, of the conflicts between privacy and the quest for knowledge.

Here’s another example: Many users install extensions that make their tabs work differently. Addons.mozilla.com hosts over three hundred extensions that affect how tabs behave.

Therefore, we can’t just throw all the tab usage data into one big bucket and then start doing statistics on it. We’d reach incorrect conclusions because that data would be from a mix of Firefox-default tab behavior and customized tab behavior. We’ll have much better results if we separate user data according to what extensions they have installed.

(Even better, we can draw comparisons between the Firefox-default tab users and the users with Extension X. What if we find that on average the users of Extension X make fewer tab-related errors, or spend less time interacting with their tabs, than vanilla Firefox users? That would be very interesting, eh?)

So we’d like to see what extensions each user has installed. But again, there is a privacy problem. There are some extensions you might not want us knowing that you have (even though it is, again, not associated with your name or email address or anything else). For example, what if you’re running an extension which is not public knowledge? You could be developing something for your organization which is proprietary, or under NDA, and you don’t want the world to know it exists yet.

Her’e what we do: We take the ID of each extension that you have and put it through a one-way hash function. We upload these hashes. They’re not readable, but they are vulnerable to guess-and-check. So if we’re looking for a known extension, like say Tabs Mix Plus, we can put the ID of Tabs Mix Plus through the same hash function and then check for the presence of a matching code. In this way we can divide data based on a limited set of known extensions; but your top-secret in-development extension that is not yet public knowledge is nothing but an opaque hash string to us.

The final question has to do with how we make the data avaiable to the research and UI design community. As I mentioned in a previous post, this is an important part of the plan for Test Pilot. What we’ve said is that we will make the data available, but only in an "anonymous, aggregated" form, one that does not contain information about any individual user from any study.

In other words, when we publish data, we want to be even stricter about it than we are with the data we collect. We want to publish only the numbers that describe the Test Pilot user group as a whole, without including any numbers that describe any one particular Test Pilot user.

What exactly that means, we are still figuring out. What will this anonymous, aggregated form be? How will we generate it, how will we make it available, and how can we make sure it’s still useful for researchers (who may be working on questions that we haven’t even thought about yet)?

As always, I would love to hear your thoughts on these matters.