“Why did my Firefox suddenly go from 3.6 to 4.0 to 5.0 in the space of half an hour?” asked my friend on the phone, in the middle of what was otherwise a conversation about role-playing games.

“Because the automatic updater system is working?” I said.

“Was Firefox 4 really so broken that you had to kill it after three months?” he asked. Except he used a ruder word than “broken”.

“No, not at all”, I said, remembering the giant launch party we had had for Firefox 4. It did feel kind of like we told everybody “Firefox 4 is the greatest thing ever, and it’s finally here! Download it now! OK now it’s obsolete already.”

“It’s just that we’ve got this new rapid-release schedule going”, I continued, “where we push out the new releases every three months, so that users always have the latest bug fixes and speed improvements and stuff.”

“OK, well, it says that Perapera-kun (the extension I use to translate Chinese) is no longer compatible. I really needed that extension.”


Some backstory:

Firefox 3.6 was released on my 30th birthday – January 21, 2010. Firefox 4.0 was released on March 22, 2011: 14 months later.

It was very frustrating both to our developers and our users that bug-fixes, speed-ups, and features that were completed in, say, February or March of 2010 had to sit unused on the shelf until every part of the the enormous monolith called Firefox 4 was ready to launch. By early 2011, users of Firefox 3.6 were complaining about problems that had already been fixed for a year; but they didn’t have the fixes because our monolithic release schedule didn’t let us give them the fixes.

Our monolithic release schedule also meant we couldn’t respond quickly to the rapidly changing and highly competitive browser market. Chrome and IE aren’t stitting still. There was a feeling within MoCo that the fourteen-month wait between Firefox 3.6 and Firefox 4.0 could have killed us competitively. Almost did kill us.

Given that, moving to a rapid release schedule — releasing a new Firefox on a regular, frequent schedule, just to give users whatever improvements we have for them, even if it’s not a huge splashy new release — seemed like pure win. It took a lot of effort inside the company to make that transition. We had to really focus on it, pull together, overcome many obstacles. That we shifted over to a whole new development process and still shipped Firefox 5 on time, even though it was just a tiny incremental update, felt like an enormous victory.

And yet… People on the Interenet are calling the new release plan a kick in the stomach, saying we’re committing corporate suicide, saying that they feel “suckered” and that “Mozilla is creating something you can’t rely on” (comment thread). They’re even making us into Hitler. (Hitler in his bunker, no less — the meme used to indicate not only that someone is the bad guy, but that they’re losing and in denial about it as well.)

When I first started reading some of this backlash, I was puzzled. Were people really getting upset because we released Firefox 5 so soon after Firefox 4? Isn’t releasing rapidly a good thing? Don’t they want new bug fixes and speed improvements as soon as possible? Yes, we stop supporting older versions, but it shouldn’t matter, because everybody should automatically be upgraded to the latest version, right?

Or were they complaining that there was too little change to justify a version number bump? Did people really care that much about version number schemes? (As a rule, I don’t pay much attention to debates over version numbering schemes, because they bore me to tears.) If we had released the exact same thing but called it Firefox 4.1 would they have been happy?

In diagnosing usability problems, we often have to look beyond what users say they want in order to perceive the underlying problem. We want to treat the disease, not the symptoms.

In this case, the symptom is the complaints about the pace, and the rapidly-increasing version numbers, of new releases. But new releases are good, and version numbers are just numbers. What’s the disease?

We thought that the new rapid release process was pure win. We underestimated two things:

The number of extensions that are breaking each time we raise the version number. And the number of people who need a stable, long-term-support version of Firefox.

First, many extensions — like my friend’s favorite extension, the Chinese popup translator PeraPeraKun — break each time a new release comes out. Faster releases means faster breakage, which means more effort for developers and for users to keep their extensions compatible.

There are two causes of extension breakage. An extension may break because we changed an API it relies on, meaning it won’t work on the new Firefox version until the extension developer goes in and rewrites the part that uses the API. But an extension may also break just because of the version number changed, even if there is no API change. This is because extensions have metadata files specifying which versions of Firefox they run with; if your extension’s metadata file says “I work with Firefox 3.5 – 4.0” then as soon as 5.0 comes out, your extension will be marked as incompatible, even if there is no API change. So an extension can lose compatibility for bureaucratic reasons — its “paperwork” isn’t up-to-date — even if there’s absolutely nothing wrong with it. It would work if that file was changed (or if add-on version compatibility checking were turned off). But an end user has no way of telling the difference between a “real” breakage and a “paperwork” breakage.

This system made sense back when every new major version number was fairly guaranteed to have API changes. In the new rapid-release world, the version number is now changing every few months, but API changes are no more frequent than they were before. API breakage is a problem that existed before; the rapid-release schedule hasn’t made it any worse. But the number of paperwork breakages has skyrocketed under the rapid-release schedule.

We can say that “80% of extensions remain compatible” but that’s hardly consolation if the one extension you rely on is not part of that 80%. Ideally its developer should be updating it; but what if its developer is nowhere to be found? What if its developer got fed up with having to change the metadata file every six weeks and stopped maintaining the extension? Maybe that one extension was the only reason you were using Firefox; then what? You could try to go back to your old version of Firefox that still works with the extension… if you can figure out where to download it, and if you can figure out how to turn off auto-updates. We don’t make either of those things easy. Even if you succeed, you’re now stuck on an obsolete browser version and you’re not getting fixes for the latest security vulnerabilities. This is a problem.

I think that long-term, we ought to move away from the whole concept of a “maximum compatible version number” in a metadata file. We should try to provide a compatibility layer with a guaranteed-stable API, so that even if we change the underlying implementation, extensions written to the compatibility layer will still work. That’s one of the goals of the Jetpack project.

The second thing we underestimated: the number of people who need a long-term-support version of Firefox. By “need” I mean that their choice is between using a long-term-support version and not using Firefox at all. Imagine being the IT manager for a large organization, a school for instance; you’d like to keep your browsers up-to-date but you have a lot ofX other things to worry about, and any major change to browser functionality means you’re going to have to re-write documentation, re-test your intranet web-apps, re-install any extensions you rely on, and re-train your users. All of these things cost you time and money. You want a browser that can be relied upon to behave consistently over time. Your ideal browser is one that automatically gets security updates, but not other kinds of updates.

If Mozilla isn’t going to provide a browser like that, you’ll look elsewhere. Simple as that.

Maybe in addition to “nightly”, “aurora”, “beta”, and “final”, we ought to have a fifth update channel. Call it “long-term stable” or call it “security updates only”; whatever we call it, there’s a huge demand for something like that, and we’re not providing it right now.

We need to start by admitting that we screwed up. We have to listen to our community when they tell us what a serious problem this is for them. We have to stop treating complaints about version numbers as if they were just a misunderstanding; yes, they’re “just” version numbers, but under the smoke of the version number argument there is the fire of a real problem. Or rather, two problems.

I still think that moving to rapid releases was a good idea on the whole. It has a lot of advantages. But like many big changes, it’s solved one big problem while creating two smaller problems. We need to face those problems and deal with them. I’m confident we can solve them, but first we need to stop pretending that they don’t exist.