Interview with David Ascher and Dan Mosedale – Thunderbird

Interview with David Ascher and Dan Mosedale – Thunderbird – Mozilla Messaging

Interviewers: Scott Swigart and Sean Campbell

Interviewee: David Ascher, and Dan Mosedale

In this interview we talk with David and Dan. In specific, we talk about:

Sean Campbell: Tell us a little bit about yourselves and give us a little background on Thunderbird.

David: I was an academic until I decided to get involved in software. Although I got started in software as a kid, it was about 10 years ago that I decided to do it professionally.

One of the first things I did was to start working for a company called ActiveState, which built commercial products and software for the open source stacks, including the Mozilla code base. That is when I got involved with Mozilla as an outside contributor. I was a community member, but working in a closed source shop building products for developers.

I got involved in leading the Thunderbird project about a year ago, at a stage when Mozilla was very interested in generating enough energy and enthusiasm around Thunderbird to build something exciting and competitive and forward-looking.

I jumped at the opportunity, and I have been swimming in the deep end of the pool ever since. I should also mention that even though I was in a closed source shop before, I have been involved in open source for more than a dozen years, primarily in the Python world. I also do a lot of talking with other people in open source foundations and open source projects, so I have a hybrid perspective.

I had actually worked at Netscape for quite a while, initially in IT and then later, I helped get Mozilla launched. I have been involved in it in one way or another ever since.

Netscape was definitely a proprietary shop, but it had always been very interested in open standards. With Mozilla, we felt like there were a bunch of us who had a little bit of experience in open source in various ways.

We wanted to see if we could get the world at large involved in browser development. I was always especially interested in the mail client.

Over the years, I have been funded by various organizations to work on Mozilla. For while I was leading the Calendar project, and more recently I worked on Firefox. When David signed up to do the Mozilla Messaging project, I offered to help. I am very excited about this space as a whole.

I think there are a lot of ways in which the whole market is underserved, and we would like to address that.

Sean: The latest numbers I have seen place Firefox as having about 20% of the market share, although I’m sure you have more specifics.

What kinds of things do you look back and think you want to do it the same way as the Firefox effort, and where do you see opportunities for improvement?

David: One issue that comes to mind is the realization that although it is an open source effort where we can distribute authority and responsibility, that doesn’t mean that everybody is equally competent in all areas.

One of the things that Firefox has implemented is the notion that, just as there are review processes for code changes, there should also be review processes for other kinds of changes.

For example, security bugs get reviewed by security experts. Code changes that have an impact on how easily the project can be internationalized or localized get reviewed by experts in those areas. User interface changes get reviewed by user experience specialists.

I think that is one of the ways in which Firefox has been able to really identify ways to increase the professionalism of the organization as a whole, as well as delivering ideal, optimal user experiences while still letting people participate in a variety of ways.

Even if you are not a user experience expert, you can contribute ideas and get usability people to comment on them. In general, that combination of decentralization and specialization is one of those lessons that we are definitely applying to Thunderbird.

There are lots of ways that we are deliberately mimicking the Firefox model, such as the way we do release driving, how we coordinate with localizers, and a large part of our toolset. We leverage a lot of the infrastructure they have put in place.

Dan: Another aspect of that experience that has been very helpful as we experiment going forward is the project management stuff. Over the years, the Mozilla Project has tried a bunch of different methods to make things happen.

Interestingly enough, it always seems to come back to our bug tracking system. I often observe that that’s a pretty painful way to carry out project management, but a lot of the other ways seem to be worse.

With the upcoming release of Thunderbird, we are going to make the process very, very bug driven and not driven from the top down as much as from the people who are working on the various pieces.

Sean: Let me ask you about the obvious elephant in the room–the closed-source Outlook client. What lessons do you draw from watching that development process or just even the feature set of Outlook as you guys look at building Thunderbird? Are there features in Outlook that you like?

There are obvious differences in the closed versus open development model, but what about after you peel that back one more layer? I have to imagine you guys occasionally look at Outlook and go, “Gosh, I really like that.” Or, “No, I really don’t like that and thank God we can make a change here.”

David: I think that is true for Outlook. But it’s also true for Mail.app. It’s also true for Gmail.

Sean: Sure; the only reason I mentioned Outlook directly is because it has a fairly dominant market share.

Dan: One of the biggest things that Outlook has done is to help frame the messaging market in terms of having calendaring included. That’s important, because it really is just another way that people communicate.

Sean: That’s true–calendaring has not always inherently been part of your mail client. Where do you see that going in the future?

David: Calendaring really fits into our general approach to how we think about features. There is always tension between which capabilities should be part of the core product and which ones should be part of extensions, for example, or add-ons.

The Mozilla core has a very nice add-on capability, supported by an add-on ecosystem and an add-on community that can help fill in parts of specific use cases that we don’t handle and that sort of thing. In general, though, if something is going to be used by a very large percentage of the user base, then it makes sense to have it as part of the core product.

Many people see the integration of calendaring as a core capability, and without it, you lose a lot of the utility that people now expect from messaging clients. There are other bits that may be more optional, such as integration of instant messaging.

That could change as the market and user expectations change, of course. What we try to do is to make a system where we can evolve a core feature set by bringing in features or experiences as add-ons so that there can be lots of iteration and lots of innovation without tying it too closely to the core product. That still gives the option at some point to bring those features and capabilities in.

The Lightning extension is a good example there. Figuring out how to do calendaring well is a huge problem set. The interoperability problems are huge. The calendar team has done a great job of building this extension on top of Thunderbird, and I think it has reached the point of utility, where it makes sense to bring into the core product.

We want to do that well, so that for people who don’t care about calendaring, it won’t get in the way. I think we can provide some leadership here–not necessarily the same way Outlook does it, but maybe better, maybe different.

We are currently figuring out how that integration will take place, but we are pretty sure it’s going to happen pretty soon.

Sean: How do you guys handle the process of security reviews, in terms of threat modeling or the different elements of your security process?

Users are pretty willing to shove anything in each other’s inbox, including proprietary documents and other things that they would not want to have leaked from the company.

Dan: Part of the answer is that we are still sorting that out. We are still very much getting spun up and growing as a project, so that is actually one of the things that is on my ‘to do’ list for the upcoming weeks, is to nail some of that down for Thunderbird 3.

Thunderbird makes some fundamentally different assumptions from Firefox. For example, by default it has JavaScript off in any HTML messages that you get. At the same time, though, there is going to be a similar process to what Firefox does, which essentially has some threat modeling and as well as code and design review.

There are also slightly different ways that users interact with a mail client, as opposed to a browser, so we will tend to observe things from that angle a little more and probably make some different choices as a result.

Scott Swigart: One of the things you talked about in your session at OSCON is that compared to a lot of open source projects, you really have to care about usability and people who can’t submit a patch to get a feature in which they are interested.

You mentioned before that you are really working on having bug-driven project management. How does that work from the perspective of an end user who is interested in getting something added but who doesn’t have the ability to code it up? What are the ways that they can interact with the project?

Dan: There are a bunch of different possibilities there. Certainly, a lot of them have to do with how much time they have and their level of technical sophistication. Working with us in Bugzilla, simply filing bugs that they find, is tremendously useful.

At the same time, there is a much larger class of users for whom the bug process isn’t all that well suited. When I said that I think having the process Bugzilla driven is a good goal, that refers mostly to the engineering point of view.

I think in some ways we need to explore other avenues. I know David has given a lot of thought to this idea of interacting with the less technical community.

David: One of the challenges we have is that by our competitors’ standards, we have a tiny, tiny staff–fewer than 10 employees covering the entire development and marketing process.

That means we have to rely on volunteers and partners to provide a lot of the work for a bunch of other things. One of the interesting areas that we are trying to explore right now is how to improve the QA process, because we’re not going to have 30 or 40 people who are trained QA engineers go through and make sure that Thunderbird works the way it is supposed to.

We are exploring ways of getting some of our early adopters and enthusiastic users to help test the software. The world of web apps has show that you can engage very large numbers of users in distributive processes of collaboration and quality assessment, and we are trying to make that work for Thunderbird users.

The answer to the broad question as to how non technical users get involved is hard to answer generally, because there are lots of different ways, and some of them will make more sense for specific users than others.

So for example, we have lots of localization teams, people in foreign countries who want to help make sure that Thunderbird is available from day one in their language and adapted to their locale. There is a process for that.

If there are people who have graphic design skills or marketing skills or other skills, there are a whole bunch of different avenues, depending on people’s backgrounds.

Scott: What about the relationship between an open source project like Thunderbird and downstream distros?

In other words, what do the downstream distros do with the project? If people in the Thunderbird community file bugs against the distro, do those get pushed up and the distros know when you guys have made changes?

Talk a little bit in general about the interaction between your projects and various distros.

Dan: I think you have actually summarized it reasonably well. In terms of the particulars of Linux distribution and UNIX distribution, they will typically take some version of Thunderbird and make it available using their build and packaging systems, possibly using some stuff related to their localizations, and then support that for their users.

Their users are encouraged to file bugs, talk to them about it, and then they start filtering that stuff back upstream. They typically maintain their own sets of patches, which we very much encourage them to try and push upstream where appropriate.

Scott: Do you find that a lot of patches do come in from the downstream distros? It seems like some distros would prefer not to carry their own patches and that they would really want to try to push stuff upstream. How would a distro see it as advantageous not to push patches upstream?

Dan: I think it tends to be a mix of short term and long term economics. For the most part, people recognize that pushing it upstream is a way to get your patch maintained over the long haul and you don’t have to worry about it anymore.

At the same time, whenever you push a patch upstream, that’s more work than just writing the code. You have to actually get it reviewed, and you have to address any concerns that the reviewers have, and those are sometimes fairly substantial.

There are also sometimes just breakdowns in communication, although we are trying to put some systems in place so that that happens less and less as time goes on. We want to have a process to know when our distributors are having any issues and when reviews are not coming promptly and that sort of thing.

Sometimes it boils down just to knowing who to talk to, on both the Mozilla side and as on the downstream side. Many of our distributors are upstream for somebody else, so downstream distributors from Thunderbird sometimes end up having intelligence about shifts in the market or possible new markets for us.

Knowing who those people are and making sure that they know who we are is a good way for us to know, for instance, where a different country is heading or where some devices are evolving. There are a lot of people who are currently interested in Thunderbird on small screen devices–nettops and things like that.

This is an example where there are steady customizations that make sense to bring upstream in a particular form. It’s important to make sure that we have those relationships with the people coordinating those patches in the various distros. The answer usually ends up being a lot of email and a lot of Bugzilla.

I think another interesting piece that distros provide is that some subset of distros, in particular those that are extremely enterprise focused, need to be able to support pieces of software for a much longer time.

So on the Firefox side of the house, and I think to some degree on the Thunderbird side of the house as well, a certain number of distros actually end up maintaining code on older branches longer than Mozilla itself maintains it.

Scott: Sure–they have to back port the fixes because they have got an enterprise agreement in place. So, they are obligated to their customers to back port fixes across hundreds of projects, where each individual project is going to have its own individual policy for what is being back ported versus what is just going into the tree for the next version.

In some ways, the distros are on the front line in terms of a region or a certain customer segment. I would imagine that they can provide a lot of intelligence to you guys about what kinds of demand they are getting from customers and what is resonating with them.

They can package up projects like yours and make them available through their package system, but at the same time, people can download the source directly and compile it. Some people can short circuit the relationship, since there is not a strict hierarchy; they don’t have to go through the distro, although they could work with you guys directly if they want to.

Dan: I agree, and I think it works particularly well in the Linux community, because there is a very wide range of Linux users where some want to be and deeply, technically involved on the front lines, yet many don’t.

Scott: As Linux is starting to show up on consumer devices being sold preinstalled on some machines and things like that, do you see any of that come through in the kinds of questions that are submitted to you?

Are you seeing any kind of shift in the user base, and does that affect at all the way that you think about things?

David: One place where that kind of shift is happening is in the user interface. We are pushing a bunch of interface changes to make Thunderbird accessible to as broad a market as possible, so that you need to know less technical jargon and details of the capabilities of the server, for example, to get it set up.

A lot of our user interface was designed a long time ago, starting with the Netscape mail client, which aimed at a fairly technical user. As we move toward the consumer market, we are moving toward a more usable interface.

We don’t particularly see it as being associated with the Linux shift, so much as we see it as being associated with where we want to see Thunderbird go.

At the same time, though, as Linux is getting into a bunch of interesting markets, we’re benefiting, because we’re one of only a few usable mail clients on Linux.

Scott: With Firefox, Mozilla has been driving the use of a lot of automated tools, like the JavaScript fuzz testing tool they’ve built and things like that. What kinds of automated tools, threat modeling, and development processes do you use for Thunderbird?

Dan: We’re definitely interested in automation at all levels, because it delivers great bang for our buck. The biggest piece that we’re putting into place now in the Thunderbird world is automated testing and test frameworks.

We’re moving toward a world where more and more developers and other community members write tests for specific things in Thunderbird, so that it becomes harder and harder for things to go squirrelly when we make big changes.

Scott: That seems to be a pervasive but relatively new trend in open source projects. I think Apache has a test framework and some of that kind of stuff, but I haven’t really seen any projects doing what might be considered test driven development.

Dan: It’s varied quite a bit over the years. I know GDB has had it for quite a long time, but there are also many, many projects that haven’t had any at all.

It’s certainly relatively new for Mozilla. I’ve been a user of Firefox nightly builds for many, many years, and during the run up to the Firefox 2 release, the nightly builds were very often unusable, so it was always sort of a crap shoot. There were many days where I’d have to spend an hour messing around and uninstalling or undoing some sort of portage.

In the run up to Firefox 2, people started being encouraged to do more and more testing, and by the time we started the run up to Firefox 3, there were quite a few tests in the tree.

It was fairly clear that as a result of this change, the nightlines were pretty much usable the whole time. It was a massive improvement over the previous time around, which it has a nice effect on the feel of people doing software development. It’s much less frustrating, because you don’t feel like you’re taking one step forward, two steps back.

Even though it slows you down in a certain way, it is noticeably more rewarding in another.

Scott: There’s also comfort in just knowing that you can run the test suite. If you’ve done a particularly large refactoring or something like that, you can run the test suite, and if the tests pass, you know that you haven’t broken anything.

David: Test coverage and automation has actually been around in open source for quite a long time, particularly in non GUI software projects. For example, all the dynamic languages–Perl, Tcl, Python, JavaScript–have had extensive test suites for years now.

The new thing that Mozilla is doing is take serious, authoritative steps like closing the tree when there’s a performance regression on a user interface test, for example. Few projects can afford the infrastructure that you need to do that kind of thing happen really well, and that’s the kind of stuff that we’re pushing on Thunderbird as well.

Dan alluded to the nightly builds. We have thousands of people using the trunk of Thunderbird on their email every day, with updates coming out every day. That is a very tight loop of people who are very interested in the project, willing to trust us not to mess with their email, which is quite a lot of trust.

And we get feedback on features immediately, so if we design a new feature and it’s got a little bug, the chances of that bug being exhibited in the next couple of days are pretty high. It’s a relatively unusual model, I think, when you have that many users using a desktop software that’s updated daily.

Sean: Have you noticed any differences in the tenor or content of feedback from Windows users as opposed to Linux users?

David: Both groups of users get sensitive if things don’t work the way they expect them to, although the expectations themselves vary tremendously from individual to individual.

Certain individuals always identify specific features as critical, and they feel that everything should be dropped until that feature gets done. The fact is that there are hundreds of those features, and most of those individuals don’t care about all the other features in that category.

So for some people, integration with Exchange is a huge, huge issue. For others, it’s integration with the Linux search engine or Instant Messenger. There are all kinds of different perspectives.

I have not noticed a Windows versus Linux dichotomy, although there does seem to be a separation between people who are enterprise focused and people who see their email client as a personal productivity tool.

One group values service like integration, whereas the other may want power user features for triaging, tagging, and things like that. There are lots of different perspectives, but I have not seen a Windows/Linux split really.

Sean: The open source world has got an application for everything: GIMP versus Photoshop, Firefox versus IE, OpenOffice versus Office. But there has always been an interesting hole around Exchange and having a really solid solution for back-end group calendaring.

I’m not saying they don’t exist, but nothing has coalesced around something on the server side. Is it just that the fire hasn’t been lit, or do you feel like there are enough starts but it just hasn’t reached critical mass?

Dan: There are probably a whole bunch of factors at play, but one particularly significant issue is that, until relatively recently, there have not been calendar standards that are working all that well in the community.

Sean: I guess what you are getting at is that first the landscape was redefined as calendaring plus email as the de facto standard for ‘messaging,’ which led to the need for that standard way of integrating calendaring, and until that process is complete, it can’t really happen.

Dan: I think that this process is very much in progress right now. iCal has been maturing for a long time, but the big thing that has happened over the last several years is CalDAV.

Sean: At some point, a standard like that is pushed by the fact that people want it so they can build something to standardize. At the same time, sometimes you wait for the standard to be there to build something that will be functional.

Dan: There was a previous attempt at a calendar server protocol called CAP. It unfortunately did not succeed, for a number of reasons.

But one of those in particular was that there was a lot of protocol design, and everyone waited until the end. Then it turned out that a lot of it was not implementable in ways that were useful to the people that wanted to implement it.

One of the things that is very different about the CalDAV process is that it was carefully designed to be usable on servers that were not necessarily native CalDAV servers. They have also made sure that people implement it every step of the way, which helps ensure the standard is actually workable.

David: I think the most important factor in which open source projects thrive and which ones fail to thrive is whether they can attract a community of people who have a shared goal and a shared vision.

One of the challenges is that client side desktop software attracts a different set of people than server side software. In particular, all the ones you talked about–GIMP, Open Office, Firefox, Thunderbird–are client side software, where there is a broad set of people who can visualize their peers or their relatives or their friends using the software.

Writing an Exchange competitor is a very different visualization exercise for the developer, although there are definitely opportunities to build an open source community around that. Maybe some of the existing open source projects in that space will grow to create a dominant Exchange replacement–we will have to wait and see.

Maybe the emergence of working, standardized calendaring models will enable that to happen, but I think also it’s hard to build a community of volunteers around enterprise software.

Most of the big success stories on the server side have been coming out of companies with communities around them. I think a real grassroots open source project on the enterprise side is trickier organization to build.

Sean: Do you feel that maybe it needs the Mark Shuttleworth of the open source Exchange Server? Somebody who’s willing to kind of just pour money into something, even if it’s not making money?

Do you think it would need that sort of corporate backed open source, as opposed to just straight up community based development?

Dan: I would hate to say it needs that, but certainly a lot of successful open source projects benefit from having a corporate backer, in particular to coordinate the interface between the customers and the developers.

Maybe there’s sort of a brokerage function that’s useful there. But then again, it’s also possible, with the widening understanding of open source, that enough people who are frustrated with Exchange want to sort of get together and make an open source replacement. Maybe the ecosystem has changed enough that that is becoming possible where it hasn’t happened before.

Scott: As you look into your crystal ball around messaging, what do you see as the trends that will define the next five years or so?

Dan: Instant messaging is definitely one of the most important pieces, and there are a couple of different parts to that. The first factor is that the younger generation seems to use open instant messaging quite a bit, and often to the exclusion of email.

I also saw an interesting quote from a recent study that suggests using instant messaging actually drops the number of interruptions that people have at work. They tend to just have instant messages around very specific things, as opposed to walking over to someone’s office and breaking into what they’re doing.

Scott: I think with instant messaging, there is the ability to respond without mentally unplugging from what you’re doing. On the other hand, if you pick up the phone and call somebody, or even go into email, you’re kind of sort of shutting down whatever process you’re working on, after which you’ve got to spin back up.

An IM can be short enough that it doesn’t necessarily break the flow of what you’re doing in the same way.

David: My own crystal ball has a couple of things in it, one of which has to do with matters of scale. Over the last five or ten years, the volume of communications people are getting is always increasing, and there’s no reason not think that won’t continue. Any incremental, linear shift just delays the problem of overload, meaning that we need to come up with a relatively radical way to looking at messaging.

The other factor is that the emergence of instant messaging, social networking, Twitter, and all of those new communication models is helping people think differently about communication overall. These models are completely different from email messages which go from individuals to individuals with almost guaranteed delivery and an expected response.

With Twitter, for example, communications are sent to some random set of people who may see the Twitter updates or may not. There’s a very different feel to those kinds of communications.

There’s a chaos around messaging communication that’s most visible in the hyper-connected Silicon Valley and the geek crowd, but I think it’s going to spread out and dramatically affect how people look at email globally.

The open source world is by nature a global distributed sort of loosely connected people, and we end up encountering some of these problems before a lot of the mainstream. I think that gives us a motivation to solve the problem soon, and it also gives us a lead on people who work in highly hierarchal organizations, because the world doesn’t actually behave that way.

The short version of that is that we’ll see more messages of different kinds, all the time. We need to rethink how we look at communication in general, and Thunderbird is an interesting platform to experiment on, hoping that it will help us come up with new approaches to help us manage, that sort of chaos and cacophony of information.

Scott: Well, that’s a fantastic way to close. Thanks for talking with us.

Dan: You bet.

David: Thank you.

Bookmark this: