Interviewee: David Ascher, and Dan Mosedale

In this interview we talk with David and Dan. In specific, we talk about:

• How Thunderbird fits into the larger software ecosystem
• Internal handling of security, feature inclusion, usability, and QA
• The relationship between Thunderbird and Linux distros
• Automated tools and processes within Thunderbird development
• The relationship between open source and proprietary messaging software
• Future directions for messaging

Sean Campbell: Tell us a little bit about yourselves and give us a little background on Thunderbird.

David: I was an academic until I decided to get involved in software. Although I got started in software as a kid, it was about 10 years ago that I decided to do it professionally.

One of the first things I did was to start working for a company called ActiveState, which built commercial products and software for the open source stacks, including the Mozilla code base. That is when I got involved with Mozilla as an outside contributor. I was a community member, but working in a closed source shop building products for developers.

I got involved in leading the Thunderbird project about a year ago, at a stage when Mozilla was very interested in generating enough energy and enthusiasm around Thunderbird to build something exciting and competitive and forward-looking.

I jumped at the opportunity, and I have been swimming in the deep end of the pool ever since. I should also mention that even though I was in a closed source shop before, I have been involved in open source for more than a dozen years, primarily in the Python world. I also do a lot of talking with other people in open source foundations and open source projects, so I have a hybrid perspective.

I had actually worked at Netscape for quite a while, initially in IT and then later, I helped get Mozilla launched. I have been involved in it in one way or another ever since.

Netscape was definitely a proprietary shop, but it had always been very interested in open standards. With Mozilla, we felt like there were a bunch of us who had a little bit of experience in open source in various ways.

We wanted to see if we could get the world at large involved in browser development. I was always especially interested in the mail client.

Over the years, I have been funded by various organizations to work on Mozilla. For while I was leading the Calendar project, and more recently I worked on Firefox. When David signed up to do the Mozilla Messaging project, I offered to help. I am very excited about this space as a whole.

I think there are a lot of ways in which the whole market is underserved, and we would like to address that.

Sean: The latest numbers I have seen place Firefox as having about 20% of the market share, although I’m sure you have more specifics.

What kinds of things do you look back and think you want to do it the same way as the Firefox effort, and where do you see opportunities for improvement?

David: One issue that comes to mind is the realization that although it is an open source effort where we can distribute authority and responsibility, that doesn’t mean that everybody is equally competent in all areas.

One of the things that Firefox has implemented is the notion that, just as there are review processes for code changes, there should also be review processes for other kinds of changes.

For example, security bugs get reviewed by security experts. Code changes that have an impact on how easily the project can be internationalized or localized get reviewed by experts in those areas. User interface changes get reviewed by user experience specialists.

I think that is one of the ways in which Firefox has been able to really identify ways to increase the professionalism of the organization as a whole, as well as delivering ideal, optimal user experiences while still letting people participate in a variety of ways.

Even if you are not a user experience expert, you can contribute ideas and get usability people to comment on them. In general, that combination of decentralization and specialization is one of those lessons that we are definitely applying to Thunderbird.

There are lots of ways that we are deliberately mimicking the Firefox model, such as the way we do release driving, how we coordinate with localizers, and a large part of our toolset. We leverage a lot of the infrastructure they have put in place.

Dan: Another aspect of that experience that has been very helpful as we experiment going forward is the project management stuff. Over the years, the Mozilla Project has tried a bunch of different methods to make things happen.

Interestingly enough, it always seems to come back to our bug tracking system. I often observe that that’s a pretty painful way to carry out project management, but a lot of the other ways seem to be worse.

With the upcoming release of Thunderbird, we are going to make the process very, very bug driven and not driven from the top down as much as from the people who are working on the various pieces.

Sean: Let me ask you about the obvious elephant in the room–the closed-source Outlook client. What lessons do you draw from watching that development process or just even the feature set of Outlook as you guys look at building Thunderbird? Are there features in Outlook that you like?

There are obvious differences in the closed versus open development model, but what about after you peel that back one more layer? I have to imagine you guys occasionally look at Outlook and go, “Gosh, I really like that.” Or, “No, I really don’t like that and thank God we can make a change here.”

David: I think that is true for Outlook. But it’s also true for Mail.app. It’s also true for Gmail.

Sean: Sure; the only reason I mentioned Outlook directly is because it has a fairly dominant market share.

Dan: One of the biggest things that Outlook has done is to help frame the messaging market in terms of having calendaring included. That’s important, because it really is just another way that people communicate.

Sean: That’s true–calendaring has not always inherently been part of your mail client. Where do you see that going in the future?

David: Calendaring really fits into our general approach to how we think about features. There is always tension between which capabilities should be part of the core product and which ones should be part of extensions, for example, or add-ons.

The Mozilla core has a very nice add-on capability, supported by an add-on ecosystem and an add-on community that can help fill in parts of specific use cases that we don’t handle and that sort of thing. In general, though, if something is going to be used by a very large percentage of the user base, then it makes sense to have it as part of the core product.

Many people see the integration of calendaring as a core capability, and without it, you lose a lot of the utility that people now expect from messaging clients. There are other bits that may be more optional, such as integration of instant messaging.

That could change as the market and user expectations change, of course. What we try to do is to make a system where we can evolve a core feature set by bringing in features or experiences as add-ons so that there can be lots of iteration and lots of innovation without tying it too closely to the core product. That still gives the option at some point to bring those features and capabilities in.

The Lightning extension is a good example there. Figuring out how to do calendaring well is a huge problem set. The interoperability problems are huge. The calendar team has done a great job of building this extension on top of Thunderbird, and I think it has reached the point of utility, where it makes sense to bring into the core product.

We want to do that well, so that for people who don’t care about calendaring, it won’t get in the way. I think we can provide some leadership here–not necessarily the same way Outlook does it, but maybe better, maybe different.

We are currently figuring out how that integration will take place, but we are pretty sure it’s going to happen pretty soon.

Sean: How do you guys handle the process of security reviews, in terms of threat modeling or the different elements of your security process?

Users are pretty willing to shove anything in each other’s inbox, including proprietary documents and other things that they would not want to have leaked from the company.

Dan: Part of the answer is that we are still sorting that out. We are still very much getting spun up and growing as a project, so that is actually one of the things that is on my ‘to do’ list for the upcoming weeks, is to nail some of that down for Thunderbird 3.

Thunderbird makes some fundamentally different assumptions from Firefox. For example, by default it has JavaScript off in any HTML messages that you get. At the same time, though, there is going to be a similar process to what Firefox does, which essentially has some threat modeling and as well as code and design review.

There are also slightly different ways that users interact with a mail client, as opposed to a browser, so we will tend to observe things from that angle a little more and probably make some different choices as a result.

Scott Swigart: One of the things you talked about in your session at OSCON is that compared to a lot of open source projects, you really have to care about usability and people who can’t submit a patch to get a feature in which they are interested.

You mentioned before that you are really working on having bug-driven project management. How does that work from the perspective of an end user who is interested in getting something added but who doesn’t have the ability to code it up? What are the ways that they can interact with the project?

Dan: There are a bunch of different possibilities there. Certainly, a lot of them have to do with how much time they have and their level of technical sophistication. Working with us in Bugzilla, simply filing bugs that they find, is tremendously useful.

At the same time, there is a much larger class of users for whom the bug process isn’t all that well suited. When I said that I think having the process Bugzilla driven is a good goal, that refers mostly to the engineering point of view.

I think in some ways we need to explore other avenues. I know David has given a lot of thought to this idea of interacting with the less technical community.

David: One of the challenges we have is that by our competitors’ standards, we have a tiny, tiny staff–fewer than 10 employees covering the entire development and marketing process.

That means we have to rely on volunteers and partners to provide a lot of the work for a bunch of other things. One of the interesting areas that we are trying to explore right now is how to improve the QA process, because we’re not going to have 30 or 40 people who are trained QA engineers go through and make sure that Thunderbird works the way it is supposed to.

We are exploring ways of getting some of our early adopters and enthusiastic users to help test the software. The world of web apps has show that you can engage very large numbers of users in distributive processes of collaboration and quality assessment, and we are trying to make that work for Thunderbird users.

The answer to the broad question as to how non technical users get involved is hard to answer generally, because there are lots of different ways, and some of them will make more sense for specific users than others.

So for example, we have lots of localization teams, people in foreign countries who want to help make sure that Thunderbird is available from day one in their language and adapted to their locale. There is a process for that.

If there are people who have graphic design skills or marketing skills or other skills, there are a whole bunch of different avenues, depending on people’s backgrounds.

Scott: What about the relationship between an open source project like Thunderbird and downstream distros?

In other words, what do the downstream distros do with the project? If people in the Thunderbird community file bugs against the distro, do those get pushed up and the distros know when you guys have made changes?

Talk a little bit in general about the interaction between your projects and various distros.

Dan: I think you have actually summarized it reasonably well. In terms of the particulars of Linux distribution and UNIX distribution, they will typically take some version of Thunderbird and make it available using their build and packaging systems, possibly using some stuff related to their localizations, and then support that for their users.

Their users are encouraged to file bugs, talk to them about it, and then they start filtering that stuff back upstream. They typically maintain their own sets of patches, which we very much encourage them to try and push upstream where appropriate.

Scott: Do you find that a lot of patches do come in from the downstream distros? It seems like some distros would prefer not to carry their own patches and that they would really want to try to push stuff upstream. How would a distro see it as advantageous not to push patches upstream?

Dan: I think it tends to be a mix of short term and long term economics. For the most part, people recognize that pushing it upstream is a way to get your patch maintained over the long haul and you don’t have to worry about it anymore.

At the same time, whenever you push a patch upstream, that’s more work than just writing the code. You have to actually get it reviewed, and you have to address any concerns that the reviewers have, and those are sometimes fairly substantial.

There are also sometimes just breakdowns in communication, although we are trying to put some systems in place so that that happens less and less as time goes on. We want to have a process to know when our distributors are having any issues and when reviews are not coming promptly and that sort of thing.

Sometimes it boils down just to knowing who to talk to, on both the Mozilla side and as on the downstream side. Many of our distributors are upstream for somebody else, so downstream distributors from Thunderbird sometimes end up having intelligence about shifts in the market or possible new markets for us.

Knowing who those people are and making sure that they know who we are is a good way for us to know, for instance, where a different country is heading or where some devices are evolving. There are a lot of people who are currently interested in Thunderbird on small screen devices–nettops and things like that.

This is an example where there are steady customizations that make sense to bring upstream in a particular form. It’s important to make sure that we have those relationships with the people coordinating those patches in the various distros. The answer usually ends up being a lot of email and a lot of Bugzilla.

I think another interesting piece that distros provide is that some subset of distros, in particular those that are extremely enterprise focused, need to be able to support pieces of software for a much longer time.

So on the Firefox side of the house, and I think to some degree on the Thunderbird side of the house as well, a certain number of distros actually end up maintaining code on older branches longer than Mozilla itself maintains it.

Scott: Sure–they have to back port the fixes because they have got an enterprise agreement in place. So, they are obligated to their customers to back port fixes across hundreds of projects, where each individual project is going to have its own individual policy for what is being back ported versus what is just going into the tree for the next version.

In some ways, the distros are on the front line in terms of a region or a certain customer segment. I would imagine that they can provide a lot of intelligence to you guys about what kinds of demand they are getting from customers and what is resonating with them.

They can package up projects like yours and make them available through their package system, but at the same time, people can download the source directly and compile it. Some people can short circuit the relationship, since there is not a strict hierarchy; they don’t have to go through the distro, although they could work with you guys directly if they want to.

Dan: I agree, and I think it works particularly well in the Linux community, because there is a very wide range of Linux users where some want to be and deeply, technically involved on the front lines, yet many don’t.

Scott: As Linux is starting to show up on consumer devices being sold preinstalled on some machines and things like that, do you see any of that come through in the kinds of questions that are submitted to you?

Are you seeing any kind of shift in the user base, and does that affect at all the way that you think about things?

David: One place where that kind of shift is happening is in the user interface. We are pushing a bunch of interface changes to make Thunderbird accessible to as broad a market as possible, so that you need to know less technical jargon and details of the capabilities of the server, for example, to get it set up.

A lot of our user interface was designed a long time ago, starting with the Netscape mail client, which aimed at a fairly technical user. As we move toward the consumer market, we are moving toward a more usable interface.

We don’t particularly see it as being associated with the Linux shift, so much as we see it as being associated with where we want to see Thunderbird go.

At the same time, though, as Linux is getting into a bunch of interesting markets, we’re benefiting, because we’re one of only a few usable mail clients on Linux.

Scott: With Firefox, Mozilla has been driving the use of a lot of automated tools, like the JavaScript fuzz testing tool they’ve built and things like that. What kinds of automated tools, threat modeling, and development processes do you use for Thunderbird?

Dan: We’re definitely interested in automation at all levels, because it delivers great bang for our buck. The biggest piece that we’re putting into place now in the Thunderbird world is automated testing and test frameworks.

We’re moving toward a world where more and more developers and other community members write tests for specific things in Thunderbird, so that it becomes harder and harder for things to go squirrelly when we make big changes.

Scott: That seems to be a pervasive but relatively new trend in open source projects. I think Apache has a test framework and some of that kind of stuff, but I haven’t really seen any projects doing what might be considered test driven development.

Dan: It’s varied quite a bit over the years. I know GDB has had it for quite a long time, but there are also many, many projects that haven’t had any at all.

It’s certainly relatively new for Mozilla. I’ve been a user of Firefox nightly builds for many, many years, and during the run up to the Firefox 2 release, the nightly builds were very often unusable, so it was always sort of a crap shoot. There were many days where I’d have to spend an hour messing around and uninstalling or undoing some sort of portage.

In the run up to Firefox 2, people started being encouraged to do more and more testing, and by the time we started the run up to Firefox 3, there were quite a few tests in the tree.

It was fairly clear that as a result of this change, the nightlines were pretty much usable the whole time. It was a massive improvement over the previous time around, which it has a nice effect on the feel of people doing software development. It’s much less frustrating, because you don’t feel like you’re taking one step forward, two steps back.

Even though it slows you down in a certain way, it is noticeably more rewarding in another.

Scott: There’s also comfort in just knowing that you can run the test suite. If you’ve done a particularly large refactoring or something like that, you can run the test suite, and if the tests pass, you know that you haven’t broken anything.

David: Test coverage and automation has actually been around in open source for quite a long time, particularly in non GUI software projects. For example, all the dynamic languages–Perl, Tcl, Python, JavaScript–have had extensive test suites for years now.

The new thing that Mozilla is doing is take serious, authoritative steps like closing the tree when there’s a performance regression on a user interface test, for example. Few projects can afford the infrastructure that you need to do that kind of thing happen really well, and that’s the kind of stuff that we’re pushing on Thunderbird as well.

Dan alluded to the nightly builds. We have thousands of people using the trunk of Thunderbird on their email every day, with updates coming out every day. That is a very tight loop of people who are very interested in the project, willing to trust us not to mess with their email, which is quite a lot of trust.

And we get feedback on features immediately, so if we design a new feature and it’s got a little bug, the chances of that bug being exhibited in the next couple of days are pretty high. It’s a relatively unusual model, I think, when you have that many users using a desktop software that’s updated daily.

Sean: Have you noticed any differences in the tenor or content of feedback from Windows users as opposed to Linux users?

David: Both groups of users get sensitive if things don’t work the way they expect them to, although the expectations themselves vary tremendously from individual to individual.

Certain individuals always identify specific features as critical, and they feel that everything should be dropped until that feature gets done. The fact is that there are hundreds of those features, and most of those individuals don’t care about all the other features in that category.

So for some people, integration with Exchange is a huge, huge issue. For others, it’s integration with the Linux search engine or Instant Messenger. There are all kinds of different perspectives.

I have not noticed a Windows versus Linux dichotomy, although there does seem to be a separation between people who are enterprise focused and people who see their email client as a personal productivity tool.

One group values service like integration, whereas the other may want power user features for triaging, tagging, and things like that. There are lots of different perspectives, but I have not seen a Windows/Linux split really.

Sean: The open source world has got an application for everything: GIMP versus Photoshop, Firefox versus IE, OpenOffice versus Office. But there has always been an interesting hole around Exchange and having a really solid solution for back-end group calendaring.

I’m not saying they don’t exist, but nothing has coalesced around something on the server side. Is it just that the fire hasn’t been lit, or do you feel like there are enough starts but it just hasn’t reached critical mass?

Dan: There are probably a whole bunch of factors at play, but one particularly significant issue is that, until relatively recently, there have not been calendar standards that are working all that well in the community.

Sean: I guess what you are getting at is that first the landscape was redefined as calendaring plus email as the de facto standard for ‘messaging,’ which led to the need for that standard way of integrating calendaring, and until that process is complete, it can’t really happen.

Dan: I think that this process is very much in progress right now. iCal has been maturing for a long time, but the big thing that has happened over the last several years is CalDAV.

Sean: At some point, a standard like that is pushed by the fact that people want it so they can build something to standardize. At the same time, sometimes you wait for the standard to be there to build something that will be functional.

Dan: There was a previous attempt at a calendar server protocol called CAP. It unfortunately did not succeed, for a number of reasons.

But one of those in particular was that there was a lot of protocol design, and everyone waited until the end. Then it turned out that a lot of it was not implementable in ways that were useful to the people that wanted to implement it.

One of the things that is very different about the CalDAV process is that it was carefully designed to be usable on servers that were not necessarily native CalDAV servers. They have also made sure that people implement it every step of the way, which helps ensure the standard is actually workable.

David: I think the most important factor in which open source projects thrive and which ones fail to thrive is whether they can attract a community of people who have a shared goal and a shared vision.

One of the challenges is that client side desktop software attracts a different set of people than server side software. In particular, all the ones you talked about–GIMP, Open Office, Firefox, Thunderbird–are client side software, where there is a broad set of people who can visualize their peers or their relatives or their friends using the software.

Writing an Exchange competitor is a very different visualization exercise for the developer, although there are definitely opportunities to build an open source community around that. Maybe some of the existing open source projects in that space will grow to create a dominant Exchange replacement–we will have to wait and see.

Maybe the emergence of working, standardized calendaring models will enable that to happen, but I think also it’s hard to build a community of volunteers around enterprise software.

Most of the big success stories on the server side have been coming out of companies with communities around them. I think a real grassroots open source project on the enterprise side is trickier organization to build.

Sean: Do you feel that maybe it needs the Mark Shuttleworth of the open source Exchange Server? Somebody who’s willing to kind of just pour money into something, even if it’s not making money?

Do you think it would need that sort of corporate backed open source, as opposed to just straight up community based development?

Dan: I would hate to say it needs that, but certainly a lot of successful open source projects benefit from having a corporate backer, in particular to coordinate the interface between the customers and the developers.

Maybe there’s sort of a brokerage function that’s useful there. But then again, it’s also possible, with the widening understanding of open source, that enough people who are frustrated with Exchange want to sort of get together and make an open source replacement. Maybe the ecosystem has changed enough that that is becoming possible where it hasn’t happened before.

Scott: As you look into your crystal ball around messaging, what do you see as the trends that will define the next five years or so?

Dan: Instant messaging is definitely one of the most important pieces, and there are a couple of different parts to that. The first factor is that the younger generation seems to use open instant messaging quite a bit, and often to the exclusion of email.

I also saw an interesting quote from a recent study that suggests using instant messaging actually drops the number of interruptions that people have at work. They tend to just have instant messages around very specific things, as opposed to walking over to someone’s office and breaking into what they’re doing.

Scott: I think with instant messaging, there is the ability to respond without mentally unplugging from what you’re doing. On the other hand, if you pick up the phone and call somebody, or even go into email, you’re kind of sort of shutting down whatever process you’re working on, after which you’ve got to spin back up.

An IM can be short enough that it doesn’t necessarily break the flow of what you’re doing in the same way.

David: My own crystal ball has a couple of things in it, one of which has to do with matters of scale. Over the last five or ten years, the volume of communications people are getting is always increasing, and there’s no reason not think that won’t continue. Any incremental, linear shift just delays the problem of overload, meaning that we need to come up with a relatively radical way to looking at messaging.

The other factor is that the emergence of instant messaging, social networking, Twitter, and all of those new communication models is helping people think differently about communication overall. These models are completely different from email messages which go from individuals to individuals with almost guaranteed delivery and an expected response.

With Twitter, for example, communications are sent to some random set of people who may see the Twitter updates or may not. There’s a very different feel to those kinds of communications.

There’s a chaos around messaging communication that’s most visible in the hyper-connected Silicon Valley and the geek crowd, but I think it’s going to spread out and dramatically affect how people look at email globally.

The open source world is by nature a global distributed sort of loosely connected people, and we end up encountering some of these problems before a lot of the mainstream. I think that gives us a motivation to solve the problem soon, and it also gives us a lead on people who work in highly hierarchal organizations, because the world doesn’t actually behave that way.

The short version of that is that we’ll see more messages of different kinds, all the time. We need to rethink how we look at communication in general, and Thunderbird is an interesting platform to experiment on, hoping that it will help us come up with new approaches to help us manage, that sort of chaos and cacophony of information.

Scott: Well, that’s a fantastic way to close. Thanks for talking with us.

Dan: You bet.

David: Thank you.

Bookmark this:

Interviewee: Gabor Cselle

In this interview we talk with Gabor Cselle. In specific, we talk about:

• Deciding between open and closed development models
• Strategies around handling QA challenges effectively
• Maintaining high usability and user security
• Business models and corporate customers as opposed to individual users
• The relationship between open-source and commercial software
• The value of avoiding feature creep

Sean Campbell: To get us started, could you please give us a little bit of background on Xobni, your role with the company, and a little bit of your work history before that?

Gabor Cselle: I’m the VP of engineering here at Xobni. I run our engineering team, manage QA and support, and was the first employee. Previously, I worked at Google as a software engineer, both in Zurich, Switzerland where they have a big engineering office and in Mountain View.

Xobni was started as a Y Combinator company. Y Combinator is a seed funding group that tries to encourage young college graduates to start companies, instead of working for “the man.”

The company was started by a graduate of MIT and a Penn State graduate in 2006. For about the first year, they were working on what we call Xobni Analytics today, which is a package that analyzes your Outlook emails just like Google Analytics analyzes your web traffic. It would chart out how many emails you got this month, how many people you emailed with, and so on.

After a while, we realized that people wouldn’t use that more than once or twice, and then they would let it fall by the wayside. We really wanted to build something that people would use for hours every day, and so we got the idea for today’s product, which is a sidebar for Outlook.

We got funding from Khosla, so after I joined, we built the current product in about six months, and launched the private data in September of 2007. Since then, we’ve been adding features and broadening the project’s scope, and we have some new exciting projects as well.

Sean: Given the overall open-source momentum these days, what was your thinking on making the project closed source from the start?

Obviously, some of it was driven by the fact that you are going to plug into Outlook, but that didn’t have to be a complete limiter, in and off itself.

Gabor: I don’t think it was a conscious decision in the very beginning. It was more about the fact that eventually we wanted to license this to enterprises, because we were working on the Analytics product back then. And obviously, Outlook is closed source–there’s the thin layer around Outlook of APIs that Microsoft makes public, but everything else is closed source.

I think we are just kind of imitating that model, and eventually we may have versions with extra features as something that you can purchase. I think that’s why it’s still closed source, and we’re probably going to continue to follow this route.

Sean: I guess the natural follow up to that is to ask what kind of push you have gotten from people to extend what you are doing with Outlook and put it in Evolution or Thunderbird?

Gabor: Actually, the day after we released, we had about 400 emails in our support queue that all said, “I’m not an Outlook user. I’m a Thunderbird user,” or “I’m a Gmail user and I really like the functionality,” etcetera.

So this is definitely something that we’ve looked at very, very intensely. We decided not to pursue the path of integrating with any other desktop clients, at least for now. We would need to revise a lot of the interface, but the main reason is really that Outlook is the number one email application in the world. There are 400 million people using it, and a lot of these people are in pain.

Outlook doesn’t do what they want it to do. They can’t search for messages quickly, especially with the older versions. We solve that one pain point really well for Outlook. It’s a huge addressable market, so that’s why we are sticking to it. The future might hold a version for other clients as well, though.

Sean: One of the things that I find interesting is how you have handled the breadth of QA questions you must have come up against. I mean, everybody uses Outlook, from the most computer illiterate to the most sophisticated users, so what they’re doing with their machines is radically different from each other. Notepad may be the only thing that gets more universal access on a Windows box.

What was your process around QA from the beginning, and how has it evolved? I’m sure you have to build something that is absolutely rock solid, and yet who knows what people are running alongside it?

Gabor: My first reaction is to say that this issue has been a lot of really hard work. When we were originally building the product and the team was just four people, we outsourced QA to a manual testing company in California. We had a good experience with that, but the problem is that the number of possible configurations and the number of things that can go wrong is very high.

I think the best decision I’ve ever made here is to bring QA in-house and to hire really good people and just try to get these issues taken care of by adding more bandwidth to them.

We hired an excellent QA lead and testers, and they’re not just testing the product–they’re testing the different integration scenarios with Outlook. They’re testing scenarios like how it works with Exchange, IMAP, POP, and various combinations of plugins.

That’s a lot of work, and it’s not particularly sexy, because everyone wants to work on a great new feature, rather than what happens when a particular ActiveDirectory setting is at one instead of zero. I’m very happy to say that we have QA engineers who are very passionate about getting these things right.

Scott Swigart: In a lot of open source projects, code’s checked in and people download stuff before it’s officially called a release, and they work with it. The model is often just to have good bug tracking, but not a lot of overt structure beyond that.

What do you think about the notion of handling bug fixes that way? What do you think would be the net result of a model like that in your case?

Gabor: That’s a very good question. I don’t think it would necessarily work for us–mainly because people don’t want to be exposed to bugs in their Outlook, and they don’t want to be the guinea pigs, because Outlook is their data store. It’s where all their personal data is, and they’re necessarily very sensitive about protecting that.

I think that our case requires in-house testing. That said, however, we originally did a staged private beta release of our software, to a limited population who had signed up on our web site.

That gave us a really strong test audience of people with a bunch of different configurations, and only once we were pretty comfortable with that did we go to the next order of magnitude.

There’s an effect at work that we call the “10X effect.” Every time you increase the number of users by a magnitude of 10X, new stuff starts popping up. For every 10X increase in user base, we get all sorts of weird scenarios that we haven’t seen before, and we go through these cases very methodically and just kind of adjust them one by one.

Scott: Another thing that open source does a lot is to use very distributed teams. You can have people working on one project from all over the globe and lots of different companies.

What kinds of advantages does it give you because you can pull everybody together in a conference room and you’ve got people working in the same building, versus what may be some of the challenges of using a highly decentralized distributed model?

Gabor: Everyone is in the office during work hours–no one really works from home on our current team. We have meetings about support, about QA, about where the product is headed, and about the current status of each of the projects that we’re currently working on.

We find that face time is very important, because if you want to describe a problem, you can just go up to the whiteboard and sketch it out. You don’t need to start up some virtual conferencing solution and try to drive your discussion through a window on the screen.

Really good ideas pop out from hallway conversations; we actually just had a conversation about how sometimes the Outlook UI would freeze when we were syncing data in the background:

We were able to fix that issue, but if we had not had that face-to-face conversation, we wouldn’t have been able to get a solid theory about why that bug was happening.

Scott: Another thing that springs to mind is that, aside from the architecture and what is going on behind the scenes, do you feel like a closed-source, centralized model facilitates getting where you want to be with the user experience and those kinds of things?

Gabor: To answer that, I should first take you on an excursion through history. The interface of Xobni has been highly praised, and some people call it “Star Trek”-like, because it looks like the computers on “Star Trek, The Next Generation.” The way that interface came about was basically that, in the very early days of Xobni, I just tried out different things in Photoshop. I came up with three basic designs: one looked just like Outlook, one looked like Outlook with bigger icons, and one I made as a joke about how far out we could take it. That is what we use now. And that’s how the basic look and feel of the interface came about.

It was basically arrived at with experimentation and talking to the other team members at the time about what they felt about the different designs. Lately, we’ve moved into a slightly heavier development model. We have a user experience expert and weekly usability meetings, we view mock ups, and we do hallway usability testing.

We also go to coffee shops. There’s a Starbucks we frequent a block from the office. We just ask people, “Do you understand this mockup? Do you want to try it out on our laptop and tell us whether it works for you?” One of us actually stands in line and says, “We’re going to pay for your latte if you have five minutes to look at some screen shots.”

Sean: Wow. Ethnographic research with a latte–it’s the grass roots ethos, for sure. [laughs]

Gabor: We’re in the financial district of San Francisco, so it’s a really good spot, as these are our target users. They all use Outlook, because they work at Morgan Stanley or Wells Fargo. They’re exactly the kind of people we want to talk to, and they’re exactly the kind of people that need to understand what our product does.

And so, back to the open source discussion. I don’t think that in a distributed team, these kinds of impromptu usability sessions would really be able to take place.

Sean: There’s the issue of reliability, and nobody wants to see the “Outlook is checking your inbox” message for the 5,000th time, just because you happen to shut it down. But at the same time, there’s a security issue, especially as you get into those large companies in the financial district and so on.

So what is your process around security? In the open source model, it’s well-known, and it boils down to “many eyeballs creates a more secure code.” I don’t mean this at all in a pejorative way, but closed source companies, by nature, need to protect their IP, and so they’re a smaller team that keeps it in-house.

What has Xobni done around the security aspects of the development model? Are you following a particular framework, whether externally developed or internally?

Gabor: Our basic premise is that we don’t ship any personal data back to us, including any part of your email, any part of your calendar, or any part of the data on your desktop. That’s been the case since the very beginning of Xobni, and it’s absolutely vital, because people are very sensitive about this issue.

Beginning in the very early versions, we shipped with something called “The Customer Experience Program,” which was an opt-in program, and it still exists, if you go into the options dialog. If you enable that, we ship back an anonymized data set about your usage patterns. It basically tells us that a user identified only by a randomly assigned number uses feature X and Y, but not Z. So, that is the one time where you can opt in and volunteer to give us some anonymized user data.

Sean: That’s an interesting bridge to another question that I have had in mind–what types of issues arise when you support a corporate enterprise sale? I imagine that there might be some concern on the part of IT organizations about supporting a broadly distributed plug-in like this.

Gabor: Actually, the first-line answer is that we’re a consumer company, and we want to speak directly to users and get them to download our products in order to address pain points that they are experiencing. We want them to be happy with our products and to be able to find the stuff in their inbox and to deal with it efficiently.

That’s much, much easier than trying to convince an IT department to get Xobni for their entire corporation.

That being said, a lot of the more progressive IT people have contacted us proactively and said, “We’re a company with 200 or a company with 5,000 people. We want Xobni on every single one of our desktops.” Of course, we encourage that, and may work out licensing that meets their needs.

I think in the future, we might have a support model specifically around this type of deployment, but that really hasn’t been worked out yet, mostly because the company is quite young and also because we are very focused on making a brilliant product that consumers really enjoy.

Scott: In the open source world, there’s the notion that the source code is free. People can download it, and companies have to figure out a way to monetize around that. They might sell support or have a more enterprise version that they charge for, with features that aren’t in the free one.

As you look at the landscape, what’s your impression around business models? Do you have a sense that the traditional model of having people download an eval version and then pay for the full version is still viable?

We run into the impression on the part of some people that everything’s moving to an open model and that open source software is built on a superior development methodology. On the other hand, there are new and innovative companies like yours that didn’t choose that route.

Do you feel that the traditional business models are alive and well, or do you see things moving to that “give it away and figure out how to charge for something” approach?

Gabor: People ask us this question a lot, and it basically comes down to, “How are you going to make money? Can you really charge for an add-in?” Our answer is that right now, we’re not focused on that. Right now, we are very focused on building a great product. We believe that if you build it, they will come.

And if you have millions of users that enjoy your product every day, somehow you will always be able to make money. Like Google. Google didn’t get started by saying, “Oh, we have search. Let’s find out how to monetize it.” They were, in the first few years, just focused on getting search better every single day, and later they found ways to monetize it.

There’s a story that the initial Google pitch stack was 70 pages long and explained how they would make money by selling search technology to enterprises, which was completely wrong at the time, but they later found a different way to monetize it, which has worked pretty well for them.

Our mindset, right now, is the same. I think we’re just building a great product and we’re going to find out how to make money eventually. People are asking us to buy pro versions. People are asking us whether they can pay if they want to install it into an enterprise.

We’re definitely looking at these models, but it’s really too early to take those conversations on in earnest, though–that still needs to wait six months or a year.

Sean: That’s fair. Let me ask you a different kind of question. There are obviously many, many good open source applications. At the same time, though, the little holy duopoly of Exchange and Outlook never seems to really find a strong open source competitor. I don’t think that people love it, but it’s the best that the world has at the moment.

Every time you try some other combination, whether it’s Zimbra or Evolution or Thunderbird or Sunbird or you try some other backend, like IMAP, POP, gmail, or whatever, they never seem to have that fidelity.

Why do you think the Exchange and Outlook combination never really sees the same strength of potential competitors as RedHat versus Windows, or MySQL versus SQL Server, or any of these other parts of, let’s say, the LAMP stack? Why is it that in that one spot, there doesn’t seem to be a parallel, even though there is almost everywhere else?

Gabor: I think you’re totally right, and I don’t know why that is. There are definitely great applications out there, such as the experience that Gmail provides. Also, Zimbra is great, and there’s just a fantastic variety of new applications in this field.

I think that Microsoft has the strength they do in the enterprise world right now because they know exactly what features they want, and they know exactly what these customers need and how to get into these enterprises. And once you have Exchange Server installed, it’s pretty hard to switch, because Exchange and Outlook have tremendous lock-in. I think that’s the key to their success and to their strength.

Sean: As a follow up to that then, since you came from Google, it’s an easy one to throw out there. What do you think about someone like Google’s relationship to open source? Or to give another example, what about Apple? Apple is seen as this amazingly open company, but they’re certainly not very open when it comes to their roadmap, or talking about other stuff like that.

With Google, you’ve got a company that builds a lot of stuff on essentially free open source, but yet they build a lot of commercialization on top of that, so there’s an interesting story there I think. Do you see any tension there?

Gabor: First, I have a really high opinion of Google, and I think that they’re excellent at software development. Also, I think their fundamental tenet of not being evil and doing good for the world is genuine, and they’re certainly sticking to it. They’ve open-sourced a lot of software already – such as Google Gears or protocol buffers, and I’m sure they want to be open and share the wealth with the world, but have to balance that with being a strong company that can generate profit.

Scott: During development, does open source fit in anything you do? I know as a developer that open source is really, really popular as developer tools for unit testing, automated builds, and for all kinds of different things. From the perspective of peeling back the covers of your organization, where do you find it useful or do you not really use it for much?

Gabor: We do use open source for build tools, release tools, unit tests, and so on. The infrastructure provided by the open source community is very useful to us, and I could imagine us contributing once we have a little higher level of resources, but so far our interaction has been very limited.

Scott: Would you mind giving a plug for some of the build tools and development tools that you really like?

Gabor: There’s always the wish list that you have. I think that NAnt, NUnit, or CruiseControl are really good, but I wish that they could each do this one more thing. There are other tools we use, all of which are an integral part of our development cycle.

Scott: One of the things that we have observed before is that open source really shines in projects that are ‘by developers, for developers,’ rather than software that targets a non-developer user base. In the case of developer tools, for example, the people building the project have an intimate understanding of what the product needs.

Open source projects tend to be very good at building a lot of infrastructure stuff–operating systems like Linux, web servers like Apache, and so forth. That seems to be where open source really innovates toward excellence. On the other hand, the really end-user-focused innovation like the iPhone, Google Maps, and the Xobni UI examples we have been talking about don’t tend to come from open-source projects.

I feel like open source is very concerned with architecture and performance and modularity, but when it comes to building a peak experience product, a lot of that’s still coming from the Apples and the Googles and the Microsofts and those kind of companies.

Do you feel like I’m on to something there? And if so, do you feel like there are fundamental reasons for that, based on how the different kinds of software are built? Or am I kind of just drawing a circle around a few things I have observed and calling it a trend?

Gabor: I have a very controversial opinion around this issue, which is that if you want to make a product with a peak experience like the iPhone for example, it’s not as much about what you build as what you take away, and about making those cuts at the right point.

For these types of decisions, you need very strong product people who understand exactly what the consumer wants and who understand exactly what the minimal feature set is that covers the maximum number of usage scenarios.

And this is something that I, as a developer, am not necessarily good at. There’s always the impulse to think of the next feature, and I always want to add, add, add, add. That’s why all of the open source tools have zillions of command-line flags and options and so on.

Whereas, if you want to make a consumer product like the iPhone, you have to make the cut somewhere, and make those things that are inside of the cut just outstandingly good instead of making a lot of things that are sort of a 75% solution.

Scott: That’s interesting from the standpoint of a lot of work I have done as a consultant for software companies. They say, “We’re going to add this feature to the product, because one important customer wants it.” And especially with smaller companies, a lot of times they can be pushed around by their bigger corporate customers to add things into the product.

Even something like MySQL does a certain amount of non-recurring engineering, and they add a certain amount of features, because a person wants it really bad for some specific scenario. I’ve never heard anyone explain it exactly the way you did, but that makes perfect sense. The iPhone’s good not because of what’s in it–it’s good because that’s all that’s in it. They really did just focus on making those things really good.

So how does a company like yours balance those two impulses? You must run into people who say, “We’d buy it if only you would add this feature.” How do you balance that with the philosophy of less is more?

Gabor: I think this question comes back to what I said earlier, which is that we’re really focused on consumers. We don’t, as a company, listen to the types of people who say, “If you just added this feature,” because right now we’re not even selling a product. So, the bottom line is that we can’t really consider a scenario where if we just added the foreign feature, we’d have $100, 000 more revenue.

And a related thought is that it’s been very painful for me personally, because I’m one of those “add, add, add” people, and other people on the engineering team are as well. Xobni used to have in the private data stage two tabs, which were “email” and “organize.” And one was about your email and the other one was about your calendars, your appointments, your outstanding tasks and so on.

We found through the customer experience improvement program that I talked about earlier, where we send back anonymized data about features if you opt in, that no one was using it.

We thought, “We built this great thing and no one is using it. Why? We should make it more prominent and maybe we should make the button blink or something.”

[laughter]

But then really, honestly, this is really and indication that people don’t want that functionality. And we decided to cut it out. It was probably one of the best decisions that we ever made, because we focused the product, and we made the things that people really do use even better. It’s worked great for us–it was just very painful at the time.

Scott: That reminds me of Firefox. We had Netscape, and then Internet Explorer became dominant, and it kind of became the browser with one more thing and one more button. And when Firefox first came out, it really made people sit back and ask what they really needed from a browser. They asked whether they really needed a lot of extraneous stuff, and a lot of them decided that Firefox really had everything they needed.

When it kind of caught fire, I think it was because it had that minimalist approach. I wonder if it depends on the size of your target, in the sense that if you target a niche, then you may need to be able to get pushed around by the niche a lot. And you may need to build a feature for a customer, because the niche is only so big.

On the other hand, if you are targeting millions of users like Xobni did, maybe you’re automatically freed from a lot of that. In essence, it seems like you asked where the pain point is for hundreds of millions of users, and that freed you from trying to be all things to all people. You just have to be the right thing for a bunch of people.

Gabor: Exactly. But finding that spot is really hard.

Scott: I can imagine.

Sean: Well, thanks for a great conversation. I’ve really enjoyed this.

Gabor: Absolutely. Thanks for setting it up.

Bookmark this:

Interviewee: Kevin Kluge

In this interview we talk with Kevin Kluge from Zimbra. In specific, we talk about:

• Zimbra’s history and finding a niche
• Profitability, ownership, and managing commercial products
• Interacting with the community
• Source code availability and its impact on the public
• Managing customer needs and usability

Sean Campbell: Kevin, would you tell us about your background and your role at Zimbra?

Kevin Kluge: At Zimbra, I’ve had a variety of engineering management positions since I started in September of 2004. The company was about nine months old then, and we hadn’t shipped any products at that time, and we hadn’t decided to go open source. I’ve managed various parts of development, QA, and support.

Prior to Zimbra, I worked in the messaging space at a variety of companies, including Onebox.com, Phone.com, and Openwave, as well as Corvigo, which was an anti-spam company, also doing mostly engineering management..

Sean: Tell me a little bit about the history of Zimbra. What’s the big picture history, going way back past the Yahoo acquisition?

Kevin: The original concept for Zimbra was for an email archiving appliance. This was prior to when I joined. When they guys started talking to potential customers, most of them were satisfied with their archiving solution, but they were dissatisfied with their mail system. So the guys had implemented some early Ajax technology around archiving, and they were able to repurpose it into a mail system.

And so the original concept morphed, and we were going after larger customers who needed a small set of functionality, but they weren’t really ISPs–it was more companies like airlines, banks, and so on, that had a large number of boundary workers that they wanted to put on the mail system.

We shipped the first product release in about April or so of 2005, and we started getting real customers in late 2005.

Sean: Was some of the drive to have Zimbra constructed the way it is an effort to make the back-end mail server integrate with a lot of front-end points, all the way to MAPI integration with Outlook, high-end version?

It’s a fairly full-featured suite, and I find that technology stacks generally have a little bit of a blind spot. I had a Mac for the last year, and there were certain things in that platform that I loved, like the add-in chat client was probably the best chat client I’ve ever seen on any platform. But at the same time, Exchange back-end integration if you really wanted a shared calendar was lacking.

The open source communities always had a little bit of a gap there, and I wondered if that was an intentional thing Zimbra was trying to shoot for–to reach out at a multi-platform level via a web interface with a really robust shared calendaring solution.

Kevin: There were two things where we thought there were real opportunities in the marketplace. Certainly calendaring was one. Even the commercial calendaring products weren’t that great, and the open source ones were even more lacking. That was certainly a large opportunity that we saw and, after doing the basic mail application and address book, that was the first thing we went after.

We’ve also done a couple of other strategic things, because we noticed that a lot of the open source groupware was very difficult to install. We decided from day one that we would have a strategy of using existing open source components where possible and we would bundle them into our download, assuming that the licensing allows us to do so.

We wrote an installation mechanism that wrapped all the different components and then we also wrote an admin interface and management tools. These allow administrators to change one thing and have it propagate to two, three, four different open source components that need to know about it. We think that this enabled people to download and use Zimbra who wouldn’t have been able to if they had to download it and compile it or download it and 10 other packages, each of which had to be a particular version.

You’re also right that we did go after a cross-platform environment. We wanted to be the best desktop neutral solution for collaboration. We didn’t want to be a Linux desktop solution or a Windows desktop solution or a Mac desktop solution or a mobile solution. We wanted to allow people to choose whatever client they wanted, from the beginning. It took us a while to implement some of the features, but that was definitely the strategy, and I think it has been very helpful in getting both community adoption and commercial customers.

Sean: It sounds a bit like what Ubuntu’s doing with desktop distributions–they keep trying to make the process more and more elegant for installs, even with the integration of Ruby and so on. They want to make it increasingly, progressively easier to get the distro down on metal.

It sounds like you want to give someone a collaboration messaging solution that’s fairly easy to just get down under the metal and have a decent set of features.

Kevin: Exactly. Our target market is the IT administrator more than the developer, which is a different skill set.

Sean: On a different topic, since you were acquired by a “for-profit” company, what’s it been like to live inside that bubble? Depending on who’s talking about the MySQL acquisition, for instance, there’s either a massive amount of consternation, fud, or benefit.

You guys have lived in that situation since September, so what’s that been like?

Kevin: Certainly for us, it has been positive. Yahoo is a great company with a lot of resources and a lot of technology in-house that we now have access to, as well as resources in things like the hosted offerings we’re now working on. We’re able to do it with a speed and a scale that we would not have been able to do if we were a stand-alone company.

In terms of the commercial side of it, that’s been helpful as well, because Yahoo has more mature sales operations facilities than what Zimbra had, so we’re working to take advantage of those. But in terms of the product strategy and how we do support for our customers and how we interact with the community, it really hasn’t brought much change.

Mostly, Yahoo’s instructions to us have been to keep doing what we’re doing. The one thing that we are doing aggressively right now that we weren’t doing before is to build out a hosted offering. You can imagine, given how Yahoo runs its data center, that would be a priority. We’re working to put up a hosted ZCS offering. And we’re going to go after a wide variety of target markets with hosted Zimbra.

Scott Swigart: Let me ask a question in a little bit of a different area, in terms of some of the prominent old-school open source projects; like Linux and Apache.

The Apache Foundation has a way of doing things, where the project has to have at least three different committers who are not part of the same company, and things like that. But a lot of the kind of vibrant open source that you hear about–SQL, Alfresco, Zimbra, Zend did their work around PHP, XenSource and their work around the Xen hypervisor. There seems to be an open source project that has a primarily–if not exclusively–for-profit wrapped around it.

How do you view the open source landscape, and what do you view as advantages and disadvantages of some of the possible models? I’m thinking of the range between the one that’s basically “community” or at least no central corporate sponsor, one where there’s a primary sponsor, and then all the way over to proprietary software where there’s a primary sponsor and the code isn’t even available.

Kevin: I can tell you a little bit about our experiences, and where we’ve arrived in trying to navigate those three different options. When we were initially discussing whether or not we wanted to go open source, there was quite a bit of concern that we’d give away all this value, and then how would we make money?

The answer that we kept coming back to that had been proven in the market is that you can charge for support and people will pay for it. That lets you give something back to the community that’s a great piece of software, but as long as you can keep your development costs under control, you can still have a very successful business model. That’s where I think a lot of the older, more established companies had ended up.

We recognized that there was an opportunity to create a great product that gives a lot of value to the community. At the same time, we saw that we could also create a little bit more profit for ourselves by adding a few additional features that we believe that enterprises or large ISPs would want, and that they would be willing to pay money for.

When we started talking about that, and looking at what we wanted to get built in the next few years, we started to identify features that we could leave out of the community edition and still have a spectacular community product, and then potentially get a little bit more profit for the company by putting it in the commercial version.

Those discussions were very lively, and very animated. We really did debate almost every single feature, and try to decide which bucket to put it in. We have been able to do that successfully, so we do have maybe five or 10 features that are available only in the commercial edition. So even though we don’t give away everything, we still do give away enough so that people can be very successful running a collaboration system.

The community is extremely helpful on two separate fronts–one is the product management front, and the other is the quality front. In terms of product management, we have a Bugzilla system that’s open to the world, and then we also maintain what we call the “PM Portal.” Basically, it’s a website where people can go and view what features we’ve had, and what releases, and what we’re thinking about for the next major release.

When we make product management decisions, we look quite a bit at the data in Bugzilla. We look at what people are voting on, and we look at the comments they’re making about the features, and the use cases that they have for those features. That plays very significantly in our decisions about what features we’re going to go do in what releases. On the quality front, we get feedback primarily from Bugzilla, and then also of course the forums. The community has really been invaluable in helping us get new releases up to a stable point quickly.

We’ve done quite a bit of learning there–initially we did beta releases, both of the Community version and the Network version, for the 3.0 and 4.0 releases. For version 4.5.0, we did not do a major beta release, and the quality of the 4.5.0 release was below where we wanted it to be. I think that was directly because we didn’t do a good beta cycle. Then for the 5.0 release, we did a beta with the community, but we only did the commercial beta toward the very end of the cycle.

We think this was the best compromise between the different needs. It allowed the early adopters in the community to get the software, and install it, and give us feedback as early as they wanted to. At the same time, it didn’t put software out there for our commercial customers to install and then potentially have trouble with it.

The community’s involvement both in helping to define the product, and then helping to refine it has been tremendous. Without the community, if you’re in a proprietary commercial situation, you have to run these large-scale beta programs. I don’t think you get feedback as quickly, and I think it’s more expensive for you as a company to get that feedback.

Scott: Does the community actually submit patches, or is their role just posting bugs that they find, or posting issues that they find, and then you task your engineers with making fixes?

Kevin: That’s an area where we really weren’t sure what we were going to get when we made the decision to go open source. In total, I think we’ve had far fewer community code contributions than we expected, and my suspicion is that’s because Zimbra is primarily a product for the IT administrator and not for the developer.

We do have a process set up to accept code contributions from the community. The few code contributions we’ve gotten have been great, but it really hasn’t had a major impact on the product, aside from the fact that we’re bundling all these other open source components. I’ve been a little bit surprised by that.

The community has contributed a number of localizations for us that have helped spread ZCS to users that probably wouldn’t have run it otherwise. That was really helpful, since we were a start-up with no internal localization team.

Scott: What feedback do you get from customers about the value of making the source code available? Could you talk a little bit about the value that you think customers get, and the value that Zimbra gets back?

Kevin: The number of people who attempt to make use of the source code is fairly small–certainly with our commercial customers, it’s a very small percentage.

But for those who do, and who have the skill set to do that, they can make custom changes that they can reapply on releases, or potentially some of them will actually go in and make bug fixes for their own personal versions, because they can’t or don’t want to wait for the next maintenance release where we would address the bug. So, it definitely creates some value there.

That’s also helped some customers to be less concerned about the Yahoo acquisition, because they know that no matter what happens with any acquisition of the Zimbra technology, the source code has been released under a licensing agreement that can’t be changed.

So it’s definitely been helpful to some extent in the sales cycle there, as well as giving customers a little bit more of a piece of mind. We wanted to see the browser continue to evolve as a platform for applications, and we felt that we were pushing the limits of what could be done with browser-based technologies in our Ajax application. By putting the source code out there, we hoped that other people would take a look at what we had done and learn from it, and go do things as good or better than we had done with other Ajax applications.

The last part that we have been working on creating is a community around Zimlets–our mechanism for extending the functionality of the web app. We do have to define Zimlet APIs, and people could write code to the Zimlet API without looking at the underlying source code. But having the source code available certainly is helpful, when you’re trying to figure out why your Zimlet doesn’t work or when you want to implement some particular feature in it.

Scott: When Sean and I were at OSCON last year, people kept starting off their presentations with almost an apology for being a for-profit company.

How do you balance between your commitment to the community and the need to be profitable? As you work through decisions about what to offer as open and free versus what to put only in the enterprise version, how do you communicate with the community about those issues?

Kevin: We’ve given a fair amount of attention to that. Others at Zimbra might answer this differently, but from my perspective, I’ve always remained focused on what it was that we were giving to the community, and to creating a fantastic open source Collaboration Suite. We think we have done that with the constraints that we’ve put on in it, in terms of not handing out five or 10 additional features.

A key consideration for me has been that, while we may choose to limit the distribution of some features, doing that should not prohibit someone from being successful with the open source version.

Sean: Since your customer base tends more toward IT admins and less toward developers, that must inhibit the standard open source model of people contributing the features they want directly.

So what’s the filtration mechanism in terms of your average IT guy saying, “I don’t like the way you built this interface, ” or, “My users are complaining about the way the web email looks, ” and that kind of thing? How do you filter those requests to get them into the product; I find that for some open source projects, that’s a little bit of a struggle.

Kevin: I’ll make a couple of comments on that, and the first is that, even without the contributions of the community, there are a lot of people in Zimbra who are passionate about usability and the interface. They’ve done a lot of thinking and work and really had some tremendous ideas in order to enable us to create the application that we have. That’s definitely been a part of our success there.

The other part where the community has been very helpful has been interacting with us via Bugzilla. And we have built forums that we spend a fair amount of time on, but ultimately for us, for something to get done, there needs to be an entry in Bugzilla.

We’ve tried to steer the community folks into putting their ideas into Bugzilla, and we also have a couple of guys on staff whose primary job is to work with the community. Part of that is going through the forums and trying to figure out what people are complaining about, or else where someone has a great idea. We make sure we get that into Bugzilla so we can evaluate it.

The other part that has helped us is that pretty much everybody knows how to use email, so we’re not developing an esoteric product. We’re developing something that all of us use every single day. Of course, we run Zimbra internally, and have for several years now, and that has helped us work on usability as well.

Scott: Would you like to close off with something?

Kevin: To me, the most educational part of this process has been learning how to work effectively with the community, and how to integrate their input into the product management process. Then how to also work with them to give them something that, in terms of the beta cycle, gives the community something that’s good enough to be useful, yet immature enough to still give them an opportunity to really impact the quality and the feature set before it got released.

We’ve spent a lot of time working with folks in the forums, and we’ve built tools around Bugzilla to let us view the data that gets in there from the community. We had to set up some processes for when the community does file a bug, what we do with it. How do we make sure that all the information we need is in there? How do we make sure that we can reproduce it, and that we know exactly what the problem statement is before it goes to development?

Certainly, the community adds a lot of value. But you don’t want your development team to turn into a support staff for the community, at least not if you’re going to scale to a very large community base, which we certainly want to see happen with Zimbra. To me, that’s been an engineering management challenge, and it’s been a lot of fun too; the end reward is really great. I’m sure we wouldn’t be where we are today with the feature set and the quality that we have, if we didn’t have the community helping us.

We’re also really excited about the future. We continue to add big features to the product, the most recent being Zimbra Desktop and our support for BlackBerry devices with Zimbra Connector for BES. We’ll keep bringing the Zimbra experience and data to new platforms. And we’re going to launch a hosted Zimbra offering shortly that will make it even easier for admins to provide their users with the Zimbra experience. The community keeps growing, too, and we’re excited to see that continue in 2008.

Scott: It’s really been great talking to you.

Kevin: Thank you.

Bookmark this:

Interviewee: Wade Olson

In this interview we talk with Wade Olson from the KDE project. In specific, we talk about:

• Background of KDE and comparison to GNOME
• Ensuring usability in an open source project
• Managing interoperability among components
• Impact of commercial acquisition of open source projects
• Cross-platform support in KDE
• Reaction to KDE 4 public release

Sean Campbell: Wade, give us a little bit of background on your role in KDE.

Wade Olson: I have been involved with KDE for several years now, both as a contributor and as a member of the e.V., which is the nonprofit organization that represents and takes care of our KDE members. And for three years, I’ve also been in the Marketing Working Group.

I think the origins of KDE are fairly common knowledge; it’s one of the desktop environments for Linux and other operating systems, and we have thousands of contributors worldwide. Its a full-blown desktop environment that handles not only window management, but also the entire application framework, to allow for an integrated experience for the end user.

• http://www.kde.org/announcements/announcement.php
• http://en.wikipedia.org/wiki/KDE#History

Sean: What do you see as the key differences are between KDE and GNOME? That seems to be the obvious comparison.

Wade: Obviously, there are a lot of similarities, and there has been a lot of work in building relationships between the two communities. The most obvious technical difference is that they are built upon two different toolkits–C versus C++, GTK+ versus the Qt toolkit. And then also philosophically speaking, GNOME has really put a lot of fantastic effort into the usability and GNOME interface guidelines to allow for a streamlined experience, and they really make sure that their interfaces are consistent. We focus more on the advantages of an interface that gives the user control, flexibility, and configurability.

That being said, when I talk about the similarities between the two desktops, don’t forget that there are various Linux architecture and desktop architecture groups that ensure that everything is done so that the different desktops and their applications are more integrated than ever before.

That might involve such things as the D-Bus system, on which we’ve both standardized, as well as smaller but important things like naming conventions and directory standards for icons so that the different theming engines can make sure to apply icons as expected for Tango or Oxygen or whatever icon themes you’re using.

• http://www.freedesktop.org/wiki/Software/dbus
• http://tango.freedesktop.org/Tango_Desktop_Project
• http://www.oxygen-icons.org/

Scott Swigart: The industry in general has a pretty laser-like focus on usability these days. How do you ensure and test for that? Just like you can’t have a developer check their own code, you definitely can’t let them police usability on their own code, so how do you handle that operationally?

Wade: It’s a fairly complex process. Unlike commercial software efforts, there’s not a strong business analyst layer where they actually go through use cases and make sure that the software adheres to some list of requirements or extra documentation.

Typically, you have members of a technical community who have to put on multiple hats or play multiple roles in the software design, requirements, and analysis phases. That’s how in the past, some Free Software projects have gotten into a little bit of a bind, in terms of usability adherence or making offerings that end up being as pretty or flexible or intuitive as commercial offerings.

To improve usability, we’ve been putting a lot of energy into making sure that all of our development staff understand the importance of usability as a key component for adoption and how to look at their code from a user’s perspective. Fortunately, we have talented usability experts in the KDE community that not only educate sub-communities on usability, they also do reviews and provide constructive feedback.

There have been plenty of instances where there may be competing software applications that are quite similar in what they can accomplish, but in the Free Software world, if one is more intuitive and usable than the other, then the uptake of that one is going to be larger.

The problem is always the person hours needed to do all the things you need to keep up on, like writing the documentation, building the web pages, writing the code, deploying it, and talk to distros that use it.

We’re expanding our user base to more and more people who are not technical and people who are used to commercial operating systems like Windows or Mac, where a lot of money gets spent on interface work and usability work.

Those users have expectations about how easy something should be to pick up, and frankly, it doesn’t matter if it’s free. If they install something or see a friend using it and it looks completely foreign or confusing to them, they’re going to stick with Windows and Mac, and we’re going to miss a chance to get someone to change over.

Sean: There seem to be a lot of retailers trying out selling Linux boxes now–how much are you guys driving towards the desktop Linux user?

Wade: KDE focuses on the desktop environment and the desktop experience, so we are somewhat biased towards that experience. Now, obviously, Linux kernel hackers and BSD kernel hackers can certainly deal with virtualization, hypervisors, and whatever they want, and we have hooks into those layers through the hardware abstraction layer and so forth.

We are certainly interested in general in hooking to the various operating system layers and kernels and being as stable and flexible as we can be with the plugging in of devices, unplugging of devices, hardware recognition, et cetera. But, just by the nature of KDE, more likely than not we’re going to focus on the desktop experience and what we can do to make that the best experience possible.

This is also another layer–people typically don’t go straight to KDE when talking about desktop experience. So if you’re buying a pre-built system off the shelf, there’s a distro in between KDE and the hardware manufacturer. A couple of obvious cases are PC BSD, which is working with iXsystems to make hardware boxes that they sell throughout the U.S., as well as Xandros and the ASUS Eee PC, which is extremely popular throughout the U.S.

So we have hardware from ASUS and iXsystems that help KDE target specific markets. There are people from Xandros and PC BSD that obviously have industry connections that benefit us–we don’t necessarily sit down and think about what markets we want to target.

Scott: There are a lot of components from independent projects that come together, and in a sense, KDE is almost a distro within a distro. How much work does it take to get all of those independent pieces to work together smoothly, especially since it has to work on so many different OSs?

Wade: Everything is obviously driven off the Trolltech Qt Toolkit, and then on top of that there is an additional layer with the KDE Libraries, which is very specific to our desktop environment.

In terms of interoperability, there’s kind of a natural, organic process that subcommunities take when growing out and branching. When you think about the natural growth or evolution, not just KDE but with free software projects in general, something might start with just one or two people that are writing some code. That might grow into a couple more people, and then maybe they have the need for a web page, and then maybe they have the need to be in a larger source code repository. Later, they might start promoting, then start translating it, and so on and so forth.

Typically there’s a natural trajectory that applications take in their growth. Along the way, as individual KDE applications get more sophisticated. they naturally begin to use pre-existing code and connect with other KDE applications. Time spent on development is precious; it should be as productive as possible. It’s not coincidence that community members have created a software atmosphere to enable developers. It has taken a massive amount of effort to make it happen effortlessly, if that makes any sense.

Scott: It sounds, then, like KDE is built up of a lot of different components, but each one of those components is responsible for ensuring their portability to the different platforms, and those different components work together to interoperate with the other components that comprise the overall shell.

Wade: Right, exactly. The first key to interoperability is actually the e.V. organization that I mentioned before, which is a worldwide group that basically keeps track of everybody and makes sure that they all play nicely together. We also rely heavily on IRC and mailing lists, which is where a lot of the communication is done. And that communication is typically involved in adhering to coding guidelines, adhering to usability standards, and really allowing people to say, “Hey, I want to do this,” and that’s where the technical discussions come in–”What’s the best way to implement that?”

We also have working groups. As I mentioned, I’m part of the marketing working group, but we also have a usability working group and other organizations that help keep people in sync in different ways. With KDE 4–as you may have read in some of my blog entries and articles–a very large part of what we tried to accomplish was to set up a common language and infrastructure in what we called the Pillars of KDE. So if you look at a multimedia framework with Phonon, the look and feel across websites and our desktop environment via Oxygen, hardware interaction, PIM data storage with Akonadi, almost everything we did was to create a pillar for a common language in a common technology to allow people to quickly and easily build applications.

If you abstract away all that complexity, almost by nature you get people working on the same page, when you take away the difficulty of connecting to a CD ROM or an FTP server, for example. It’s important for code consistency, it’s important for code complexity, and it’s also important for making developers as productive as possible during their limited time, since, for many people, this isn’t their day job.

Sean: Let’s talk about the Trolltech/Qt acquisition, starting with the question of why you think Nokia bought it, and what do you think they’re going to do with it, and what effect does it have on you?

There’s an interesting dynamic happening in the past year or so, where companies decide they love a certain open source project, so they buy it and plan to build on top of it. A lot of nervousness tends to develop around those acquisitions, about things like how much there is still going to be a community process behind it and how comfortable developers are going to be continuing to contribute.

There are successes all around, like Red Hat’s been great with CentOS. The jury is still out about MySQL with Sun, since it’s still a brand new thing, but Qt must have cut right to the bone with you guys–what’s your feeling about it?

Wade: Let me preface by stating that I’m neither a Trolltech nor Nokia employee, so opinion has the same weight as any other observer. If you look at issues that people have had philosophically and historically with KDE, from a Free Software standpoint, it was with the affiliation with Trolltech and Qt and the licensing. So because of that, for the past several years, Trolltech has truly gone out of their way to try to eliminate all of those obstacles and barriers and frustrations that some of the people that are on the very free side of the open source spectrum have had with this affiliation.

They’ve created various poison pill contracts and things that have really said that no matter what happens, Qt is going to be there for KDE to use. And there’s no reason to expect that there was anything nefarious going on. This was with best intentions, and they have certainly spent time and money with lawyers to make sure that everything was locked up from a licensing standpoint. We take them at their word, and we are obviously appreciative of that work.

I don’t think there’s a lot of necessary concern or fear from the fact that it might go away, because certainly there are various abilities to fork and continue to use this freely. It’s not as if it just goes away, and KDE is left standing out in the cold.

With Trolltech lately, we’ve been hitting this great stride. You may or may not realize that they do sponsor several KDE developers, as well as get-togethers and meetings that we have. Some of the code they write now with WebKit is beneficial to us from a browser perspective. They have integrated WebKit into their code repository, based off of KHTML and work by Apple in the WebKit community. And very recently, they’ve been doing all of the Phonon multimedia work in our SVN repository, completely openly.

So, really, I think the only concern that I’ve seen in the KDE community is the fact that things have been going so well that we just are praying that it continues with Nokia and that things don’t slow up. Might they fund as much? I have no idea. Might the developers be forced to work on other things? I have no idea. I just know that things have been going so swimmingly, that’s the only apprehension.

I do know that people are talking with Nokia, and Nokia is doing all the right things thus far, as I understand, as far as scheduling meetings and talking about intentions. I don’t think they want to screw up a good thing, and trust me, with that toolkit, there is certainly a hugely beneficial relationship between KDE and Trolltech. Who better to test your toolkit and provide feedback than hundreds of talented programmers stressing its capabilities and millions of users?

Sean: How does a closed source company not screw up the acquisition of an open source one, in your mind?

Wade: I think the key is to really go back and actually look at the various motivations of why someone would commit their own personal time and vested interest to work on something like this. We’ve already talked about why some open source projects work and why some fail. It’s all about people having an emotional connection and attachments that make it more worthwhile to them than doing something else, whether it’s working down at a soup kitchen, working for a nonprofit, or sanding their hardwood floors, for that matter.

Some people spend their free time doing it, and some people get paid to do it. But either way, it’s the emotional connection, and it’s the passion, and it’s the community that builds around it, and the friendships. Trust me, KDE’s code is exciting, but it’s not that exciting. People love to work on it because they love to get together and talk about it, and they have the same passions and interests. That’s where companies really need to look. They cannot screw up that ecosystem.

Even a company that really gets open source–like Red Hat, when they buy JBoss–has to think about the community. And the fear is, what if all of the people go away? Because now it’s a corporate entity that people might not be interested in, instead of their little, home grown, comfortable part of the world.

Scott: Old-school open source projects like Linux and Apache don’t really have a chief sponsor that’s driving the development. But some of the more modern open source projects–like Alfresco, Zed, and PHP–have a company wrapped around them that’s really driving a lot of the innovation, and we’re starting to see a wave of those companies being acquired by these much larger companies.

The thing that I wonder is, if these large companies do screw it up, they have essentially paid a whole bunch of money for nothing, because one thing open source lets you do is fork the code. It isn’t far enough into the future yet to see how this all plays out, but one could imagine that if Sun really mismanages MySQL, somebody else could form a company around it. The same thing with Qt or Zen or any of these other things.

Wade: Obviously, because of licensing considerations, different projects have different options when things like this happen. I have not pored through the licensing on Qt in particular, and I can’t tell you what will or will not happen, or how poison pills are engaged, or what the ability is to fork it.

In general, though, not only are companies paying for historical results, but they’re paying for future results as well. They are certainly are not just paying for the MySQL historical codebase and saying, “OK, thanks, we’ll take it over now,” but they still want those people to contribute, and they still want it built upon.

Scott: Right–they’re paying for the mindshare. To me, what they acquire when they acquire these companies is not the code, but the expertise around the code.

Wade: A somewhat similar situation could be considered with any open source community that relies on Java, for example, like the Eclipse community or JUnit, or any one of many Sourceforge projects. Sun could have taken them any which way. They could have closed down Java, or locked it down, or stopped working on it, or open sourced it. That was up to some speculation as well.

You have to hook your horse up to a cart, and that has obviously played some role in the GNOME community saying “We want GTK. We want our own toolkit, and we want it to be free.” There are benefits and detriments of closed source versus open source. Qt is what it is, and it’s incredibly powerful, but it’s not entirely free because of that.

Scott: Remind me what different platforms KDE runs on outside of Linux.

Wade: It runs on BSD and Linux and any *nix platform, and there’s actually a lot of renewed interest in having it run on Solaris. Obviously when we talk about Plasma, the desktop itself, that’s only going to run on Unix like systems that we already do. On the application level, as long as you have the KDE libraries and the Qt framework–which is cross platform–then it is game on. Going forward, we’ll see KDE applications run and run well on Windows and Mac platforms.

It’s just a matter of figuring out all the nuances–what exactly makes something cross platform in theory as well as practice: installation directories, multimedia engines, rendering models–the stuff that makes Windows and Mac different.

Scott: I guess the reason I ask is that to projects like OpenOffice for example, cross-platform is very important, whereas it seems like KDE could get away with being mainly a Linux thing.

On the other hand, OSs like Solaris and OpenSolaris might be important because they create a little bit of competition with Linux, which is probably healthy for both Solaris and Linux. In your view, how important is it for KDE to be cross platform?

Wade: There actually has been a fair amount of debate on that approaching the 4.0 release. Obviously, when people work on things that they want to work on, there’s really no stopping anyone. It’s not as if you can just assign someone to work on a task. It’s not a development job where you can say, “I’m sorry, I don’t want you working on that anymore. I’m assigning you to project 123.”

If you don’t want to waste resources having someone work on KDE and Solaris, you can’t just say “Go back to working on BSD. We need to fix that up.” It’s really all based on people’s interest level, and that’s the secret to great code–when someone has passion for an area.

It comes down to the “can do” versus “should do” question–we can do a lot, but should we do it? From a marketing standpoint, my stance is that if someone’s going to work on it, just please do it well. If you think about how people get exposed to KDE and open source and Free Software, you always want the first impression to be a good one.

KDE is obviously at home on Unix and Unix-like operating systems, and then there was the expanded ability that people could view it through VNC or RDP servers to actually connect to a remote client and see what KDE and open source and Free Software is like.

Then it was expanded further with the Live CDs like Knoppix that spin up. Then it was expanded again with VMware Server and other virtualized servers, where you could literally be on Windows and just get a VMware image and boot it up and see what it’s like. The final frontier, then, is the application layer where we’re actually installing and deploying applications on other platforms.

Again, from a marketing standpoint, I just say please make it a good first impression. I am not interested in leaving a Windows or a Mac user with the impression that it’s half baked or not as good as it should be.

There was debate about whether would it drive people to use KDE or Free Software in general. We talked about whether OpenOffice and Firefox have necessarily gotten more users onto platforms like Linux or BSD or Solaris. A lot of Windows users probably say, “Hey, that’s great. I’m going to continue using Windows with these free applications.” Is that a good thing or a bad thing?

So, if you’re porting for KDE adoption and recognition from an application perspective, you’ve got millions more people that you could reach. However, if you’re looking for people to make the move philosophically to Free Software and to have that desktop environment, maybe you’re actually hindering your cause. That’s where the debate comes in, and that was an interesting period of time.

People basically said do what you want to do, and please do it well. If nothing else, what we’re looking for is to increase visibility and awareness of KDE and the world class software that we’re making. In addition, we want to let developers know that a lot of people that write free software for Windows and Mac, and that there are other options that they have as well that they might take an interest in, and we might grow our developer base.

Scott: That’s always been one of the things that’s interesting to me about open software. Closed source sometimes has to guess what people are going to want. They have to be very speculative about the features and plan the next release and try to make sure they have “killer features” that are going to make people upgrade and that kind of stuff.

Open source often takes the approach that if a feature or capability is really important, it’s important enough that somebody will show up to do the work.” And if nobody does, it shows that maybe it’s not that important.

Wade: Right. You figure out the source of the priority and the urgency. Was it Sun Microsystems saying, “I want this to work well”? Was it a couple of influential users? Was it something strategically that needed to be done? It’s very interesting what is worked on and not worked on in free software, and that’s obviously not specific to KDE.

Scott: Great. A lot of times, when we get to this point in these conversations, we’ll hand the microphone to the interviewee and say, “What did we not ask about that you think is interesting?”

Wade: This interview is happening just after our big 4.0 release, which we worked on in parallel to the 3.5 series for years. That’s a big deal, and it really is like the architecting and building a house analogy, where we have worked on the foundation to make this series one that is just going to be modern and performant and stable, like any other commercial offering, or better, over the next several years.

And certainly, there has been much press and discussion about whether we released too late, too soon, what the expectations were, did we over-hype it, etc. A lot of people were saying, “It should’ve had another release cycle,” or “It should have had more QA,” or “Why didn’t you just wait for the next thing?” Everybody chimes in with their pet project and why it should have just waited just a little bit longer, until their particular area of interest had been satisfied. So that’s certainly one thing that was tough.

Obviously, there was a massive amount of forethought and discussion and reasoning, because the 4.0 release was very significant, and any time you’re planning on something to be the foundation of your entire community for the next 5, 6, 17 years, you don’t do so lightly.

We’re very excited about it, and I think people are going to see that the upcoming releases are going to show very accelerated development. During the crunch time leading up to the 4.0 release, when you looked at the speed of commits to our code base, as well as the number of people working on it, just as with any software project right before the release, you saw a swelling of activity.

What’s been amazing is that it hasn’t slowed sown, now that the release is out. People did not take a breather and say, “Oh my God, I made it across the finish line. We got Version 4 out.” It’s actually only grown. Week after week, we have more and more application developers going in and saying, “I’m going to use that framework and port my application over to the KDE 4 code base. I’m going to use these new libraries and toolkits and Pillars of KDE that you’ve given me.”

With upcoming releases, I think that’s going to prove out how quickly this code base matures and how quickly things become stable.

Scott: There’s no better measure of success than people adopting it and moving toward it. In any community, but especially in open source, you’re always kind of herding cats, and there are always going to be strong opinions and outspoken people. You can never be everything to everybody, but if you’re getting good adoption, that’s the best validation that you could ask for.

Wade: I think, historically, things are going to point to this being the correct decision, to release at this time, because the foundation, the architecture, the pillars were sufficiently mature, from an API freezing standpoint. We’ve really seen an uptick in the QA loop, the feedback, and the amount of people that are going in and reporting bugs. A lot of applications are going in and using this technology now.

Obviously, if we would have delayed to work further on some things, we would have postponed the involvement of other groups, such as translators and documentation writers and QA and things like that.

The important thing is that at our release event, we announced that we have a very standardized release schedule going forward, so that distros can plan on those releases happening, where we have smaller sub point releases every month. And we’ve adhered to that thus far, where, in February, and now in March, we’ve had 4.0.1 and 4.0.2, and that the major point releases are going to be every six months, and so we plan on 4.1 coming out late this July.

Scott: Well, thanks for talking today. This has been a great conversation.

Wade: Thank you.

Bookmark this:

Interviewee: Jim Gettys

In this interview we talk with Jim Gettys – VP of Software Engineering on the OLPC project. In specific, we talk about:

• OLPC base systems software
• OLPC user-facing software
• Platform innovation for the developing world
• Future directions for the OLPC platform
• Engaging kids with OLPC and open software

Sean Campbell: Hi, Jim. Could you introduce yourself and your role with OLPC?

Jim Gettys: Sure. I was one of the first people working on OLPC, and have been the vice president of software. We split the software tasks into two pieces–the base systems and the child-facing software. I deal primarily with the base system software.

I’m also one of the original authors of the X Window system, and I was the editor of the HTTP 1.1 specification for a long time.

Sean: Tell us a little bit about the software on OLPC machines.

Jim: We’re running Linux on the machines, which allows us to take a number of approaches that have been very difficult to do on other systems. Many of the children in many parts of the world–in fact, a majority of children in the world–go home to locations that have no electricity whatsoever, and many of them have schools with no electricity as well.

So, we’ve worked very hard not only on cost but also to make low-power systems, because not only is it hard to get the power, but power is very expensive. That makes it a fundamental design requirement, because just using everyday laptops would leave over half the world’s kids out in the cold. At a school Nicholas Negroponte equipped several years ago in rural Thailand with conventional laptops, fuel for the generator became the most expensive operating expense at that school. And, we’ve built the most rugged laptop in the world.

We focus on two goals in our user interface.

First is enabling collaboration. Most kids learn from other kids, and we also need to enable teachers to collaborate with the kids.

Secondly, a big issue is that conventional interfaces–and it doesn’t matter whether it’s Windows or Macintosh or Linux–are all really inappropriate for young kids, because it’s really hard to use a conventional desktop before you can read, and even more difficult if the child’s parents cannot read either, leaving the child in the dark with no help. How many times have your children come to you asking “What does this dialog box mean?”

Sean: Obviously, you also want a machine that can be distributed with a standard configuration, regardless of the locale or language, right?

Jim: The parents may never have seen a computer before, and in one of the trials, during early discussions about giving each of the kids their own laptop, one of the parents asked, “What is the Internet?”

So we can’t presume that the people know what the Internet is or that the parents can read or the kids can read. It’s a very different world out there than most people think it is.

Sean: It reminds me of well-intentioned efforts where, for instance, someone gives a remote village tractors, but they don’t have a fuel-distribution system. So, six months later, they’re all rusting in the fields.

Jim: We have to presume that there isn’t much, if any, infrastructure.

Sean: I understand that you started with Fedora Linux as your base OS. How did the target populations, areas for deployment, and other constraints that you have to deal with affect that decision?

Jim: Other than worrying about power management and suspended resume in a serious way, the main thing was that we needed a very small base system, because we need to fit everything onto the one gigabyte of flash on the machine.

While our friends at Microsoft are putting Windows onto the OLPC, they don’t expect to be able to actually run off of the base one gigabyte flash, so they have to add an SD card to have enough space, which increases the unit cost.

Other than configuring Fedora to be quite small, at the base system level, there wasn’t a huge amount of engineering work.

Scott Swigart: There seems to be some symmetry between your base set of software that’s going to run on the box and a prepackaged OEM image, but you take it another whole step. This software may never get updated–it may sit on that box exactly as is, and the users may not have the capability, for a period of time, to really even understand how they could update it.

Jim: We need the machines to be usable by the time the kids get them, by themselves. While we hope for and deployments plan to provide Internet access, there is no guarantee that this access will be initially available or reliable. But if Internet is available, we expect regular updates, both for security and for new applications (activities in our terminology).

Scott: Since you can’t expect them to buy their own word processor, for example, you must have had to make a lot of tough decisions about what went into that base software build to get the most value out of that limited memory.

Jim: We have a very large spectrum of software to choose from. Last time I looked, Debian had more than 20,000 software packages, and Fedora has, I think, about 10,000. And there are many options for any given piece of software, so the main thing is careful picking and choosing.

One good example is an office suite. Something like Microsoft Office or OpenOffice is completely inappropriate for a seven-year old. What a child needs is a simple, basic text editor that can at least support font changing and basic simple formatting, but doesn’t throw in the kitchen sink.

Rather than starting from scratch, we’re using a derivative of a program called AbiWord, which runs on both Linux and Windows. It’s much simpler than either OpenOffice or Microsoft Office, and in fact, my 13-year old prefers it to either of them, just because it’s much simpler and it’s all she wants or needs.

That’s what matters for kids; and most children (lucky enough to attend school) only get 5-6 years of basic primary education. We are not training these kids to be developed-world office workers who are doing high-powered PowerPoint presentations or OpenOffice Presents presentations for management. Learning to read and write successfully is what they’re in school to do.

We chose AbiWord because, above and beyond the fact that it’s a lot simpler and smaller than OpenOffice, it also supports real-time collaboration. So two or three kids can actually sit down and share a document in real-time, and they can all be editing on it, or the teacher can work with a student that way and actually be able to edit the document with the child, in real-time. That capability is really important to us, so the kids can learn from each other and from their teachers.

Our point of view is not that more complex things shouldn’t be eventually available to the kids. The set of people we aim at is really fundamentally different than where most laptops have been used.

Sean: You mentioned AbiWord’s collaborative capabilities and the fact that there’s some novel stuff in the system.

Jim: There are a couple of really novel things in this software, both at the base technology level and at the user-interface level, and one of the things that drives that is the view that kids and their teachers will be constantly collaborating.

From our perspective, it’s essential that two or three children can sit down underneath a tree and be able to work together on their computer, without having to go through some weird network configuration exercise. A group of seven-year-olds should be able to immediately run things like this collaborative version of AbiWord, very, very easily.

The concept of “presence” and being able to show you who’s around you in your network area has been brought really far forward in our user interface. We made it really easy for the kids to find out who’s out there doing what, and to very easily invite their friends to work with them. From a user-facing point of view, the whole concept of presence is brought all the way to the front.

Otherwise, while things like what has been done with AbiWord have been possible to do before, the configuration of collaboration has been so painful that it has typically not happened. Abiword by itself has had these collaboration features for a while, but they, as in other such programs, have gone nearly unused since they have been too difficult to configure.

We have succeeded at changing that equation, both to make it very easy from a user-interface standpoint for the kids and their teachers to be able to work together, and also to make it a lot easier for the application developer to actually build a collaborative application in the first place.

Our core applications, for example, now do that kind of sharing. The kids can work on a document together, play music together, and if you want to look at a program and edit it into something else, then the kids can share the text editor of the program, and so on and so forth.

Collaboration is fundamental to our view of what the software should be, because the kids learn from each other and from their teachers most of the time. This is not an isolated world where you should only work on the thing you’re supposed to be working on by yourself, right?

Sean: It seems to me that there are a number of novel things in the platform. There’s the power management that you’ve talked about, there’s the mesh network, there’s the display technology. At the systems level, some of the stuff in the system was pretty novel.

Jim: For example, at the power management level, just making it work fast is really important. I don’t know what your laptop is like, but my HP laptop takes over ten seconds to resume.

There’s no reason why our systems should take so long to resume. To give you an idea, when I measured a 200 megahertz StrongARM running Linux, it was able to go from coming out of reset until it was scheduling user-level processes in ten milliseconds. I measured that on an oscilloscope with one of the other people working here, to confirm that I didn’t screw up the measurement. The OLPC XO-1 hardware is believed good to about 50ms; right now, Linux’s USB stack is taking most of our current approximately 1 second resume time.

Scott: That’s pretty darn fast.

Jim: That’s the way systems can work. Why doesn’t your laptop resume quickly?

Sean: It seems to me that you guys did do a fair amount of groundbreaking R&D to build this, some of which will probably percolate out to the wider community.

Jim: That’s why our $188 machine takes about a second to wake up, and my HP laptop that cost $2,000 takes ten seconds.

There was actually quite a saga about getting our hardware to work properly for resume. I blogged about it quite a bit, and you can read my entries about how painful it was getting the hardware to work properly for suspend/resume.

Sean: Some other stuff too, like the mesh network, seems like it would have to have a lot of impact at the base system level.

Jim: Well, yes, and no. The issue is that, since most kids go home at night and don’t have electricity, we can’t presume that there are places where there’s either an access point to plug in or an Internet cable to that access point.

That just doesn’t exist in these places. Yet we’d like, as much as possible, for the kids to be able to work together, and preferably to be able to access stuff in the school as well. The point of the mesh networking is, in large part, to make it possible to extend the reach of the network to a larger area. The mesh does that, and does that quite effectively, although there are still some interesting problems at a technical level.

The main point is that, if you have a kid, and there are kids who live between him and the school, you’d like it to be possible for him to still get connectivity back to the school, or to other kids. That way, the set of people who can work together is much larger than just those who can hear each other directly.

Scott: Can you talk about memory use? The average programmer today is insanely sloppy. I have four gig of memory on my laptop; I have six gig on a quad, on the corner of my desk. Nobody cares any more.

Jim: Those who like software that actually works fast will find that the smaller they make it, the faster it runs. But yes, many programmers have gotten very, very sloppy, and they only make things run fast enough.

One of the nice things to have been watching is that, starting about a year and half ago, the Firefox people realized that they had a memory disaster on their hands, and they’ve been working very seriously on making Firefox’s memory footprint much smaller than it was; recent data shows Firefox 3 will likely beat all comers, which I expect will force everyone’s hands to do better. For example, when I look at my Firefox usage now, it never has a virtual address space of more than about 200 to 250 megabytes, whereas a year ago, it would have easily been 500 or a gigabyte.

Sean: One other thing I’m curious about–why Fedora over something else?

Jim: Red Hat put up a significant number of engineers to help us out, and the choice of which Linux distribution really doesn’t make that much difference.

Sean: What’s an example of something that didn’t make it into the final design, hardware-wise?

Jim: One of the things that wasn’t feasible at the time we started building this generation of hardware–in terms of both memory and power–was 3-D hardware, although we do have alpha blending in both hardware and software. We’d also have loved a touch screen, which was not feasible.

There are a few really wonderful 3-D education applications, and I certainly would have liked to have had that. We will as technology moves on.

Sean: What are some of the other things you feel like are on the road map? It sounds like there is probably a lot of innovation that you can continue to do at a software level on the existing hardware.

Jim: The software will never be done.

Sean: But the hardware is going to continue to evolve, too. On both fronts, what do you see coming?

Jim: Cheaper or lower power. Those are our first two things. Before capability, we prioritize cost and power, so the next generation has to run at significantly less power and cost significantly less. Anything we can get without impacting that, of course, we would say yes to.

The reasoning for both of those two primary goals is pretty obvious: to get machines to more of the kids in the world and to make power less of a problem for them.

Power drives cost in hidden ways that most people don’t think about. As soon as you get off the grid, the cost of power goes way, way up. As noted above, Nicholas’s school in Thailand shows how unforgiving this problem is, and it has been entirely neglected by the computer industry. Ruggedness is also key; our system will withstand much harsher treatment and environments than conventional systems can; we need to push yet further in these areas.

If you use solar panels, which we are using to charge our machines in many locations, the cost is very sensitive to the amount of power consumption, so lower power reduces costs all through the system.

We want to get it below $100, and the system-level power consumption needs to get down to well under a watt. Understand that with our display in gray-scale mode, the display by itself only takes a couple of milliwatts. I’d like to have a system where the system itself is not consuming very much either.

While significant power savings are beginning to show up by others following our lead using LED backlights industry wide, conventional flat-panels still take more power, due to the widespread use of LVDS drivers.

Sean: In your initial deployment, outside of the areas of power and cost, are your initial users encouraging you to go in a certain direction? What is some of the broad feedback that you’re getting?

Jim: It varies a lot. For instance, people who want to deploy it in urban areas in developed parts of the world respond to it as just another laptop, just maybe cheaper and more rugged.

It’s when you get into the more remote areas that you start hearing things like “oh, this thing can actually be repaired by a kid with a single screwdriver,” and “its power consumption is a quarter or tenth of the alternative.” Some aspects of the machine are valued very differently, depending on the environment you’re in.

Sean: I was thinking more of the developing world users, where the parents may never work with a computer. What’s coming out of that that’s helping to drive your thinking?

Jim: The really nice thing is hearing back from the kids themselves. A lot of that is anecdotal, but the easiest numbers to get quickly have to do with truancy and kids coming to school because they want to, for a change.

I think the more encouraging things are what impact the machines have on the kids and their families. In some of the trials, parents have gone from thinking of the laptops as a foreign and potentially bad thing to regarding them as a possible way for their kids to have a better life.

One of the really amazing things was a description I heard when I visited trials in Peru. One of the teachers said that they had a child who was not getting enough to eat and he had been very disruptive. Since having a computer, he was now engaged in the school and learning, and he had become quite an expert at using the computer. Rather than being disruptive, he was now actively helping teach other kids in the classroom. This kid has gone from being disruptive to being an asset in the school.

Sean: I remember having a Commodore 64 as a kid, and that feeling really seems to make sense to me. Some people seem to innately gravitate toward computers, and other people don’t.

Jim: Whatever their passion is, we want to get the kids engaged in learning. For some kids, it happens to be computing itself. In other kids, it may be music. In other kids, it may be reading literature. In other kids, it may be mathematics. The system should enable the child to go where their passions lead them.

Scott: Another thing I wanted to ask you about is, how do individual software packages get into the project?

Jim: There are a small set of things that you can consider to be really vital, core functions of a computer–a web browser, a simple text editor, and that sort of thing.

The other piece of the question is more subtle. One of the problems, from our perspective, that is now plaguing the entire industry is that, due to security concerns and insecure systems, the behavior of system administrators has been to lock down systems such that you can never exchange software.

That flies in the face of having a vibrant community of people–including kids–developing new activities to be able to share. The collaboration I talked about before includes collaboratively building new software together. That’s one of the areas where we want to be able to grab the kids’ passions.

We’ve worked quite a bit on trying to increase the level of security to make it safe for activities to be shared between people. We worry about security as much as we do because that’s the only way to really get an explosion of interesting new software and for the kids and their teachers to be able to build new things and share them among themselves and kids all over the world.

One of the delightful surprises that shows how persistent people can be is Ben Schwartz, a student at Harvard who, regrettably, I initially overlooked in getting machines to for developing software. He wrote the wonderful “acoustic tape measure” program for measuring distance with audio.

You take two XOs and point them at each other, and by timing the difference between radio and the acoustic signature that gets transmitted from the speaker, you can tell how far apart they are. This Harvard student did this on his own time. He wrote this activity and had it working before he had his own machine for testing.

Sean: That’s a great example of a programmer making the physical world do something. It’s a powerful feeling when you first use code to physically move something in the environment, whether it’s the CD tray or whatever.

Ideally, if you build a good platform, whether it’s a software platform or an API or whatever, it’s a platform for experimentation. People do stuff with it that you maybe never thought of. It seems like it’s easy for software.

Jim: The acoustic tape measure is a classic example of that. We didn’t think about that activity in advance. We also have a wonderful activity that exploits the XO-1′s unique audio input port (which enables DC input). Every child has a simple oscilloscope and data logger at their finger-tips, that can use extremely low cost sensors, such as a simple thermistor. Computing is now a basic skill in our culture. For interested children, it is very important that the computer itself be transparent. That our platform is an open-source platform allows a child to investigate all the way down to the first instructions that the machine executes. We want the kids to be able to learn computing at a deep, fundamental level. And that’s hard to do with commercial software, where the software hides how it works.

We have a special key on the keyboard that will show you the source to applications. That is now getting used in more and more of our applications. If you’re in the web browser, for example, it shows you what the HTML is underneath.

The show source key in most of our applications and activities shows you the Python or other source that defines that activity immediately, and it allows for editing and experimentation.

Or if you’re in Audio Editor that’s in the process of coming together right now, when you first hit that key, you’ll see the basic application. If you hit it again, then you can see things like what the MIDI sequence might be for the music you’re playing. If you hit it again, you can look at the Csound program that defines how to synthesize that sound.

So we’re able to show people what goes on behind the curtains.

Sean: That’s great, Jim. Thanks for taking the time to talk today.

Jim: Thank you.

Bookmark this:

Interviewee: Mike Shaver

In this interview with talk with Mike Shaver, Chief Evangelist for Mozilla. In specific, we talk about:

• Governance of the project and how people get involved
• Changing contributors who fill key roles
• Roles of independent contributors versus employees
• Ensuring high usability in an open-source project
• Protecting reliability in a world of many extensions
• Tools used by the Mozilla team
• Reflections on open and closed development models

Scott Swigart: There are a lot of different governance models for open source projects. With Sun projects, for example, they often have it written that control of the project has to rest with Sun employees. Apache has a different model. Apache projects have to have maintainers who are from at least three different companies so that no one company can hijack a project.

Talk a little bit about how the Firefox kind of governance fits in that spectrum.

Mike Shaver: There are some pieces of our model that are quite similar to Apache’s. For someone to get commit access to the source tree, or become an owner of a section of code, people need to vouch for the person, and not all those people can be from the same company. We’re not as concerned about it as Apache is though.

I think one of the reasons that’s the case is because a good number of our contributors are employed by the same company, the Mozilla Corporation. But our role in the corporation is very different from that of a traditional software company. We’re not doing anything other than working on this code. It’s not just part of our business to build the Mozilla project; it’s all of what we do. It’s what we’re chartered to do.

So we’re structurally set up as an organization to be aligned with the project, which means we don’t really have the same kind of risk. No one’s thinking IBM’s going to run off with Mozilla if Mozilla doesn’t have the defenses in place, and they’re big contributors.

The Mozilla project is very healthy, obviously, and a lot of the people who do come into our employment relationship come into that relationship because they’ve been successful as Mozilla leaders and developers.

Scott: Talk about that. What is the way that people advance through the ranks? Because, again, that’s something that varies from project to project. You guys have terms like “sheriffs”; there are special security people, and you’ve got other contributors. How do people get their toe in the water with Mozilla and climb all the way up to the top of the food chain?

Mike: It certainly isn’t necessarily a linear progression. There are some of the things you described, like being a sheriff, that’s really more of a role than a rank. People are sheriffs, meaning that they’re responsible for watching out. They’re on integration watch for a given day, watching as things come into a source tree, making sure that our tests all stay in line, helping get the right people on problems that come up in integration.

Most generally, people get involved with Mozilla initially by scratching an itch of theirs. So we’ll see people come into the project, on the technical side certainly, with a patch or investigation leading to a patch for something that they care about. And sometimes, that’s where the level of involvement ends; people can pop in and out, they only have that one itch to scratch, and it’s scratched now, so they can go back to doing whatever it was that bumped them into that problem. But some people do really “get the bug” and stick with the project.

Generally, the path to getting more privileges in the project is not extremely well formalized. Some pieces, such as access control over the source code repository, we obviously take very seriously, and there are a number of nomination and review processes that happen there, and similarly on the security group side. But for the most part, people’s influence comes from the respect they have from other members in the community.

Whether they’re employed full time, or they’re grad students who can spend a couple of hours a month working on something, if they’ve demonstrated that they understand the technology and Mozilla’s values, those are the things that lead to people becoming more influential.

Most of the time when we have somebody move into a role of ownership over a code, which really means that they’re there to settle disputes and break ties about which way a given patch should go or decide whether something’s ready to check in or not, it’s usually a nonevent within the community. It’s something where it’s obvious that the person’s been acting in that role and providing good leadership and guidance in that capacity. It’s more a matter of formalizing it so that the people who aren’t as familiar with the project know whom to go to and what the effect of the person’s decision is.

Sean Campbell: When we were at OSCON, I watched some guys from Google who are very involved with Subversion. They talked about managing the community, and they talked a lot about how people grow through the community, but then they also had an interesting conversation about – for lack of a better phrase – how you demote somebody.

I think whatever process a project uses, it often comes to the test not when people are being promoted, but when someone has to be removed from a position.

So tell me a little bit about that.

Mike: The most common case is that somebody acts as a bad owner, simply because the person is not invested in it anymore. The person is not spending a lot of time on it and has moved on to something else in their life. And in that case, the news is really more about someone new becoming an owner.

In a couple of cases, we’ve had to actively seek out a replacement, and that tends to come from cases where the owner in question is not thinking or acting with enough breadth of interest in mind, where the person’s specific needs for that code are being met, but it’s not integrating either technically or, in some cases, according to project values at a higher level.

Sean: So people lock too much into their view of the feature and can’t really back out of that.

Mike: Yes, exactly. I mean, there’s taste and judgment involved in the role of anyone who’s a committer, but sometimes someone can go too far.

Sean: Right, right.

Mike: You often have to trade off an old capability in order to get a new capability, or to fit into a new model. And we had an example, to speak with some generality, where there was someone who was an employee of a company and was an owner of a piece of code that was not widely used in the project but was used by a number of downstream projects.

And the patches from this given company were getting checked in, but we went to look at some of that stuff some months later, and there were questions about it. Did this stuff get reviewed well enough? Did it get reviewed in light of the way all the other people are using this code, and not just the people who were most active in the project at the time?

We actually did end up replacing an owner there. It wasn’t really acrimonious. It’s not like the person was, you know, grabbing the door frame and screaming on the way out.

Sean: [Laughs]

Mike: To them it’s sort of old news. By the time you’ve gone through a review like that, and it’s obvious there was a real falling down on the part of an owner, something needs to be done. But it was really fine in this case actually, because even in talking with the company in question and saying, “We’re finding a new owner for this code because the stuff that came in really didn’t go through the things it should have. It wasn’t treated with same breadth of analysis that we need in order for our code to be useful to a lot of people.”

And they were OK with that. The reason they were involved in the first place is because they like the quality of the code and the nature of code that comes out of that collaborative process. So rather than it being a source of tension between, if you will, the core of the project and this company that had some contributors in it – they were glad to see that this code that they obviously care about was going to be that much better owned.

And as far as I know, it didn’t have repercussions on the person’s employment or anything. It wasn’t that the person was hired because he or she was the owner, or that kind of thing, which is good.

But in this case, it turned out really well. And it was an opportunity for us to really bring another development community into not only our project, but also our values. We’re not doing this in a punitive way; it’s because we care about this code, and it feels like it’s not getting the love it needs right now. So we found some people who have other history with it and who are willing to take a different view on it.

The new owner gets support from other people who have been around longer and in ownership roles longer. It’s been pretty positive.

Scott: One of the things I’m curious about – because there’s a Mozilla Foundation with employees – What are some of the things that can be done only by employees vs. people in the community?

Mike: In our view, if something has to be done by an employee, generally that’s sort of a bug. We can’t always afford to fix those bugs, in the sense that it’s more disruption for purity than the value you get at the other end. For example, we do partner distributions with different companies. We need to start spinning that stuff up with them before they are ready to announce it, so employees are involved in things like that.

We have some contractual relationships around revenue that for legal and other reasons need to be performed by employees. Someone’s got a legal relationship and can preserve whatever the legally required privacy and secrecy stuff.

But generally, when you look at technical tasks, when you look at things around how the software’s produced, or reviewed, or determined to be released, I struggle to think of anything where employment is a requirement.

In some cases the role is demanding enough that full time commitment is really important. But that doesn’t necessarily mean that they need to be employed by us. An example of this is Firefox. When we first started out, the lead of Firefox at the time, Ben Goodger, switched from being a Mozilla Foundation employee to being a Google employee. And he continued to be the owner of Firefox for some time.

He isn’t anymore; he’s moved on to other things. And Mike Connor stepped into those shoes quite ably, and Mike is an employee. But if someone else was paying him to do a good job on Mozilla, to contribute full-time and keep up with it, that’s the real condition.

Where we have cases that don’t meet that criteria, where you would need to be an employee to do certain work, we’re very tentative about undertaking that kind of work. Scaling through the community, not just technically but in terms of organizational resilience, is really important to us.

One of the luxuries of being a part of a nonprofit is that we’re not held against next quarter’s revenue numbers. We have only one shareholder, and they aren’t in it for the money. So, we don’t have to produce a return there. We’re really about making sure that we are creating software and a software culture that can last 10, 20, 70 years.

Where we have an opportunity, we try to bring people in who may not be able to commit full time to work on this. They may have other things in their lives, or they just may not be that interested. They want to spend some time on this, but not make it their lives. Those are opportunities for us to really diffuse ourselves through a broader part of the software industry.

And those kinds of things give us, as I say, great resiliency. People sometimes switch from that part-time mode and want to work on Mozilla full time. We get some great hires and full time contributors that way. They bring Mozilla’s values into other environments: around transparency, around support for standards and universality of technology.

And they also bring to us their own perspectives on software development. You know, “This is how we do it at Big Co.” Or “This is how we do it in this tiny little agile Web development shop. Here are some things we learned about how to test, how to build tools, how to instrument things.” The conversation around software development, in some ways I think, is even more important to us than the specific software.

Before we had Firefox, we had the Mozilla Suite, which for its many wonderful attributes was not nearly as successful in the marketplace, as I’m sure you’ve seen. But the project that created it is very much the same. And that we were able to weather having basically no market share for five, six years but still built a culture of software development and a culture around that software – of marketing, of user support, of design and tools, and so forth – is what let us continue to be there until we found our Firefox.

And it might be that in five, 10 years, we’ll need to find the next Firefox. So we need to be about the process that leads to it, as much as we are about the artifact.

Sean: One of the things I like to ask people we talk to is, “How do you ensure usability in an open source project?” We get interesting responses, and you guys, in particular, stood out to me, the same way OpenOffice did, because you, even more than OpenOffice, have the “grandmother is downloading in Iowa” problem.

Mike: Yep.

Sean: It just has to work. It can’t get in the way. I’m curious, because when I look at the new features page for you guys, it’s very much written in a “to you, the developer” tone. But at the same time, I think everybody can look at open source projects at times and say, “How usable is this by a nontechnical person?”

There are a bunch of questions here, but I’ll start by asking how you get in a feature that you know users really need, but no developer is clamoring for it, and maybe it’s not really that sexy of a feature to write.

Having written code, we all know how that works. How do you motivate the community to add in that feature? Do you have examples of stuff that you guys have added to Mozilla that maybe the dev community wasn’t clamoring for, but you got 50,000 user requests for?

Mike: So there are a bunch of interesting questions there for sure. I’ll take them backward. One of the best ways to motivate the developer community at large is to go get yourself 130 million users.

Sean: Right. [Laughs]

Mike: Because the things that are not sexy – like, I’ve got to go figure out why this one plug in crashes when I’m running on Panther and I have Quicksilver installed – that’s not necessarily a sexy technical problem. But it can be exciting to fix something that’s going to affect “only” a couple of hundred thousand people.

Sean: Right, you got all these Quicksilver users. And so you can look like a hero because you can say you solved this problem for them.

Mike: It’s true that at some point you have problems that affect a small enough subset of people that it’s not economical to have somebody work on it. All software, propriety, open, whatever, has this class of problem – whether it’s technical or usability related.

But as you broaden the pool of people who are exposed to your product, and as you make it easier for people to move from being users to being followers – I mean followers of the project, not followers in the cultlike sense. Some people contribute by helping test. You want to increase the chances that somebody who cares about that problem is going to be able to connect with someone who can fix it.

And the motivation piece is helped a lot by the fact we have a really direct connection with our users. You’re not seeing a ticket trickle down through tier 1, tier 2, tier 3, support escalations, and then a big “what do we fix in this bucket” release.

Sometimes we do that. We look for high-profile crashes or bad plug in interactions. But often you can read the bug report and the person who’s hitting that problem is right there. It’s a person whom you are helping, and it’s not just a software artifact.

Now, going the other way on usability issues, that can be a trap, right? We need to write software that’s usable not just for one person, but usable for 130 million, or 150 million, or hopefully a quarter of a billion people. And that’s a hard problem.

I think it’s a problem that’s generally hard for software that scales. I think it would be very hard to do that well at our size without being as open as we are. I would venture to say impossible, but I try not to use that word very often.

Here’s a good example. There’s a thread that’s flaming away right now on one of our application development forums. We’re making some changes to bookmarks in Firefox 3. This is pretty much the first time we’ve changed how bookmarks work since about 1994. But we’ve learned a lot about how people use them. The Web’s bigger now, you need to track more on it, and so forth.

Sean: Sure.

Mike: But a lot of people, especially those who are more technically advanced – they’ve been using bookmarks since 1994. They have muscle memory. They have whole parts of their reptilian hindbrain devoted to how bookmarking works today.

Sean: [Laughs]

Mike: So, making changes there – even when we have strong belief and as much evidence as you can have that it’s going to improve the lot of users – I think it’s going to be good for users and be below the threshold of “It’s too much to learn how to use this.” Even when we have that, there’s still a very vocal portion of our community that’s conservative.

Which is good. That conservatism helps us in a lot of core technical ways. It keeps us from getting ahead of our users. We don’t want to have the Apple Newton problem, where we have something great but no one really knows what to do with it yet.

And that’s a hard tension on the usability side, for sure, between how the Web works today and making it work better. And also, helping users develop new habits and patterns for managing information, for understanding the security characteristics of what they are going to do.

Another good example of this is the anti phishing features we have in Firefox 2. With that feature, you can still click through to the site. But because we flagged it, now a different part of my brain’s working, and I’m being a little more suspicious of this site. Or maybe I’m scared enough that I’m going to ask somebody who knows.

In Firefox 3, we have another kind of protection against sites that have been identified as trying to install malware on the user’s system, and you can’t click through that warning. With phishing you could look at the page and still decide, “I’m not going to put any data in this. I want to look to see how the phishing attack works.” In the malware case, by the time you’ve loaded that page, we’ve lost. We can’t undo that action.

But Firefox is all about the importance of choice, and the importance of putting users in control. That makes it easy to throw stones at this new behavior. We have people who say, “I know better, I should be able to click through.”

But ultimately, we’re responsible as the agent of the user. As we’re building this software, we want them to be able to use the Web in a more reliable and trustworthy way. We’re saying, “If the site’s in this small list, no. Just don’t let the user through.” If you want to, you can disable that feature. You can disable malware protection entirely. And if you are technically advanced enough to do that, then we will give you some rope and try not to watch too carefully what you do with it.

But, for the vast majority of users, providing a click through path would make the product less usable, even though it would be more palatable to some people, especially those who know most about security in our community. And these are discussions that, for better or worse, we have in the open. This is where our process happens. You can see how the sausage is made.

Those cases can be somewhat distracting for us. I think in a more closed environment, where you have good designers, as I very much believe we do, they can get together in a room. They can figure out what the problem space looks like. They can design some good solutions and iterate them.

In our case, the bar for those features is that much higher, where it needs to survive feedback from anybody who wants to give it. Which isn’t to say all feedback is necessarily followed. Even we have bad ideas; other people have bad ideas. And we do have to make hard choices about trading off different capabilities.

But you have to be able to have at least a thick enough skin to go through that process and see all that feedback, which tends to be negative, because if you like something you just kind of say, “Eh, I won’t say anything,” unless you are really excited about it. But if there’s anything about it you don’t like, people jump on that stuff. Stuff has to make it through that process.

I think for us, it’s made us more conservative about adding new user facing features, which I think is good in a tool as ubiquitous as the browser. But it also taught us a lot about how to make the features we do add understandable and easily usable by a really broad range of expertise. So I think the product’s much better for that.

OpenOffice is exactly the same sort of case. And your reference to grandmothers downloading in Iowa was the reason that, for us, the whole of the installation experience is really important. We try to keep the download size small. I think we’re just over five megs now, because we know there’s limited time between “I think I’d like to try this Firefox thing” and “Oh, dinner’s ready.” You come back from dinner and never notice that you downloaded it somewhere in your huge downloads folder, and you’re never going to try it.

Scott: Right. Right.

Mike: Keeping that loop short is really important. When you use Firefox for the first time, it pulls in all your bookmarks, it pulls in all your history, all your saved passwords, et cetera. Because you want it to be really easy to start using and be really easy to stop.

You can go back and use IE if you want, because we don’t want you to have to feel like you’re committing your Web to us forever.

Scott: I mean one of the things that you guys obviously have to focus a lot on is reliability. And like a lot of open source projects, you have this modular, pluggable design. And we found that a lot of times that’s really important for projects to be successful. It’s important on Apache that even if people aren’t maybe up to the level they need to be to work on the Apache Web server itself, the barrier to writing a module is pretty low, right?

Mike: Yeah.

Scott: And Firefox, you have the same kind of thing, right? You might not be working on the browser itself, but you might want to try to write an extension to it.

How do you guys architect Firefox to help keep bad extensions from crashing the browser? How do you, for lack of a better term, keep bad code from making your good code look bad?

Mike: Yeah. Not well enough, I think. People using Firefox sometimes load it up with a pile of extensions and then saying, “Performance isn’t what it used to be” or “I get this one weird crash.”

There is an important trade off there, right? We could certainly restrict the attachment surface for plug ins. We could say, “We are going to run you out of process. We’re going to run you in this safer way. We’re going to give you five things you can do to the browser.” And we could become much more robust that way.

Or we could say that you can run arbitrary code. And it’s up to the users to determine whether the given extension is going to be good enough for them or not.

We err, and to some extent I guess it is erring more on the side of permitting the extension developer to do whatever we could do in the product. And there certainly are challenges that come along with that, from helping the users understand that when they install an extension that they are trusting the extension code to do whatever they trust Firefox to do.

But also we try to help developers understand the impact of what they’re doing on Firefox. People write extension often because it’s something that they are going to use, but it’s also something that could be used by any of these 130 million people.

And an individual extension developer may not have access to the same breadth of feedback and breadth of testing that we have across Firefox itself. And we have a quarter of a million people using just our most recent Firefox 3.0 beta right now. Many extensions don’t get that kind of coverage and that kind of detailed analysis by a large group of people.

What we generally do is try to make the things that people want to do with extensions easy and safe. And so in Firefox 3.0 for example, we have some utility libraries we’ve built in, which we call Fuel, which make it easier to work with bookmarks, make it easier to work with tabs and history and so forth, because we were finding that a lot of extension authors were spending a lot of time on that boilerplate. And we want them to spend more time on their cool idea.

But also they were doing it in ways that made it hard for us to make changes later on without breaking their extensions, or they were doing it in a way that interfered with other extensions or interfered with performance.

So we tend more to deal with that class of problem by encouraging better behavior rather than preventing worse. We are certainly interested in looking at models for sandboxing of content, for having extensions that can run all over our privilege mode. They may not have access to as much data or be called in a different way.

But one of the most valuable parts about the extension model for us isn’t just what other people can do with it. It’s that it is a proving ground for our own ideas.

You see stuff coming out of the labs. You see stuff coming out of individual developers or people who are out in the extension development community, if you will, who have a great idea, like session restore, or some of the new changes made to tabbed browsing in Firefox 2.0, and a bunch of stuff in Firefox 3.0 as well. We can look at those extensions. We can say, “Hey, if we want to put this feature in Firefox, we can find out on a pretty broad scale how people are going to react to it by getting 100,000 people to install the extension.”

Sean: Right.

Mike: And that gives us the kind of feedback that you can’t get through even the most passionate online conversation, right? You can put it in front of people and see, in a classical usability test sense, or through instrumentation, or just through surveys afterward, is this a better behavior? What are people blogging about it? What are people using? Are they leaving it installed? And that’s a really powerful channel for us.

Scott: Some applications, and maybe Firefox already does this, will sort of shame the extension developer a little bit. You’ll start up the application and it’ll say, “Hey, last time I was running, this extension crashed me. You’re going to have to manually re enable it if you want it to load.” Is there any kind of architecture like that or is that something that you guys have looked at?

Mike: So we do something similar for session restore, for example. If the browser crashes and you start it up the next time, it says, “You can try this session again, but if it was something you were looking at that crashed you, I can predict what’s going to happen.”

Scott: Right.

Mike: One of the things that is challenging about doing what you are describing for extensions is that the way extensions really work in Firefox is by sort of mutating the application itself. In some cases we could tell that a given piece of code was running. We can look at some kinds of crashes and say, “Ah, well, we don’t normally take that path and this extension did, so that’s likely what’s happening here.”

But in a lot of cases, the way extensions work are by taking parts of the applications that aren’t connected together and connecting them to each other. So they do their damage, but we’re running Firefox code at the time of the crash. One of the things that we have done in Firefox 3.0 is use a new crash-reporting system called Breakpad, which was developed by us and Google for a couple of reasons.

One is, the previous one we were using wasn’t open source. It was the only non–open source code we were shipping and the source of much heartache for that reason. But the value of those crash reports was just way too high. It improved the product so much. But it also didn’t work, for example, on Intel Macs, of which as I am sure you are aware, a couple have been sold.

Sean: Yeah, just a few. [Laughs]

Mike: Just a few. So we have this new crash-reporting software. It also handles threads, which is something that we have been using for a long time, and the crash-reporting software from the ancient Netscape days really didn’t handle threads very well.

We get better crash-reporting data. And one of those things we do report with crashes is what are the extensions that are installed. So we’re able to do correlation that way and then do outreach to authors. And they can then see: this is what these stacks look like. We are able to work with them with my team, specifically in evangelism and developer relations, to say, “Hey, this thing you’re doing is breaking stuff. Let’s get it updated.”

One of the nice things also is that we have spent a lot of time making the extension update model for Firefox pretty smooth. You can push a new update out and it’ll get to users. You don’t have to make sure they all come back to your site and re download and run an installer. It checks for compatibility. It does it in the background.

And so when we do have a fix for something like that out, or more important, when an extension developer has a fix for that kind of thing out, we can get it to users pretty quickly.

Sean: Cool.

Mike: And that’s something we are looking forward to a lot as Firefox 3.0 rolls out, being able to become even more responsive and agile not only with our own product, but with extensions as well.

Scott: Let me ask a question in a different area. There was this huge news story a while back where Coverity had promoted 11 open source projects to their Rung 2. Coverity didn’t make these claims, but the way the news reported it was almost like, “Hey, here is the silver bullet. Coverity’s product scanned the code and found all the defects. You fix them, and you have a secure product.”

Talk a little bit about how tools like Coverity’s factor into how you guys make a secure product. And then what’s the story beyond that? What are the limitations of code scanning tools? What do you have to do beyond that to ensure a secure product?

Mike: Yeah. So certainly we are engineers and we like it when computers can do work that lets humans not do that work. So we are all about tools of various kinds. And certainly security standards are no exception to that.

Our experience with various source code checkers, and I’m not sure what our most recent results with Coverity were, is that maybe they’re not as useful on our code because we are a project that uses a lot of C++ where most of the open source projects are still largely architected in C. Or it could be because we have a very dynamic architecture. You can’t always tell how the pieces are wired together without understanding it or running it.

Generally, the level of false positives is incredibly high. The number of possible vulnerabilities that are reported as “This could be a problem, though it may not actually be one” are vastly smaller. And the number of actual vulnerabilities that are reported are quite small.

And we do act on those where we see them. But I think if you look at it, the Coverity “rungs” are really ratings of how responsive you are to Coverity reports. They’re not really certifying any absolute level of security in the code.

Scott: Right.

Mike: Which is fine. Responsiveness in security is very important. But what they are really saying there is that these groups have responded to all the problems reported to them.

Well it turns out that’s the way they call it. And that’s certainly something we could invest in. We could go in and make sure that we tweak our codes so they don’t trigger any false positives and so forth. But in our experience, that doesn’t correlate with improved security of products.

Which isn’t to say people shouldn’t be using these tools. I think to say, “You know that there were false positives from Mozilla, so we’re not going to bother looking at it” is pretty reckless. Each project needs to look at the tools and figure out which ones are going to pay off best in terms of improvement to security of products.

We do our own. We released last year at Black Hat a set of tools that we built for fuzzing various network protocols and JavaScript language, which we had given to other browser vendors as well, ahead of that release. We found that those helped find a much larger set of bugs and potential problems in our code than the traditional sort of static code scanners.

Where we do use tools similar to Coverity, they are actually ones that we are building in-house and when working with researchers on projects like Elsa. This will let us do static analysis and transformation of our code. And we have a huge code base, and what we really want to do is be able to update the patterns in it as we learn about new vulnerabilities or performance patterns or portability or take advantage of new language features.

When we started to build Mozilla in 1997, C++ compilers were not nearly as capable as they are now. So there were a lot of features we just couldn’t use if we wanted to run on everything from Win 32 to AIX to OS/2 and so forth.

Scott: Right.

Mike: So we can take advantage of, for example, C++ exceptions in a way we couldn’t before. We are writing tools to help analyze our code and also to write big chunks of the code.

Sean: Interesting.

Mike: So that’s where we are putting our investment in terms of the tools. But I think those that are trying to make a secure product and aren’t doing everything they can to help use the computer to find issues, not as a substitute for human analysis but as an adjunct to it, are doing themselves and their users a disservice.

Scott: One of the things that we do a lot of the times at the end of these interviews is hand the microphone to the person we are interviewing and say, “We covered a lot of stuff that was interesting to us, but what’s maybe interesting to you that we didn’t specifically ask about?”

Mike: Yeah, I think the whole series you’re doing here about the software processes that are different – well, we’ll see whether they are different or more similar – between open source and proprietary model, I think is really interesting.

And I think it’s going to be interesting to see how those differences in process manifest in terms of users affecting things, whether it’s agility of features or improved responsiveness on bugs, or security issues, or days of vulnerability. Whatever we decide we want to optimize for in a given path.

But one of the things that we are doing with Firefox – and we hope with the Web – is making it a more robust and effective way to build an application. So it’s interesting for me to reflect on the questions you’ve asked and look through the various blog posts and so forth.

The kinds of processes that we want to encourage there, the kinds of things that we want to make easier for developers or make their results more effective, is how can we bake those things into the Web?

And we are certainly very active in various standards communities. We are very active in working with language developers and Web application developers and other browser developers to be sure to have these capabilities. But doing them in a way that doesn’t require the forklift upgrade of the development process.

Like, you don’t have to throw out this app if it has everything you need except video and rewrite it in something that has video. We want to take the barrier points away. But I think we will do well to keep in mind the development processes that lead to those apps as we do that, and make sure that what we are doing builds well for people who are using traditional proprietary methods, whether they are doing more collaborative development or whether they are trying to find a hybrid, as MySQL did.

And it looks like it’s worked out well for them, based on the acquisition announcement.

Scott: [Laughs] That’s right. Mike, thanks for taking the time to chat.

Mike: Thank you.

Bookmark this: