Interviewee: Brad Wilson

In this interview we talk with Brad Wilson – Software Developer in Microsoft’s OfficeLabs team. In specific, we talk about:

• CodePlex, the Microsoft open source repository
• Community participation other than coding
• Open-source governance at Microsoft
• Open-source inroads and future at Microsoft

Sean Campbell: Brad, tell us a little bit about your background.

Brad Wilson: Sure. I’m a professional software developer, and I have been for about 15 years. I came to Microsoft just about three years ago. My first stop at Microsoft was the Patterns and Practices Group. I worked a little bit on the Enterprise Library, but mostly ObjectBuilder, Composite UI Application Block, and the Smart Client Software Factory.

After I left Patterns and Practices, I went to go work on CodePlex, which is the repository for open source projects that Microsoft runs.

My history with open source is that, for most of my career, I’ve worked in small companies–Microsoft being an obvious exception–all in the Windows space. I really started getting into open source around 2000.

That timing also coincided with the time I started picking up agile practices, because I got introduced to a lot of that stuff all around the same time. NUnit, the unit testing framework for .NET, was my first exposure to a tool that I really ended up using and liking a lot that I also got the source code for, which was kind of strange, but I found it pretty intriguing. Now, working for Microsoft, it is kind of strange to think that I actually contribute to several open source projects, but I do.

Jim Newkirk and I–he was the guy who worked on NUnit–came up with a new unit testing framework called xUnit.net, so I contribute to that pretty regularly. The ObjectBuilder project that I started on in patterns and practices–we kind of spun that off–was embedded inside of the Composite UI Application Block. We spun it off as its own project on CodePlex, and I’m still an active contributor there as well. While I was at CodePlex, I also worked on a source control client for Team Foundation Server.

So, quite a few open source projects, even though I’m a Microsoft employee, which most people think isn’t really possible. But I think CodePlex has proven that Microsoft is very interested not only in consuming open source, at times, but also in producing it.

Scott Swigart: You mentioned NUnit, xUnit, and the ObjectBuilder. Those are kind of “by developer, for developer” open source projects. Are those typical of the open source projects that you’ve contributed to and the kind of open source software that you typically find yourself using?

Brad: That’s true–the open source projects I contribute to tend to be the “by developer, for developer” kind, although I also consume things that are less about development. Like one tool that I’ve used forever is the GIMP graphics editor, which I love a lot. But I’ve never contributed to it because, mostly, I’m an amateur graphics maker, so I’m not so interested in contributing to the source base.

There are probably other things that I use that are open source, and I might not be aware of it. I think that’s probably true for a lot of people, or at least we don’t think about them in terms of being open source projects. We just think about them in terms of something that we use, like web developers still use all kinds of different browsers. Firefox is really popular, and most people don’t think of it as open source–they just think of it as a browser. But, obviously, it’s a huge open source effort.

Scott: Do you spend time actually working on CodePlex?

Brad: I was a software developer on the CodePlex team, so I did work on the web site itself. Actually, a lot of the time I was at CodePlex, I was working on the source control client, which we released as open source. I spent probably half of my time on the CodePlex team working on that project full time.

Scott: Educate me about that a little bit. That was a command line utility that would let people do check-ins and check-outs from CodePlex, to give them sort of a Subversion-like experience, or a CVS-like experience?

Brad: Right. We chose Team Foundation Server for our source control. Personally, especially before I came to Microsoft, I had been a big fan of CVS and Subversion, and I like those kinds of tools. When we chose Team Foundation Server for source control, the source control server itself was fine, but the Team Explorer client that’s built into Visual Studio didn’t really work the way I was comfortable working.

The biggest thing that our client gives that the original version of Team Explorer didn’t do very well is the ability to work offline. Most people who are using Team Foundation Server are going to be using it on the LAN at a corporation, where they’re going to have connectivity pretty much all the time.

But in open source projects, you’re very often doing work in places where you have slow or no connectivity. Having to constantly hit a server that might be behind a firewall or whatever isn’t really conducive to that kind of work. We really wanted to provide that offline, edit-merge-commit style of working that people who are familiar with some of the other open source sites would be more comfortable with.

Scott: Has a community formed around some of these efforts that you’ve worked on? You mentioned NUnit, which obviously has its own community. Are there communities around xUnit, Subversion, the CodePlex client, and things like that, where people post suggestions and maybe even post code itself?

Brad: There is some community; the unit testing framework is probably the biggest of the three. None of them are really huge, but certainly, people are reporting bugs, discussing features, asking questions and occasionally providing code.

Scott: One of the bigger source code repositories before CodePlex was SourceForge. How much did you guys look at that as inspiration in working on CodePlex? And where do you think CodePlex goes beyond something like SourceForge, in terms of the experience that it provides for the users?

Brad: There are clearly a lot of similarities between CodePlex and other open source repositories like SourceForge, CollabNet, and things like that. I think the goal for each is different. I can’t necessarily speak to what the goals are for SourceForge and CollabNet. It was really Jim and Sandy’s idea–the other guy who helped start CodePlex–to provide an environment that was more friendly to Windows developers, especially those who didn’t have experience with the existing sites, like SourceForge.

Some of those people really prefer that sort of Team Explorer experience, so we wanted to provide that kind of environment for them. Another goal for CodePlex is to have really high quality projects, so we built some features into the software that discourage “publish it and forget it” types of projects. And we do try to look around the site and understand what people are using and what they aren’t using.

One thing that we really focus on is taking community feedback–letting people know that we really want them to report bugs against our site and talk about the features of our site. And we really do actively go out and look at those things and implement them. I think the users are pretty happy with the turnarounds and our ability to listen to their feedback.

As to where we might go in the future, there are a lot of possibilities. CodePlex as it sits today is a very interesting site, but you can certainly imagine that there are lots of things that could be done, as well as features that other sites aren’t offering.

Scott: What sorts of features would you like the most?

Brad: I think the biggest opportunity that CodePlex has in general is to provide a stickier community than what you get at SourceForge. You want to be able to relate to the people working on your project like an extended team, as opposed to a bunch of disconnected individuals. Personally, I would like to see some tighter integration of community features–things like presence information and chatting, and maybe even integrate some of that stuff into Visual Studio. I think that would be really cool.

Scott: Let me drill into something there. A lot of people spend a lot of time writing and thinking about how you build a vibrant community around your project, and how you manage personalities and a lot of that stuff.

But people don’t really talk much about how important they feel like the site–whether it’s CodePlex or one of these other sites–and the functionality it provides is in building and holding together a community.

You’ve worked on NUnit and things like that, which were hosted on SourceForge, and you have a lot of experience with CodePlex. How much does it lower the barrier to entry and help keep a community cohesive, if the tool the people are using has these great collaborative features?

Brad: That’s a good point. SourceForge’s UI seems to be driven very strongly around the idea of the downloader. I feel like when I was using SourceForge that the learning curve for any other activity besides downloading tended to be high.

I definitely think that some sites can certainly do things that make it easier for people to be able to contribute, especially to contribute in small ways. A lot of times, these sites are about the big contributions. I want to write a bug. I want to start a big discussion thread.

But there are also a lot of opportunities for small contributions. We added voting on work items and found it to be really popular with all the project owners because it was a really low friction way for people to say they’re interested in something. So, the design of the site, the design of the tools and any integration you can give can certainly have a huge impact, in the long-term, on the kinds of communities you can build for your projects.

Scott: There are some projects that are really big, and there are things like the Linux kernel or Apache web server, and they’ve been around for a very long time. They’ve got hundreds or thousands of people working on them. And on one end, they tend to be really transparent. There’s a saying in the Apache community that “if it didn’t happen in the mailing list, it didn’t happen.”

On the other hand, unless I have the ability to write code, I don’t really have a lot of ability to get a feature into that project. If there’s something I want, I could go to the mailing list and say, “Hi, I have this great idea.” And the response I’ll probably get is “Yeah that sounds cool. When will the code be ready?” On the other hand, with smaller projects, there tends to be something about the culture of the project where people are more willing to just take feedback that comes in. And the small group that’s working on the project–or even the individual that’s working on the project–takes that feedback and works on implementing it.

Have you bumped into those same sorts of experiences in your working with open source? And in the projects you’ve worked on, how have you seen it work where somebody might have an idea and either on one side they’re expected to be able to code it if they really have the idea, or maybe on another project they’re not? It’s fine to just propose a suggestion and kind of hope that somebody else picks it up and works on it.

Brad: There’s no question that there’s definitely a coding bias to participation, especially in large projects. But it can be really intimidating, and the whole mailing list thing is a great example. There are a lot of people who are not comfortable communicating in email with people, especially if you get into a community where you feel like you’re going to be shutdown or made fun of for asking easy questions, or suggesting what some people think of as silly features.

There are definitely a lot of opportunities in the sites to be able to provide these alternative means of contributing. For example, voting or tagging models can help. Someone can easily chime in on a feature that someone has requested and say it’s important, and before you know it, 10, 15, 20 people have said it’s important, and it’s the most important thing on the list. All of this is done in a way that’s non-confrontational, whereas email can sometimes feel a little confrontational.

Scott: You mentioned that sometimes people look at you strange when you say you work for Microsoft, but you contribute to open source projects.

We’ve talked to people in Patterns and Practices who work on the Enterprise Library, and they’re in this one realm that’s a little different than Apache or something like that, where it’s under an open source license, and people can take what Patterns and Practices produces, and they can make their own fork of it and they can do their own work on it.

What they can’t really do is contribute code back to the Enterprise Library. Microsoft has corporate customers who depend on that, and Microsoft makes certain assurances about that. The company has things like the Security Development Lifecycle and certain processes that they put code through, and they have to know the chain of ownership and know, for example, that some code wasn’t copied and pasted from something else that somebody else owns.

But then, I’m guessing that there’re other projects–maybe some of the ones that you’ve worked on on CodePlex–that would be open to taking community contributions. Can you talk about that balance at all? Is that something you’ve run into in any of the stuff you’ve worked on?

Brad: There’s definitely a spectrum. In terms of xUnit.net, which is the unit testing framework, Jim and I set up the project so that the unit testing framework itself is sort of owned by Microsoft. We don’t really take contributions on that, but that part is actually really small. We have a second project, which is related to it, that allows people to extend the unit testing framework. Almost all the work we do is actually in that second project. We do take community contributions for it, and that’s worked out pretty well.

Microsoft does own what you contribute, in a sense. Really, in these open source projects, you have to be very careful–this has nothing to do with Microsoft, but you have to really careful about things like copyright assignments and IP assignments and things like that. All the legal stuff can actually get small projects who aren’t paying attention to it in a little bit of trouble.

Microsoft has a system called an Assignment Agreement, and if you want to contribute code to this project, you say, “I agree to assign the ownership of this stuff to Microsoft.” Of course, it’s immediately pushed back out, in our case, in the Microsoft Public License, which basically says you can do anything you want with it. You’re not really losing the ability to use that code, but Microsoft is sort of taking stewardship of it.

Scott: Microsoft owns it, but you could fork it and do whatever you want with it, essentially.

Brad: Right, exactly. It’s like the Enterprise Library scenario, except we can take contributions, and now other people can contribute to the code base. It’s just that someone has to maintain control over all the legal issues, to make sure that there aren’t any cases of one person claiming to own these seven lines of code, and another claims to own these other nine lines of code. In cases like that, you can’t really untangle it all, so it’s pretty important to address all of that.

There’s another really big project in CodePlex that is very interesting. The Developer Division has an AJAX library called ASP.NET AJAX, and they also have a project called the ASP.NET AJAX Control Toolkit that really takes a lot of contributions from outside of Microsoft.

And again, it’s the sort of a model where we built the core, and we contribute some of the stuff that sits around outside of it, and the community contributes some. And it’s a great, single place you can come and get a whole bunch of great utilities to wrap around the AJAX library.

I think this is going to be a pretty common pattern for teams that are interested in having community feedback, which is to set up these projects around something that isn’t open source–or that is open source but doesn’t take outside contributions. They’ll make it so the community can contribute, and yet everybody who wants to have access to it can come and get it all in one space.

Scott: What kind of decision criteria determine what goes in the small core? People at Microsoft might be writing stuff that goes into the larger section that can take community contributions, but then the same people at Microsoft might also be writing stuff that goes into the somewhat proprietary core. How do you decide what needs to go into one versus the other?

Brad: I can’t really speak to the way the AJAX stuff went, because I haven’t worked on that team, but Jim and I faced the same question when we were dealing with xUnit.net. Our philosophy was that we wanted to have as little there as possible, because we wanted to just set the framework and make it extremely extensible.

Some of it was really a test of all the extensibility points that we put into our framework. How much can we actually push off into this other project and let people look at it and fix bugs in it, and basically get as much of the code as possible out to where it can be worked on by the community. Obviously, that would be a huge bottleneck if the whole thing was just me and Jim.

So, for us, we looked to sort of put the right amount, and the smallest amount we could get away with, into that core project, and push as much out for community inspection and contribution as we could.

Scott: Is the source code available for the core project, or is stuff in the core project because that’s the part where it is kind of proprietary and people can’t see the source?

Brad: In the case of xUnit.net, the source is available. Both projects are actually on CodePlex. It’s just that in the core project, we don’t take contributions in the form of source code. I want to make it clear that there are other ways to contribute, but we don’t take source code. We do provide the source code for everything, because Jim and I are long-time open source advocates. Of course, he did NUnit and put the source code out there for NUnit. He is a big believer in sharing source code.

There seems to be, especially in the “by developers, for developers” scenario, a whole set of stuff that belongs in the canon. It belongs to everybody, because how many implementations of a doubly linked list does the world need, right?

There are definitely things that should just be there for you to use, and Jim and I feel that unit testing frameworks are one of those things.

Sean: I’ve got a broader question for you, as well. What do you think Microsoft has learned from the open source community, and what do you think they still have left to learn, if you could just pick a couple things and say, “I don’t think we’ve fully learned how to do this one particular thing, but it would be to our benefit if we do?”

Brad: Most people probably don’t realize that Microsoft has been a consumer of open source for a long time. If you go all the way back to the first release of Windows NT, the TCP/IP stack that was in Windows NT 3.1, which was released in 1993, actually came from BSD.

Most people think, “Oh, Microsoft hates open source,” but it’s not actually true. There’s quite a bit of reusing of code, I think, from time to time, like the TCP/IP stack. There’s also the use of applications that are open source, so it’s hard to paint Microsoft with one big brush, even though a lot of people would prefer to do it that way. There are certainly groups that are more open to it and groups that are more closed to it.

I think Microsoft has felt recently–right or wrong–that the open source community was out to get them. And some of that is the extreme end of the Slashdot kind of people who say that every misstep by Microsoft is an example of the evil tyranny of giant software companies. But there’s a lot of common ground, both within and outside of Microsoft, about how beneficial open source can be, which is not necessarily to say that we should open source Windows or Office, because those are clearly good money-makers for us.

I also feel like Microsoft is a bit of a target, just because of who they are, and that any use of open source needs to be looked at very critically, to make sure that we’re not exposing ourselves to a risk that a smaller company wouldn’t necessarily have.

Sean: Coming from a background where you appreciate the benefits of open source and you have some understanding of it, now you’re in a closed source company that’s also making inroads toward open source in various ways. What do you think about this kind of move to a model where people take an open source product in-house, and then they essentially build services on top of it or they service significant product efforts on top of it?

Examples come to mind, like Google. They’re considered an open source company, and they’ve done a lot, but the fact is that they get a lot of props for being an open source company, but it’s not like they’ve open-sourced their page rank algorithm. There are also a lot of software as a service ventures that are building that way. SugarCRM has gotten beaten around a lot for the way they kind of use it as a bumper sticker on what they do, that they’re open source and things like that.

What do you think about approaches like that? Do you think that’s a natural evolution for some open source efforts, or do you think that’s just a phase? How do you think that’s going to play out?

Brad: Just like Microsoft is not monolithic, the open source community itself isn’t really monolithic either. I come at open source from sort of a permissive angle–I favor licenses like BSD and MIT and the Microsoft Public License. There are certainly a fairly significant number of people who come at open source from sort of a viral angle, the GPL-style, LGPL-style licenses. In some sense, there’s a conflict brewing.

Google’s a very interesting example, because they really do leverage a lot of open source without necessarily contributing back. And to me, as a permissive kind of a guy, I think, that’s just fine. Some of the people who are on the sort of GPL-LGPL side of the house are probably less excited about that. They would like some of that source code to come back out. And I think that there’s opportunity all along the spectrum.

It’s hard to say that one thing is necessarily going to prevail over the other. I do think, though, for sure, that companies are going to rely more and more on open source, especially small companies, because bootstrapping yourself on the back of available source code and available services is a way to get a company going without spending a lot of money on it. Reinventing the wheel really is an extremely expensive and time-consuming process.

Scott: In the case of proprietary companies, somebody does something, it changes the marketplace, and then everybody else in the marketplace reacts to it, so open source is a disruptive force. And so are other things, like virtualization is disruptive to IT.

I think Microsoft is an example of a proprietary company that responded to exactly what you talked about, with their ISV Empower program, where startups can get Microsoft software for dirt cheap. And to me, just from the outside, it seems like a smart move, because the five-person company may become a 250-person company, and I’m sure Microsoft wants those companies to start out using their software.

At the same time, I see a lot of companies taking a blended approach. Adobe is open sourcing certain things. Microsoft is doing some interesting stuff, like you said, around the AJAX Toolkit and around the Enterprise Library.

What do you think about the future? Do you see Microsoft continuing to go down the road of a blended approach of having proprietary, closed-source stuff, but potentially an increasing amount of stuff where the source code is available?

Brad: I definitely think that Microsoft, right now, is in what I would call a growth period in terms of sharing source code. And not only sharing it, but sharing it in a way that makes it really reusable. I think we got beat up a little bit with the whole Shared Source Initiative, about the idea that you can look at the source, but you can’t really do anything with it. That’s a first level of utility, though, and in some cases, like some of the libraries that came with Visual Studio, there’s no question that it was helpful.

As time goes on, Microsoft is clearly becoming more open. I think having whole groups–like Patterns and Practices and the CodePlex team and the team that’s responsible for the open source licenses here–is a good sign. It’s an especially good sign to see some of the really, historically, closed-source, proprietary teams open themselves up. Knowing that the Developer Division is not only willing but eager to take people’s contributions on the AJAX Control Toolkit is a great sign. It bodes well for the future here.

Scott: Brad, thanks for taking the time to chat with us.

Brad: Thank you.

Bookmark this:

Interviewee: Mike Shaver

In this interview with talk with Mike Shaver, Chief Evangelist for Mozilla. In specific, we talk about:

• Governance of the project and how people get involved
• Changing contributors who fill key roles
• Roles of independent contributors versus employees
• Ensuring high usability in an open-source project
• Protecting reliability in a world of many extensions
• Tools used by the Mozilla team
• Reflections on open and closed development models

Scott Swigart: There are a lot of different governance models for open source projects. With Sun projects, for example, they often have it written that control of the project has to rest with Sun employees. Apache has a different model. Apache projects have to have maintainers who are from at least three different companies so that no one company can hijack a project.

Talk a little bit about how the Firefox kind of governance fits in that spectrum.

Mike Shaver: There are some pieces of our model that are quite similar to Apache’s. For someone to get commit access to the source tree, or become an owner of a section of code, people need to vouch for the person, and not all those people can be from the same company. We’re not as concerned about it as Apache is though.

I think one of the reasons that’s the case is because a good number of our contributors are employed by the same company, the Mozilla Corporation. But our role in the corporation is very different from that of a traditional software company. We’re not doing anything other than working on this code. It’s not just part of our business to build the Mozilla project; it’s all of what we do. It’s what we’re chartered to do.

So we’re structurally set up as an organization to be aligned with the project, which means we don’t really have the same kind of risk. No one’s thinking IBM’s going to run off with Mozilla if Mozilla doesn’t have the defenses in place, and they’re big contributors.

The Mozilla project is very healthy, obviously, and a lot of the people who do come into our employment relationship come into that relationship because they’ve been successful as Mozilla leaders and developers.

Scott: Talk about that. What is the way that people advance through the ranks? Because, again, that’s something that varies from project to project. You guys have terms like “sheriffs”; there are special security people, and you’ve got other contributors. How do people get their toe in the water with Mozilla and climb all the way up to the top of the food chain?

Mike: It certainly isn’t necessarily a linear progression. There are some of the things you described, like being a sheriff, that’s really more of a role than a rank. People are sheriffs, meaning that they’re responsible for watching out. They’re on integration watch for a given day, watching as things come into a source tree, making sure that our tests all stay in line, helping get the right people on problems that come up in integration.

Most generally, people get involved with Mozilla initially by scratching an itch of theirs. So we’ll see people come into the project, on the technical side certainly, with a patch or investigation leading to a patch for something that they care about. And sometimes, that’s where the level of involvement ends; people can pop in and out, they only have that one itch to scratch, and it’s scratched now, so they can go back to doing whatever it was that bumped them into that problem. But some people do really “get the bug” and stick with the project.

Generally, the path to getting more privileges in the project is not extremely well formalized. Some pieces, such as access control over the source code repository, we obviously take very seriously, and there are a number of nomination and review processes that happen there, and similarly on the security group side. But for the most part, people’s influence comes from the respect they have from other members in the community.

Whether they’re employed full time, or they’re grad students who can spend a couple of hours a month working on something, if they’ve demonstrated that they understand the technology and Mozilla’s values, those are the things that lead to people becoming more influential.

Most of the time when we have somebody move into a role of ownership over a code, which really means that they’re there to settle disputes and break ties about which way a given patch should go or decide whether something’s ready to check in or not, it’s usually a nonevent within the community. It’s something where it’s obvious that the person’s been acting in that role and providing good leadership and guidance in that capacity. It’s more a matter of formalizing it so that the people who aren’t as familiar with the project know whom to go to and what the effect of the person’s decision is.

Sean Campbell: When we were at OSCON, I watched some guys from Google who are very involved with Subversion. They talked about managing the community, and they talked a lot about how people grow through the community, but then they also had an interesting conversation about – for lack of a better phrase – how you demote somebody.

I think whatever process a project uses, it often comes to the test not when people are being promoted, but when someone has to be removed from a position.

So tell me a little bit about that.

Mike: The most common case is that somebody acts as a bad owner, simply because the person is not invested in it anymore. The person is not spending a lot of time on it and has moved on to something else in their life. And in that case, the news is really more about someone new becoming an owner.

In a couple of cases, we’ve had to actively seek out a replacement, and that tends to come from cases where the owner in question is not thinking or acting with enough breadth of interest in mind, where the person’s specific needs for that code are being met, but it’s not integrating either technically or, in some cases, according to project values at a higher level.

Sean: So people lock too much into their view of the feature and can’t really back out of that.

Mike: Yes, exactly. I mean, there’s taste and judgment involved in the role of anyone who’s a committer, but sometimes someone can go too far.

Sean: Right, right.

Mike: You often have to trade off an old capability in order to get a new capability, or to fit into a new model. And we had an example, to speak with some generality, where there was someone who was an employee of a company and was an owner of a piece of code that was not widely used in the project but was used by a number of downstream projects.

And the patches from this given company were getting checked in, but we went to look at some of that stuff some months later, and there were questions about it. Did this stuff get reviewed well enough? Did it get reviewed in light of the way all the other people are using this code, and not just the people who were most active in the project at the time?

We actually did end up replacing an owner there. It wasn’t really acrimonious. It’s not like the person was, you know, grabbing the door frame and screaming on the way out.

Sean: [Laughs]

Mike: To them it’s sort of old news. By the time you’ve gone through a review like that, and it’s obvious there was a real falling down on the part of an owner, something needs to be done. But it was really fine in this case actually, because even in talking with the company in question and saying, “We’re finding a new owner for this code because the stuff that came in really didn’t go through the things it should have. It wasn’t treated with same breadth of analysis that we need in order for our code to be useful to a lot of people.”

And they were OK with that. The reason they were involved in the first place is because they like the quality of the code and the nature of code that comes out of that collaborative process. So rather than it being a source of tension between, if you will, the core of the project and this company that had some contributors in it – they were glad to see that this code that they obviously care about was going to be that much better owned.

And as far as I know, it didn’t have repercussions on the person’s employment or anything. It wasn’t that the person was hired because he or she was the owner, or that kind of thing, which is good.

But in this case, it turned out really well. And it was an opportunity for us to really bring another development community into not only our project, but also our values. We’re not doing this in a punitive way; it’s because we care about this code, and it feels like it’s not getting the love it needs right now. So we found some people who have other history with it and who are willing to take a different view on it.

The new owner gets support from other people who have been around longer and in ownership roles longer. It’s been pretty positive.

Scott: One of the things I’m curious about – because there’s a Mozilla Foundation with employees – What are some of the things that can be done only by employees vs. people in the community?

Mike: In our view, if something has to be done by an employee, generally that’s sort of a bug. We can’t always afford to fix those bugs, in the sense that it’s more disruption for purity than the value you get at the other end. For example, we do partner distributions with different companies. We need to start spinning that stuff up with them before they are ready to announce it, so employees are involved in things like that.

We have some contractual relationships around revenue that for legal and other reasons need to be performed by employees. Someone’s got a legal relationship and can preserve whatever the legally required privacy and secrecy stuff.

But generally, when you look at technical tasks, when you look at things around how the software’s produced, or reviewed, or determined to be released, I struggle to think of anything where employment is a requirement.

In some cases the role is demanding enough that full time commitment is really important. But that doesn’t necessarily mean that they need to be employed by us. An example of this is Firefox. When we first started out, the lead of Firefox at the time, Ben Goodger, switched from being a Mozilla Foundation employee to being a Google employee. And he continued to be the owner of Firefox for some time.

He isn’t anymore; he’s moved on to other things. And Mike Connor stepped into those shoes quite ably, and Mike is an employee. But if someone else was paying him to do a good job on Mozilla, to contribute full-time and keep up with it, that’s the real condition.

Where we have cases that don’t meet that criteria, where you would need to be an employee to do certain work, we’re very tentative about undertaking that kind of work. Scaling through the community, not just technically but in terms of organizational resilience, is really important to us.

One of the luxuries of being a part of a nonprofit is that we’re not held against next quarter’s revenue numbers. We have only one shareholder, and they aren’t in it for the money. So, we don’t have to produce a return there. We’re really about making sure that we are creating software and a software culture that can last 10, 20, 70 years.

Where we have an opportunity, we try to bring people in who may not be able to commit full time to work on this. They may have other things in their lives, or they just may not be that interested. They want to spend some time on this, but not make it their lives. Those are opportunities for us to really diffuse ourselves through a broader part of the software industry.

And those kinds of things give us, as I say, great resiliency. People sometimes switch from that part-time mode and want to work on Mozilla full time. We get some great hires and full time contributors that way. They bring Mozilla’s values into other environments: around transparency, around support for standards and universality of technology.

And they also bring to us their own perspectives on software development. You know, “This is how we do it at Big Co.” Or “This is how we do it in this tiny little agile Web development shop. Here are some things we learned about how to test, how to build tools, how to instrument things.” The conversation around software development, in some ways I think, is even more important to us than the specific software.

Before we had Firefox, we had the Mozilla Suite, which for its many wonderful attributes was not nearly as successful in the marketplace, as I’m sure you’ve seen. But the project that created it is very much the same. And that we were able to weather having basically no market share for five, six years but still built a culture of software development and a culture around that software – of marketing, of user support, of design and tools, and so forth – is what let us continue to be there until we found our Firefox.

And it might be that in five, 10 years, we’ll need to find the next Firefox. So we need to be about the process that leads to it, as much as we are about the artifact.

Sean: One of the things I like to ask people we talk to is, “How do you ensure usability in an open source project?” We get interesting responses, and you guys, in particular, stood out to me, the same way OpenOffice did, because you, even more than OpenOffice, have the “grandmother is downloading in Iowa” problem.

Mike: Yep.

Sean: It just has to work. It can’t get in the way. I’m curious, because when I look at the new features page for you guys, it’s very much written in a “to you, the developer” tone. But at the same time, I think everybody can look at open source projects at times and say, “How usable is this by a nontechnical person?”

There are a bunch of questions here, but I’ll start by asking how you get in a feature that you know users really need, but no developer is clamoring for it, and maybe it’s not really that sexy of a feature to write.

Having written code, we all know how that works. How do you motivate the community to add in that feature? Do you have examples of stuff that you guys have added to Mozilla that maybe the dev community wasn’t clamoring for, but you got 50,000 user requests for?

Mike: So there are a bunch of interesting questions there for sure. I’ll take them backward. One of the best ways to motivate the developer community at large is to go get yourself 130 million users.

Sean: Right. [Laughs]

Mike: Because the things that are not sexy – like, I’ve got to go figure out why this one plug in crashes when I’m running on Panther and I have Quicksilver installed – that’s not necessarily a sexy technical problem. But it can be exciting to fix something that’s going to affect “only” a couple of hundred thousand people.

Sean: Right, you got all these Quicksilver users. And so you can look like a hero because you can say you solved this problem for them.

Mike: It’s true that at some point you have problems that affect a small enough subset of people that it’s not economical to have somebody work on it. All software, propriety, open, whatever, has this class of problem – whether it’s technical or usability related.

But as you broaden the pool of people who are exposed to your product, and as you make it easier for people to move from being users to being followers – I mean followers of the project, not followers in the cultlike sense. Some people contribute by helping test. You want to increase the chances that somebody who cares about that problem is going to be able to connect with someone who can fix it.

And the motivation piece is helped a lot by the fact we have a really direct connection with our users. You’re not seeing a ticket trickle down through tier 1, tier 2, tier 3, support escalations, and then a big “what do we fix in this bucket” release.

Sometimes we do that. We look for high-profile crashes or bad plug in interactions. But often you can read the bug report and the person who’s hitting that problem is right there. It’s a person whom you are helping, and it’s not just a software artifact.

Now, going the other way on usability issues, that can be a trap, right? We need to write software that’s usable not just for one person, but usable for 130 million, or 150 million, or hopefully a quarter of a billion people. And that’s a hard problem.

I think it’s a problem that’s generally hard for software that scales. I think it would be very hard to do that well at our size without being as open as we are. I would venture to say impossible, but I try not to use that word very often.

Here’s a good example. There’s a thread that’s flaming away right now on one of our application development forums. We’re making some changes to bookmarks in Firefox 3. This is pretty much the first time we’ve changed how bookmarks work since about 1994. But we’ve learned a lot about how people use them. The Web’s bigger now, you need to track more on it, and so forth.

Sean: Sure.

Mike: But a lot of people, especially those who are more technically advanced – they’ve been using bookmarks since 1994. They have muscle memory. They have whole parts of their reptilian hindbrain devoted to how bookmarking works today.

Sean: [Laughs]

Mike: So, making changes there – even when we have strong belief and as much evidence as you can have that it’s going to improve the lot of users – I think it’s going to be good for users and be below the threshold of “It’s too much to learn how to use this.” Even when we have that, there’s still a very vocal portion of our community that’s conservative.

Which is good. That conservatism helps us in a lot of core technical ways. It keeps us from getting ahead of our users. We don’t want to have the Apple Newton problem, where we have something great but no one really knows what to do with it yet.

And that’s a hard tension on the usability side, for sure, between how the Web works today and making it work better. And also, helping users develop new habits and patterns for managing information, for understanding the security characteristics of what they are going to do.

Another good example of this is the anti phishing features we have in Firefox 2. With that feature, you can still click through to the site. But because we flagged it, now a different part of my brain’s working, and I’m being a little more suspicious of this site. Or maybe I’m scared enough that I’m going to ask somebody who knows.

In Firefox 3, we have another kind of protection against sites that have been identified as trying to install malware on the user’s system, and you can’t click through that warning. With phishing you could look at the page and still decide, “I’m not going to put any data in this. I want to look to see how the phishing attack works.” In the malware case, by the time you’ve loaded that page, we’ve lost. We can’t undo that action.

But Firefox is all about the importance of choice, and the importance of putting users in control. That makes it easy to throw stones at this new behavior. We have people who say, “I know better, I should be able to click through.”

But ultimately, we’re responsible as the agent of the user. As we’re building this software, we want them to be able to use the Web in a more reliable and trustworthy way. We’re saying, “If the site’s in this small list, no. Just don’t let the user through.” If you want to, you can disable that feature. You can disable malware protection entirely. And if you are technically advanced enough to do that, then we will give you some rope and try not to watch too carefully what you do with it.

But, for the vast majority of users, providing a click through path would make the product less usable, even though it would be more palatable to some people, especially those who know most about security in our community. And these are discussions that, for better or worse, we have in the open. This is where our process happens. You can see how the sausage is made.

Those cases can be somewhat distracting for us. I think in a more closed environment, where you have good designers, as I very much believe we do, they can get together in a room. They can figure out what the problem space looks like. They can design some good solutions and iterate them.

In our case, the bar for those features is that much higher, where it needs to survive feedback from anybody who wants to give it. Which isn’t to say all feedback is necessarily followed. Even we have bad ideas; other people have bad ideas. And we do have to make hard choices about trading off different capabilities.

But you have to be able to have at least a thick enough skin to go through that process and see all that feedback, which tends to be negative, because if you like something you just kind of say, “Eh, I won’t say anything,” unless you are really excited about it. But if there’s anything about it you don’t like, people jump on that stuff. Stuff has to make it through that process.

I think for us, it’s made us more conservative about adding new user facing features, which I think is good in a tool as ubiquitous as the browser. But it also taught us a lot about how to make the features we do add understandable and easily usable by a really broad range of expertise. So I think the product’s much better for that.

OpenOffice is exactly the same sort of case. And your reference to grandmothers downloading in Iowa was the reason that, for us, the whole of the installation experience is really important. We try to keep the download size small. I think we’re just over five megs now, because we know there’s limited time between “I think I’d like to try this Firefox thing” and “Oh, dinner’s ready.” You come back from dinner and never notice that you downloaded it somewhere in your huge downloads folder, and you’re never going to try it.

Scott: Right. Right.

Mike: Keeping that loop short is really important. When you use Firefox for the first time, it pulls in all your bookmarks, it pulls in all your history, all your saved passwords, et cetera. Because you want it to be really easy to start using and be really easy to stop.

You can go back and use IE if you want, because we don’t want you to have to feel like you’re committing your Web to us forever.

Scott: I mean one of the things that you guys obviously have to focus a lot on is reliability. And like a lot of open source projects, you have this modular, pluggable design. And we found that a lot of times that’s really important for projects to be successful. It’s important on Apache that even if people aren’t maybe up to the level they need to be to work on the Apache Web server itself, the barrier to writing a module is pretty low, right?

Mike: Yeah.

Scott: And Firefox, you have the same kind of thing, right? You might not be working on the browser itself, but you might want to try to write an extension to it.

How do you guys architect Firefox to help keep bad extensions from crashing the browser? How do you, for lack of a better term, keep bad code from making your good code look bad?

Mike: Yeah. Not well enough, I think. People using Firefox sometimes load it up with a pile of extensions and then saying, “Performance isn’t what it used to be” or “I get this one weird crash.”

There is an important trade off there, right? We could certainly restrict the attachment surface for plug ins. We could say, “We are going to run you out of process. We’re going to run you in this safer way. We’re going to give you five things you can do to the browser.” And we could become much more robust that way.

Or we could say that you can run arbitrary code. And it’s up to the users to determine whether the given extension is going to be good enough for them or not.

We err, and to some extent I guess it is erring more on the side of permitting the extension developer to do whatever we could do in the product. And there certainly are challenges that come along with that, from helping the users understand that when they install an extension that they are trusting the extension code to do whatever they trust Firefox to do.

But also we try to help developers understand the impact of what they’re doing on Firefox. People write extension often because it’s something that they are going to use, but it’s also something that could be used by any of these 130 million people.

And an individual extension developer may not have access to the same breadth of feedback and breadth of testing that we have across Firefox itself. And we have a quarter of a million people using just our most recent Firefox 3.0 beta right now. Many extensions don’t get that kind of coverage and that kind of detailed analysis by a large group of people.

What we generally do is try to make the things that people want to do with extensions easy and safe. And so in Firefox 3.0 for example, we have some utility libraries we’ve built in, which we call Fuel, which make it easier to work with bookmarks, make it easier to work with tabs and history and so forth, because we were finding that a lot of extension authors were spending a lot of time on that boilerplate. And we want them to spend more time on their cool idea.

But also they were doing it in ways that made it hard for us to make changes later on without breaking their extensions, or they were doing it in a way that interfered with other extensions or interfered with performance.

So we tend more to deal with that class of problem by encouraging better behavior rather than preventing worse. We are certainly interested in looking at models for sandboxing of content, for having extensions that can run all over our privilege mode. They may not have access to as much data or be called in a different way.

But one of the most valuable parts about the extension model for us isn’t just what other people can do with it. It’s that it is a proving ground for our own ideas.

You see stuff coming out of the labs. You see stuff coming out of individual developers or people who are out in the extension development community, if you will, who have a great idea, like session restore, or some of the new changes made to tabbed browsing in Firefox 2.0, and a bunch of stuff in Firefox 3.0 as well. We can look at those extensions. We can say, “Hey, if we want to put this feature in Firefox, we can find out on a pretty broad scale how people are going to react to it by getting 100,000 people to install the extension.”

Sean: Right.

Mike: And that gives us the kind of feedback that you can’t get through even the most passionate online conversation, right? You can put it in front of people and see, in a classical usability test sense, or through instrumentation, or just through surveys afterward, is this a better behavior? What are people blogging about it? What are people using? Are they leaving it installed? And that’s a really powerful channel for us.

Scott: Some applications, and maybe Firefox already does this, will sort of shame the extension developer a little bit. You’ll start up the application and it’ll say, “Hey, last time I was running, this extension crashed me. You’re going to have to manually re enable it if you want it to load.” Is there any kind of architecture like that or is that something that you guys have looked at?

Mike: So we do something similar for session restore, for example. If the browser crashes and you start it up the next time, it says, “You can try this session again, but if it was something you were looking at that crashed you, I can predict what’s going to happen.”

Scott: Right.

Mike: One of the things that is challenging about doing what you are describing for extensions is that the way extensions really work in Firefox is by sort of mutating the application itself. In some cases we could tell that a given piece of code was running. We can look at some kinds of crashes and say, “Ah, well, we don’t normally take that path and this extension did, so that’s likely what’s happening here.”

But in a lot of cases, the way extensions work are by taking parts of the applications that aren’t connected together and connecting them to each other. So they do their damage, but we’re running Firefox code at the time of the crash. One of the things that we have done in Firefox 3.0 is use a new crash-reporting system called Breakpad, which was developed by us and Google for a couple of reasons.

One is, the previous one we were using wasn’t open source. It was the only non–open source code we were shipping and the source of much heartache for that reason. But the value of those crash reports was just way too high. It improved the product so much. But it also didn’t work, for example, on Intel Macs, of which as I am sure you are aware, a couple have been sold.

Sean: Yeah, just a few. [Laughs]

Mike: Just a few. So we have this new crash-reporting software. It also handles threads, which is something that we have been using for a long time, and the crash-reporting software from the ancient Netscape days really didn’t handle threads very well.

We get better crash-reporting data. And one of those things we do report with crashes is what are the extensions that are installed. So we’re able to do correlation that way and then do outreach to authors. And they can then see: this is what these stacks look like. We are able to work with them with my team, specifically in evangelism and developer relations, to say, “Hey, this thing you’re doing is breaking stuff. Let’s get it updated.”

One of the nice things also is that we have spent a lot of time making the extension update model for Firefox pretty smooth. You can push a new update out and it’ll get to users. You don’t have to make sure they all come back to your site and re download and run an installer. It checks for compatibility. It does it in the background.

And so when we do have a fix for something like that out, or more important, when an extension developer has a fix for that kind of thing out, we can get it to users pretty quickly.

Sean: Cool.

Mike: And that’s something we are looking forward to a lot as Firefox 3.0 rolls out, being able to become even more responsive and agile not only with our own product, but with extensions as well.

Scott: Let me ask a question in a different area. There was this huge news story a while back where Coverity had promoted 11 open source projects to their Rung 2. Coverity didn’t make these claims, but the way the news reported it was almost like, “Hey, here is the silver bullet. Coverity’s product scanned the code and found all the defects. You fix them, and you have a secure product.”

Talk a little bit about how tools like Coverity’s factor into how you guys make a secure product. And then what’s the story beyond that? What are the limitations of code scanning tools? What do you have to do beyond that to ensure a secure product?

Mike: Yeah. So certainly we are engineers and we like it when computers can do work that lets humans not do that work. So we are all about tools of various kinds. And certainly security standards are no exception to that.

Our experience with various source code checkers, and I’m not sure what our most recent results with Coverity were, is that maybe they’re not as useful on our code because we are a project that uses a lot of C++ where most of the open source projects are still largely architected in C. Or it could be because we have a very dynamic architecture. You can’t always tell how the pieces are wired together without understanding it or running it.

Generally, the level of false positives is incredibly high. The number of possible vulnerabilities that are reported as “This could be a problem, though it may not actually be one” are vastly smaller. And the number of actual vulnerabilities that are reported are quite small.

And we do act on those where we see them. But I think if you look at it, the Coverity “rungs” are really ratings of how responsive you are to Coverity reports. They’re not really certifying any absolute level of security in the code.

Scott: Right.

Mike: Which is fine. Responsiveness in security is very important. But what they are really saying there is that these groups have responded to all the problems reported to them.

Well it turns out that’s the way they call it. And that’s certainly something we could invest in. We could go in and make sure that we tweak our codes so they don’t trigger any false positives and so forth. But in our experience, that doesn’t correlate with improved security of products.

Which isn’t to say people shouldn’t be using these tools. I think to say, “You know that there were false positives from Mozilla, so we’re not going to bother looking at it” is pretty reckless. Each project needs to look at the tools and figure out which ones are going to pay off best in terms of improvement to security of products.

We do our own. We released last year at Black Hat a set of tools that we built for fuzzing various network protocols and JavaScript language, which we had given to other browser vendors as well, ahead of that release. We found that those helped find a much larger set of bugs and potential problems in our code than the traditional sort of static code scanners.

Where we do use tools similar to Coverity, they are actually ones that we are building in-house and when working with researchers on projects like Elsa. This will let us do static analysis and transformation of our code. And we have a huge code base, and what we really want to do is be able to update the patterns in it as we learn about new vulnerabilities or performance patterns or portability or take advantage of new language features.

When we started to build Mozilla in 1997, C++ compilers were not nearly as capable as they are now. So there were a lot of features we just couldn’t use if we wanted to run on everything from Win 32 to AIX to OS/2 and so forth.

Scott: Right.

Mike: So we can take advantage of, for example, C++ exceptions in a way we couldn’t before. We are writing tools to help analyze our code and also to write big chunks of the code.

Sean: Interesting.

Mike: So that’s where we are putting our investment in terms of the tools. But I think those that are trying to make a secure product and aren’t doing everything they can to help use the computer to find issues, not as a substitute for human analysis but as an adjunct to it, are doing themselves and their users a disservice.

Scott: One of the things that we do a lot of the times at the end of these interviews is hand the microphone to the person we are interviewing and say, “We covered a lot of stuff that was interesting to us, but what’s maybe interesting to you that we didn’t specifically ask about?”

Mike: Yeah, I think the whole series you’re doing here about the software processes that are different – well, we’ll see whether they are different or more similar – between open source and proprietary model, I think is really interesting.

And I think it’s going to be interesting to see how those differences in process manifest in terms of users affecting things, whether it’s agility of features or improved responsiveness on bugs, or security issues, or days of vulnerability. Whatever we decide we want to optimize for in a given path.

But one of the things that we are doing with Firefox – and we hope with the Web – is making it a more robust and effective way to build an application. So it’s interesting for me to reflect on the questions you’ve asked and look through the various blog posts and so forth.

The kinds of processes that we want to encourage there, the kinds of things that we want to make easier for developers or make their results more effective, is how can we bake those things into the Web?

And we are certainly very active in various standards communities. We are very active in working with language developers and Web application developers and other browser developers to be sure to have these capabilities. But doing them in a way that doesn’t require the forklift upgrade of the development process.

Like, you don’t have to throw out this app if it has everything you need except video and rewrite it in something that has video. We want to take the barrier points away. But I think we will do well to keep in mind the development processes that lead to those apps as we do that, and make sure that what we are doing builds well for people who are using traditional proprietary methods, whether they are doing more collaborative development or whether they are trying to find a hybrid, as MySQL did.

And it looks like it’s worked out well for them, based on the acquisition announcement.

Scott: [Laughs] That’s right. Mike, thanks for taking the time to chat.

Mike: Thank you.

Bookmark this:

Interviewee: Justin Erenkrantz

In this interview with Justin Erenkrantz we talked to him about:

• The Apache Foundation.
• Project Management Committees and the Apache Foundation.
• Some of the reasons Jason feels Apache has been so successful.
• What other open source projects might look to Apache for in terms of inspiration due to Apache’s longevity.
• The Apache Incubator and it’s role as part of the Apache Software Foundation.
• What is the on horizon for the Apache Web Server.

Justin:  So my name is Justin Erenkrantz. I’m currently president of the Apache Software Foundation (ASF), also on the board of directors. And I’ve been a contributor to the Apache HTTP Server, the Apache Portable Runtime, and Subversion, and some other projects for quite a while now.

Scott:  Talk a little bit about Apache and talk about how it’s built.

Justin:  The Foundation as a whole has over 50 different projects. There’s the Web server, Tomcat, SpamAssassin, Geronimo. There’s a whole variety of projects. So there’s an overall Foundation, with committers for each project. They’re relatively isolated. The fact that I have access and I work on HTTP Server doesn’t mean that I have access to, say, Maven.

Each project kind of gets its own merit. Our culture is a meritocracy, and so we expect people to show up on the public mailing lists and start contributing, and eventually they’ll be recognized. Eventually they’ll get a vote, and this vote will allow them to be able to commit (code). Across the Foundation, we have about almost 1,600 committers that can commit to some of the 50‑something projects. And with that, you get to make changes.

You also get something called a veto, which is something that we can probably talk about a little bit later. That’s one of the core governance structures that we have. There’s also a thing called the Project Management Committees, and those are the groups that are responsible for each one of those projects.

Above that is the board of directors. And in comparison with some of the other open source organizations, our board at Apache doesn’t get involved with technical details. We’re not going to get in and say, "Oh, you need to change this variable name." That’s not at all what we do. We’re just there to make sure the organization is running, make sure that everybody’s happy and getting along. You won’t see a director really getting involved in a technical discussion, unless they’re a part of that project to begin with.

Scott:  Does the board provide more of a steering functionality, then?

Justin:  No, not even that. We’re almost completely hands‑off. There are times when personality clashes happen in each project, and try to mediate situations. Again, we’re not going to make technical decisions.
There are some things that are centralized. So one of my responsibilities, as president, is I’m responsible for the day‑to‑day operations of the Foundation. We have a Subversion server. We have issue tracking. We have websites. All of that is centrally managed by our infrastructure, too.

Scott:  One of the things you mentioned was votes and vetoes as the way the project is governed. So expand on that a little bit. How does that come into play?

Justin:  For example, there’s the HTTP Server project. Within that, it’s governed by a Project Management Committee, a PMC. Looking at your past interviews, other people have mentioned this. This kind of structure has replicated itself on how we have it at Apache.

These are groups of committers who are responsible for the project. On HTTP Server, there are maybe about 60 to 70 people on the PMC. Every single one of those people has what we call a binding vote. The votes are used in two main ways.

The first way is for release. Within Apache, across any of the projects — this is one of the hard and fast rules — there must be three binding votes before it can be released. That means we must find three people out of 60 people on HTTP Server to say, "Yep, this is a good release. We’re going to put the Apache brand on it. It’s going to be the ‘Apache HTTP Server’." That’s the release part of the vote.

The other part is a veto. If even one of those 60 people say, "This change is bad, I’m going to veto it," that means the change doesn’t make it in.

One of the ASF’s founders, Roy Fielding, refers to it as a kind of a shot gun. It’s kind of like, "OK, I’m done discussing this with you. You’re not listening to reason. Veto. Stop. We’re not going any further."

And it’s usually a last resort. Vetos are uncommon. They’re not something that happens every week or every month. Generally, if there does happen to be a lot of vetoes, it means that people aren’t willing to compromise. So, that may be something where maybe the Board of Directors might say, "Hey, you know, we might keep a close eye on it. Is there anything you need to talk about, do you need any help to resolve this…"

But, generally vetoes are relatively rare but they do give a power to the 60 people to say, "You know what, there’s not going to be any change that I disagree with, there’s not going to be anything where I say ‘Oh my God, I can’t live with this change being made’." So that’s an enormous power and it’s given to the members of the PMC.

Sean:  We found these nuclear options in place across many projects, but they don’t get used very often. It that involvement in open source takes considerable time and you do it because you believe in moving the project forward. It seems like going into this, people know that the only way to make progress is by consensus. No one person is just going to get their way. And if they really want to, they have another option, which is to go fork the source. But that has extremely high barriers also.

Justin:  Yeah, absolutely. But as I said, generally the vetoes tend to be very rare. It’s almost like mutually assured destruction. That’s the point. One thing that has happened a couple of times over the 10 or 11 years of the HTTP Server Project, is where there were vetoes on both sides. "We’ve got to do it this way. No, we’ve got to do it that way". There was this huge flame war for a couple of weeks and then they finally said, "You know what, we’re not going to agree". They even had a telephone call. They were doing everything, it was just this big mess. And eventually they said, "You know what, we’re just going to leave it to a vote. We can’t agree, we need to move forwards, this is blocking us. OK, we’ll go ahead and resolve this, whatever way the vote may turn out".

Scott:  Talk a little about the history of Apache. How did it get its start and what are some of the major evolutions it’s had getting to the present point?

Justin:  Apache has its roots back in the early days of the World Wide Web. The story begins with NCSA Web server from Illinois Urbana‑Champaign. They were running the NCSA Web server. Eventually, a lot of people left to go to Netscape. The NCSA code eventually got abandoned, more or less. I think there were nine people who found each other on Usenet and said, "Hey, I have a patch for NCSA. OK. Why don’t we start trading patches?"

They got together and they started to exchange patches and start coming up with a new version of this NCSA service. They started taking it in this new direction. They started saying "OK. Maybe we should get another group going." So they founded something called the Apache Group. It was an informal thing. They did that for about four or five years, starting in ‘93 or ‘92 (started in Feb ’95) Eventually they got to the point where other people said, "Hey, we like what you are doing."

By this point Apache had already gone through, [inaudible] a Web server, and made it up to version 1.2. They created The Apache Software Foundation in 1999, and started doing things besides just a Web server.

That was the start of the Apache Software Foundation. The early initial project was the Web server, and that is still what a lot of people think Apache is. Now you have close to 60 projects.

Scott:  Focusing on the Apache Web Server, what are some of the secrets of its success?

Justin:  For the web server, more than anything, it’s been the way we designed and supported all the standards. And it’s free. That was the tagline that Roy Fielding had on his website for a long time, "Apache, the best web server money can’t buy."

One thing that really speaks well of our community has been the lack of forks. The community embraces anybody who shows up. The project has evolved and widened, from just the Web server itself, where people have wanted to do new things. People wanted to do an FTP server, a mail server. It can do all those things today. The community has been characterized by being willing to be open to just about anything.

Scott:  It seems like open source projects that are modular do better because working on the core of an open source project might have a high bar. If it’s modular, you can write modules without going through the scrutiny of submitting code to the core. Modules give you a way to get your feet wet and participate. At the same time you need a really healthy community too. It is the personalities and way the governance structure is set up around it. It has to be really healthy as well. Just to say it back in my own words, those two things seem like they came together with Apache…

Justin:  Yeah, and if you look at one of the key evolution points between the original NCSA server and Apache, it’s when (early Apache developer) Robert Thau modularized the whole thing one weekend. By and large, most of what he did 10, 12, 13 years ago is still present in the code base and technical architecture. By modularizing, he did a really good job of cleaning up the earlier NCSA code base.

Scott:  So what is version 2 all about?

Justin:  Version 2 was all about threading and portability. With Apache 1.3 they added Win32, Netware, and OS/2 support.

Version 2 started out with a number of internal forks. One of them looked at Netscape’s portability runtime. Other developers did their own portability library, implementing the same function on three platforms and hiding the implementation details."

There ended up being a licensing dispute with the Netscape/Mozilla guys that prohibited the Apache guys from using the NSPR runtime. That spawned the Apache Portable Runtime project. That’s a lot of different projects now, but if you look at why it happened, it had to do with the licensing issue. If you look at the Foundation now, I think one of the things we are well known for is the terms of our licensing.  It’s a key differentiator from, say, the Free Software Foundation.

Scott:  Right, right.

Sean:  Apache’s been around for a really long time, and it’s obviously seen as one of the more successful open source projects, to say the least. What do you think other open source projects look to Apache for in terms of inspiration when they’re starting up?

Justin:  I think by and large, what you see most people copying are the governance structures and the licensing. Those are two things projects have been copying. I think you can see that in Eclipse: they almost use some of the exact same terminology.

Scott:  You still there?

Sean:  One of the things you mentioned earlier that has always been intriguing to me, was portability. I can see that it’s really important for Apache to be portable between different Linux flavors, and maybe even be portable to embedded devices and things like that. How important is it, from a practical standpoint, that Apache runs on more than just Linux?

Justin:  Extremely important. We have contributors who are only interested in supporting a NetWare or Windows or OS/2, even BeOS in the past. It’s where we’ve gotten some of the diversity of the community. It’s a hook to get people into the community. "Here’s a little something I know about, I know my operating system, and I’ll contribute this patch. Hey, there’s something else that may not be platform‑specific."

Scott:  Does Apache take the standpoint that it should run equally well across operating systems? OpenOffice, for example, wants to be pretty much the same OpenOffice regardless of where it’s running. Does Apache run differently depending on…

Justin:  Absolutely differently. Basically, our approach is in whatever platforms people want to maintain, that’s what gets supported. By and large, on the Apache HTTP Server, we have one guy who does the Win32, and it’s been his baby for many, many years. There are other people who contributed a little bit to the Win32, but he’s this one person who had been the individual who is responsible for it.

It’s not a dictate that, "Oh we have to support that." If someone is interested in supporting an OS, great! We’re not going to stop them, but it’s not going to be a mission statement, that we have to support all these platforms equally.

Actually, if you do look at our HTTP server mission statement, it says, "Apache HTTP Server Project is an effort to develop and maintain an open source HTTP server for modern operating systems including Unix and Windows NT." So, it’s in our mission statement, but the only reason it’s there is because we have the contributors to provide that support.

Scott:  One other thing that varies from project to project is where the code comes from. If you take a look at MySQL, pretty much everybody working on it works for the MySQL company. If you look at other things like the Linux kernel, a lot of that comes from corporate developers: IBM, Red Hat and a lot of people. Do you have a sense for where the Apache code comes from? How much of it is from corporate‑sponsored developers versus the proverbial guy‑in‑his‑garage?

Justin:  I think it comes from a wide number of sources. What you will see is that contributors remain the same even when they move from job to job. That’s definitely been the case within the HTTP Server, that’s been the case for some of these older projects as well. One day they may be working for IBM, the next day they may be working for Red Hat, and then they may be working for some other company. They may be working for Google, maybe doing it on the side, that’s what you tend to see. Some of these contributors may have started out working at Sun or HP, then they move but they’re still working on it. They still contribute to the project.

Scott:  There are certain people who look at open source and they think it’s all written by people in their garages, contributing. Other people look at it and say, "It’s all written by people working for corporations." How important do you think big corporate sponsorship is to a project like Apache, and does that also create certain challenges for the project?

Justin:  It’s a balance. You see some people who are getting paid to work on it. They work on it all day during normal business hours.

Then you see people who are the exact opposite, who may be working as a system administrator or something else, and they only time that they can work on it is on the weekends. So you see the overlap.

One of the key things in Apache, another quote from Roy Fielding is, "If it didn’t happen on the mailing lists, it didn’t happen." All of the discussions, all of the decisions, have to be made on our published mailing list. That allows people who may be in different time zones, or different work schedules, to coordinate through this mailing list.

They can read it during the day when they’re at work, during the night when they’re at home, whatever works for them. That way, decisions aren’t made in a face‑to‑face meeting, or a call, or an IRC, all the decisions have to happen on a mailing list.

Scott:  So IBM, just to take a big company name, can’t get something into Apache just because they’re IBM and they want it. If one person out of sixty people vetoes it, it doesn’t really matter how badly a big company wanted certain code in, it’s not going in.

Justin:  That’s right. The other aspect of it, the thing Apache has been addressing the last couple of years, is how new projects come into Apache through something called the ASF Incubator. This is about how they operate as an Apache project. They have to get all of the legal paperwork in place, so we can say, "Yes, we can release this under the common Apache license." That’s how we are trying to get new projects, and that’s why you’re seeing growth in the number of our projects, because incubator keeps spinning out new projects.

It’s always a concern that in order to graduate from incubator and become a full‑fledged project, you have to have diversity. Basically, you can’t have any one company dominate the project.

The rule we use, that you see pop‑up again and again in Apache, is the rule of three. There must be at least three committers that are diverse. The discussion that is going on right now is, "What is the definition of diverse?" An example: "Well, I work for IBM, and I work on this project full‑time, but there’s another guy from a completely different division who isn’t getting paid to do this who’s also working on it." Should that be counted as a separate individual? That is a discussion now. Some of these companies are so big, it’s like the old joke of, "Oh, you’re from London!  You must know so-and-so."

Scott:  In open source projects what gets checked in is the source code. With Apache, it looks like there’s this thing called the Apache HTTP Test Project.  Is that essentially like a test suite for Apache?

Justin:  Yes. Yes it is.

Scott:  OK. And what’s that focused on? Is that mainly functional testing?

Justin:  Yeah, it’s basically a Perl‑driven test suite, originally from the Mod Perl guys. They had this whole Apache test tool kit that they used as a kind of smoke test. And we said, "Hey, we’d like to take that." And we extended it from there.

Generally, what you’ll see is you’ll see people will use that as a kind of smoke test before they do a release. We talked earlier before about that you need to have three plus one in order to do a release. But we haven’t said anything about how people make up their minds, and say, "Yes, release this." And so generally what people do is they run tests on their favorite platform.

One of the things that we did with 2.0, and still do to some extent is "eat our own dogfood."  In the early days in of the 2.0 series for the Apache HTTP Server, we would say, "OK, we have a release candidate. We’re going to put it up on Apache.org. We’re going to go run it for 72 hours and it can’t crash."

Basically that was kind of another way of doing the acceptance testing. Saying, "OK, we can run it on a site that gets this much traffic. It didn’t crash so it’s probably going to be OK for you."

Scott: Are there tests specifically looking for vulnerabilities like buffer overruns, or is that really outside of the scope…

Justin:  In the past when there’s been some type of buffer overflow, or some type of CDE vulnerability, generally you write and check in a test to make sure it won’t show back up in regression.  I think that basically depends on if we can come up with an easy reproducible test case.

But I think you won’t see some test cases there that are typically for the vulnerabilities.

Scott:  A while back we talked to Michael Howard, who’s a security guru at Microsoft.  There’s a lot of things they do, but one in particular was banning certain APIs like strcpy because they were just inherently vulnerable.

Justin:  Yeah. Basically we do some things and put them in Apache 2.0 with the APR path (code base). [sp]
If somebody actually tries to call these functions, it’s going to expand onto, "Why are you trying to do this?" There have been some cases where we put it in the file to say, "Don’t do this. Don’t call this." Or say, "Oh you’re going to call this? Well then we’re going redirect you to a safer version."

But, we don’t have the flexibility of say a Microsoft, and say, "Oh we want to have this new security API in the operating system." That’s not something that we have influence on.

We generally have to look into the constraints of the operating system and work with that.

Scott:  Sure.

Justin:  Over the past few years, I think there have only been one or two cases where there were remote root exploits, and that speaks well for us.

Scott:  When people are posting patches, are the security implications discussed on the mailing list?

Justin:  Oh, absolutely. You’ll get people saying, "Hey there’s something with this vulnerability, or this will break this or that."  So, yeah there’s this constant vigilance for the security.

Scott:  Are there a lot of security-related tests that are put in proactively? I hear about things like "fuzz testing" and other proactive ways to probe the surface area for vulnerabilities. Does that kind of thing happen….

Justin:  There are security product providers, Coverity is the one that pops to my mind. They’ll say, "Hey, we ran our tool on your code and here’s a report of vulnerabilities." We take a look at the reports and say, "Thank you very much." And then analyze them ourselves.

But it’s really triggered by what the committers are interested in. We’ll see committers who are very interested in conformance to the protocol specs. We’ll see people who are interested in security, people who are interested in performance, etc.

But we don’t tell the committers what to be interested in from the top‑down. It’s more like, "John is interested in security so he’s really focused on tying up all the security issues."

Scott:  I’ve bumped into companies like Coverity that use open source to market their tools because open-source provides a large, free code base they can throw at their tool.

Justin:  Absolutely. Yeah, I remember when we first looked at Coverity, the amount of false positives were generally high.  We’d look at the code and determine, "No, there isn’t a vulnerability there. What the tool is reporting can’t actually happen." There were maybe a handful of actual things that we said, "Yes this is an issue".

They may not have been as severe as what the tool was claiming, but we said, "OK, we’ll clean this up."

Scott: What do you see on the horizon for the Apache Web server?

Justin:  The IETF is forming a new working group to do an editorial revision of the HTTP stack. Their work might lead to the next generation of the HTTP protocol. And I think that is something that we will be very much involved with.

Scott:  What about in the incubator? 

Justin:  Every Board meeting we’re graduating things like ServiceMix which is an ESB and component suite based on the Java Business Interface. One that is going to be new, probably the next board meeting, is a standard C++ library, which we’re getting from RogueWave. There’s things like Abdera, which is an Atom feed. We’re seeing a lot of things like ActiveMQ which is event‑based messaging.

What you probably see a lot in Apache are low‑level infrastructure type things. You’re not going to see things like, say, OpenOffice. You’re going to see things that people can pick and choose to build larger applications. I think that’s what our niche really is.

Scott:  What’s your sense for what happens to an Apache Web server release between the time you’re done with it and it gets distributed by a Red Hat, Oracle, Solaris , etc?

Justin:  We have contributors from Red Hat. A lot of the people who are doing distros and ensuring that it gets in front of the users are involved in our community. Our philosophy has been why are you making this huge patch set for this particular product? Get it upstream, get it back to us, we want to take it. I think generally, for the most part, you will see that there isn’t a lot of variation when it gets into the distribution because these people have been working with us.

Sean:  Let me do a follow‑up on that. Some say the strength of a closed source project is that the company may be able to provide more of an integrated stack. You take a look at something like Microsoft ships and they might say, "You should use this because it’s an integrated stack, and there’s a single vendor you go to for support." Obviously there are some projects that are tightly coupled together, such as Suversion and Apache. What would you say to that from the open source side?

Justin:  Right.

Sean:  It seems like in open source, the communities are not isolated.  There’s a fair amount of core maintainer communication that’s going on.

Justin:  Yeah, I think you see that. I think that’s why you see a number of committers in multiple communities. As you get used to it, you start to follow the dependency chain, and you get into those communities and say, "Hey, I just broke this for you over here, but here’s the patch to fix it." You tend to see a lot of that happening.

Sean:  Are there things you see in other open source projects that look interesting and might influence Apache?

Justin:  I like what Ubuntu has been doing, where they say, "We’re doing a release every six months. (no matter what), we’re going to have a release." That’s a very hard thing to do. That requires some of the dynamics that Canonical has with their contributors. That’s something that I think they do really well.

Generally our philosophy has been, “we’re releasing when it’s ready”, and some think that’s a good philosophy. You don’t want to promise something, but then you think, "Well it’s been so long since the last version." There are all these changes that sit there and keep getting improved upon. But if you have the regular release cycles, I think that’s a good thing.

Scott:  Justin, we’re out of time, but this has been a great conversation.  Thanks for taking the time to chat with us.

Bookmark this:

Interviewee: Robert Bray

In this interview with Bob, Architect for Geospatial Products at Autodesk, we asked him about:

• Introduction of open source within Autodesk
• Role of project steering committees
• Decision to create an open source product
• Open source impact on closed source product
• Developer and engineer adjustments to open source process
• Differences between feature design in open and closed source
• Test and documentation for Autodesk open source
• Community contribution to open source projects
• Business reasons for creating open source products

Sean: Bob, why don’t you give us some background on your role with Autodesk.

Robert: I’ve been in software development working for independent software vendors for about 17 years now. My current title is Architect for Geospatial Products. So, I’ve gone through the various ranks, from being a programmer in the trenches, in the cubes, to managing a product development team, to my current role as architect.

So, as I said, I’ve been through various things, mainly in closed source. Until roughly 18 months ago when we open sourced MapGuide. I think it was about six months before that that we really started heavily investigating that idea and started to put the ball in motion to actually make that happen. Basically, I’ve been involved in an open source project very actively for the last 18 months, and I won’t say that I was the key driver that made the decision that we were going to open source this, but I basically took it from that decision point and said, “OK, these are all the things we need to do…” and made that happen.

The product itself is MapGuide.The open source version is called MapGuide Open Source. The closed version is Autodesk MapGuide Enterprise. The codebases are the same. There’s one open source code stream for the whole product. There are a couple of minor functional differences between Enterprise and open source, but they are extremely minor, and we are trying to close as many of those as we can, actually. What we really have with the Enterprise version is basically a commercially tested and supported version. It’s much like the Red Hat model or the MySQL model, in that respect.

Sean: So, just out of curiosity then, if there are some functional differences, what led to those? I mean, they’re minor, like you said, but was some of that just by the nature of having to switch from closed source to open source? Is there some functionality that just couldn’t make it over that transform right away?

Robert: Basically, that is exactly it. We have a commercial desktop mapping product called AutoCAD Map 3D. We used some technology from that in the Enterprise version of MapGuide, and we do that for compatibility between the two products. So, those particular pieces, our coordinate system library, in particular, and our raster image handling library; those are things that we just could not open source at the time.

Sean:So, when Autodesk made the switch, did you model against an existing open source project in terms of how you wanted to handle QA or how you wanted to establish a community feel? You look at different products, like Subversion – I’ve heard they had one vote in the entire history of the project, right?

Robert: [laughs]

Sean: Literally. I watched their presentation at OSCON, and at least that’s the public face of things – that they had one vote. Or, for example, we talked to PostgresSQL yesterday, and they said, “It’s a very communal democracy” in terms of their decision process. They don’t really have a hierarchy. I know it varies across the projects, so did you look at the open source community and say, “Well, now that we’re going to open source this, we’d like to mimic this particular project” at least in some aspect?

Robert: Absolutely. So, when we made this decision, one of the first things I did was take a broad look across all of the projects. I was mainly focused on looking at the geospatial projects, because, if you will, our kin in the community is all of the open source projects – things like MapServer, GDAL, PostGIS… All those open source projects are very close in nature to us, so I certainly did look at those.

I also read a couple of books. The one I liked best was Karl Fogel’s book, “Producing Free Open Source Software.” Of course, he led the Subversion effort, but if you read that book you wouldn’t know there was only one vote in Subversion.

Sean: [laughs]

Robert: [laughs]

Sean: I was a little surprised, too. But, these were two guys from Google, so technically they’re reinforcing each other in their viewpoint, at least. It’s not just one guy saying it. They’re committers on the project…

Robert: I’m shocked.

Sean: Yeah, yeah. I was a little bit, too. But that’s part of the fun part of this investigation is some of the stuff we hear, and then we get to cycle back and say, “Really? So tell us a little more about that.”

Robert: So, we definitely took the approach of forming a community, and the project, and I say not from day one, but within six months, became a community driven effort. MapGuide has always been a product that, it’s almost like there’s a religious following of users out there. Even when it was a commercial product, it had some real hardcore, dedicated users.

So a handful of those users said, “Wow! This is going to be great!” They knew nothing about open source at the time either, but they kind of all jumped on board right away, started testing the open source version, and started getting active in that.

So, when we formed the community, basically, because we were modeling after MapServer and some of the other open source projects on the geospatial side and I really felt that was important we basically created a project steering committee for the product out of that group of people. So that project steering committee, right now, consists of three people from Autodesk and four people from outside in the community.

Sean: Are those people then the chief committers on the project?

Robert: Actually, that’s an interesting thing, because I would say there are a couple of people there that are active committers; the rest of them are users.

Sean: Got you. Because the interesting thing to me about this is that, obviously, a lot of these projects have grown up over time. PostgresSQL has a 20 year history, and to become a committer on their project, they told us, “Well, we wait two to three years to figure out if we can make you a committer, because we’ve been around for 20 years.”

But, if you birth a project out of the ether how do you determine who’s going to be a committer? Because, obviously, there’s probably a stampede of people at the start that say, “I’d like to be a committer, I want to be involved at the highest levels,” that kind of thing. How do you go through that winnowing out process?

Robert: There’s a URL I can send you, (http://mapguide.osgeo.org/psc.html) but basically what I did was draft a set of rules for the project steering committee to follow that include who can be a committer and how they become a committer.

We basically go through a similar process to, I think, Postgres – although we probably wouldn’t be two years in the road doing it. Basically, people start submitting some code from the open source community. They submit it to a developer, we’ll have that developer review it, and then do the actual commit. It’s almost in a patch style, right? Then, once we’re comfortable with them, we’ll vote to make them a committer – it’s a project steering committee vote.

Sean: Got you.

Robert: So, it’s kind of interesting, because, again, the project steering committee is not the core developers; again, it’s a core set of mainly users.

Sean: Once you transitioned over and said, “I want to make this an open source product, ” since you came mainly from a closed source background, what was the thing that most pleased you about being able to transition into an open source development model? What was the thing that you were, I guess, either most afraid of or most reluctant to see transition over? Because, obviously, the models are pretty different.

Robert: I think I was scared to death for quite a while…

Sean: [laughs]

Robert: …because it was completely unfamiliar territory to me. I’ll be honest, right? I’d never worked with an open source project before. I’ve used a few, but I’ve never really participated in an open source project. So, from my perspective, open source projects were these big free for all things, and I’m like, “Oh, my God. What are we getting into here?”

But, when I stepped back and actually looked at it, what I realized is that open source projects…yeah, they run a little differently than commercial projects, but they follow a fairly well-defined process; or at least the large projects do, that are successful. You can see a well established process there. It’s not a free for all, anybody can commit code, and a thing like that. So, that was my biggest fear going in; it was the step into the unknown, right?

Sean: What do you think was the biggest lesson learned from that? Now that you’ve got a couple of years at it, what would be the thing you might’ve done a little bit differently, given an opportunity? It’s apparent you wouldn’t have changed any decision to go open source, and that rings through clear. But, is there anything you would’ve changed during the process of going in that direction?

Robert: If anything, I would’ve been more open with the community right from the get go. I think I had a lot of reservations going into it, and so I didn’t really communicate as much as I should’ve right up front about what we were doing, how we were going about it, and all of that. So, I think I would’ve been much more open around communication right from the get go.

Sean: OK…

Robert: That’s the biggest lesson I think I learned.

Sean: …then what was kind of the biggest positive that you got? I didn’t want to leave that off the table either.

Robert: The biggest positive is that from a community perspective we are getting a lot more up front review; a lot more users using features before they go into, say, the commercial version or whatever, right? We’re getting a lot more early feedback on things that we’re thinking about doing or the ways we’re thinking about implementing new features, lots of early feedback that has a very, I would say, positive influence on the end result of the code that we actually write.

Sean:Well, I’m curious, too. That’s a functional change in the product, but have there been any change to the engineering process on the closed source side of the product? So, you guys open source it, that obviously rattles around and after a couple of years; I’m wondering if that has had any kind of change in the way you guys have developed internally, just having an open source mirror of the project, so to speak.

Robert: Well, it’s not an open source mirror. We only have one copy of the code, and that’s the public one. So, any change that we make to the code, whether it’s for the commercial release or for the open source release, typically gets vetted through the same process. We’ll go to request for comment. That’ll go out to the open source community, because it’s going into the open source version, one way or the other. So, we get early feedback on pretty much everything that goes in.

Of course some features that we may plan are for the enterprise version only, and those of course do not go through public review. Also we reserve the right to decide which features go into which enterprise release and we make no announcements about that because we are a public company right?

Sean: Right, right.

Robert: We don’t comment on that. We don’t do anything like that in the RFC process. But at the same time, the open source community certainly gets to vet all this stuff early. So, any code changes, and basically the whole product’s development process, are morphed into more of an open source development process for both the commercial and open source releases, because, again, we only have one copy of the code and we have one development process.The entire engineering team has basically transitioned to an open process, which is quite interesting.

Sean: Taking it down one level, as we talked to people sometimes as they’ve gone through this change, and the engineering team has a variety of differing reactions to it…various resistance points, or to put it maybe slightly more diplomatically, just things they have to learn as they go through the process. So, anything that would be worth talking about there in terms of lessons learned by the engineering team as they walked through this?

Robert: Yeah, I think there were a lot of lessons learned. Immediately it comes back around to being open about the kind of change we want to make and how you actually communicate with an open source community, because communication within the internal engineering team is quite different in my experience than communication with an open source community.

Sean: Now all of a sudden it’s a much more communal feel and development managers all of the sudden, I assume, had to make the switch from assuming everybody was on the payroll; there had to be a little bit more negotiation that affected the process of getting something developed, vetted, built, etc.

Robert: That’s definitely part of it. But also for our internal developers, I mean, they’re used to this kind of hierarchy of being able to walk down the hall and talk to somebody else. They’re used to being able to call a meeting and go in and start scribbling some stuff on a whiteboard.

With an open source development community that doesn’t really work anymore. You need to use the mailing lists a lot more. You need to use things like IRC for communications. So, it was really a complete change in the way you communicate.

Scott: So, that’s one of the things that I wanted to drill into because you’re obviously steeped in closed source development methodology. You’ve been in open source long enough to where you’re now immersed in that.
What do you see as the differences, kind of taking them side-by-side when you think about the aspects of the SDL, right? …how’re ideas envisioned, how they’re kind of tasked for somebody to code them, how they’re put through QA, how a product is released… If you don’t mind, I wouldn’t mind just kind of walking through how each of those is sort of fundamentally different in a closed source versus open source project.

Robert: Sure.

Scott: So, I guess starting with analysis and design and that sort of thing, how do you see analysis and design and requirements differing in open and closed source?

Robert: Sure. The requirements from a closed source perspective, I mean, we have project managers who basically go out and talk to our users. They’ll do focus groups. They’ll do surveys; all kinds of things like that just to basically pin down to a set of requirements for release of commercial software.

For open source software, you have a community of users who propose some ideas or a developer who proposes an idea. It gets kicked around on a mailing list for awhile. Eventually somebody either just picks it up and implements it, or the other option is they just walk in with something they’ve already written and say, “Hey, how about this? Isn’t this great?” So, the difference is really to me that it’s a little bit more organic, if you will, in the open source community. Whereas analysis and requirements and product definition is a much more rigid process in the closed source world, in the open source world it kind of happens organically.

Scott: So, where do you see those potentially? I’m guessing that both sides have their advantages. What do you see as the advantages of focus groups, meeting with customers, and project planning and what do you see as being the advantages of kind of more of the organic method of sort of showing up with a feature and saying, “What about this?”

Robert: If you show up with a feature and it’s already got a prototype or an implementation done – it’s a lot easier to talk about because you can actually see it and you can play with it. Whereas in the closed source side, you’re talking more in the abstract, if you will. So, the advantage in the open source side is really that you typically get a lot more users providing some feedback on it, at least if they’re really interested. Eventually you get to experience it before it actually goes in, which are both useful things.

On the closed source side, I mean, the advantage is that the release potentially comes out with a set of features that are much more…I won’t say well-defined because I think that’s the wrong term, but a release will typically come out that is much more cohesive as a whole. Instead of having a bunch of new things sprinkled about like a typical open source project has that’re mostly unrelated, a typical commercial product release will have the sort of features that are more or less a cohesive unit, if you know what I mean. Does that make sense?

Scott: Yeah, yeah. When I was a developer one of the problems with me kind of deciding what should be in the product was that I could have a project manager come along and say, “The product should really do X.” I would try to convince them that it shouldn’t, not because I had a better knowledge of the customer than them, but because I knew what a mess that would be to try to code given the existing architecture.

Do you feel that that kind of thing happens on an open source project because a lot of the people working on it kind of know what it would take to code it? Like, they might shy towards or away from certain things?

Robert: Yeah, you don’t see that in the open source world in my experience.

Scott: OK. So, ideas get proposed on the mailing list, things like that. You guys have internal developers who are still working on it. Is it that everybody sort of volunteers to work on different things or do your own internal developers get tasked with certain features? Obviously in the community it’s sort of all volunteers, so somebody external has to decide to pick something up and run with it. I guess, talk a little about how that works in terms of actually getting the work done.

Robert: So, it doesn’t work all that differently for us because we do have an enterprise version of the product and we do have a product manager who still does all those typical closed source things. What typically happens is that the features that we foresee as important to get into the product will basically go through a similar process as a typical closed source world, but then we transition them over to the open source world.

So, we’ll basically write up, you know… if we have an idea we’ll either code it first, or more likely we’ll write what we call a “request for comment” document that basically describes the change we want to make then actually put it out there to the open source community to vet. Again, because we’re sponsoring it, the things that have somebody behind them who’s actually going to write the code will typically get approved by the PSC.

But, the benefit that I’ve seen in that is that we get a lot of really good feedback, positive influence, and changes to those RFCs before they actually get coded because we have a pretty broad user community out there in open source basically providing us with some really solid feedback.

Scott: Then in terms of things like QA and documentation, how much of that kind of gets done by the community? How much of that do you guys still have to take on?

Robert: For the open source project we do no QA. The open source community is responsible for all QA. So, between the software development team itself and the open source community, that’s where all testing happens for the open source releases. We do not have any type of internal QA or formalized QA process on the open source project.
We’ll decide that we want to cut a release, and the project steering committee will release a beta. We’ll get some users testing it, providing us with bugs. We’ll fix those bugs.

Autodesk has traditionally done betas, but they’re fairly late in the game. I would like to see us actually release our betas a lot earlier than we currently do to get more of that community testing done earlier in the cycle. Then, we have more of a chance to fix bugs, and make changes before we do our final release. We’ll go through basically one or two betas and a couple of release candidates. We’ve done up to two. We may do a third on 1.2, and then we’ll release the code.

So, with open source there’s no formal QA process other than the unit test has to run successfully on every build. But basically, it’s users hammering it, whereas for the closed source project we have our rigid, internal QA process. We have our internal QA team that does a fairly formal quality assurance testing that you would expect from commercial software.

Scott: Do you feel that one way or the other is more effective at kind of wringing the bugs out of the product?

Robert: What my gut tells me right now is that a blend of the two is actually best. The nice thing about the rigid QA testing is that when you release the product, you know exactly which code paths have been tested and which ones have not. What you don’t get out of that typically is the kind of real world testing that a user will put it through. So, my current thinking is that a blend of the two approaches would be better.

Sean: Just a piece of context – how late is late for you guys? If the project has a 36-month development cycle, or 24 months, or whatever, from the initial release definition or spec process down to when you actually have people buying the product, how far along on that trajectory are you guys typically releasing betas?

Robert: In commercial software typically our first beta release is about four months before release, and that for us is late. Part of that, though, is because Autodesk has adopted an annual release cycle, so that has dictated shortened cycles for everything. But, if we could find a way to release those new features earlier for beta testers to test, I think that would be a huge improvement and a way to blend the best of the open source world with the closed source commercial QA testing.

Scott: One of the challenges people sometimes run into when they open source an existing project is that on day one when you open source it, all the jobs on the project are already filled. You already have a team of 10 or 100 or 500 or whatever number of people who are engineers employed working on the projects, and they’re already sort of doing everything. So, there isn’t necessarily a feature that’s just waiting for somebody in the community to write.

So, from this perspective that kind of caused a lot of consternation internally about what was that going to mean for the people whose job was to work on the project. It kind of created issues with the community on day one about OK, well great, what do you really want us to do? I mean, what’s really open for us to work on that you’re not already working on?
Did you guys run into anything like that on this? I’m interested in kind of comparing the experiences. You obviously have a different product, and I’m interested if you ran into similar things or not really.

Robert: I would say we ran into the exact same thing. In fact, community contribution I think has been slow. It was really slow in the first year for two reasons: one, it’s a fairly large and complex codebase. It’s not the easiest thing to jump into. Two, for that exact reason. We had developers that were doing everything, so why do I need to do anything?

That has changed, however, I think in the last…I think this last six months or so we’re starting to see much more significant community contribution. Again, MapGuide is more of an application development platform for web mapping than anything else. It’s not really a product you install and start using. You install it. You build an application. You display it to a bunch of users. You refine it. A typical web product, right? So, what we’ve seen is that a number of people have seen limitations with respect to what they can do particularly around the client end of the product. They want more flexibility in the AJAX client. So, they started with some simple submissions – really just adding some little functionality.

But, of late we’ve actually had a solutions organization that got heavily interested in MapGuide early; DM Solutions in Ottawa. Basically, what they did is they actually wrote a new client front end; a new AJAX front end for MapGuide. They are actually now in the process of donating that back to the project. So, that’s a very significant piece of code that grew out of a need. Once they started developing some applications, they thought, “Hey, I’ve got some limitations here. We could probably do something about this with our experience.” So, they started writing some code. Sooner or later they had deployed it a couple of times for a couple of their customers. Eventually they came back to the project and said, “We think this will be useful for everyone, so we’d like to donate this.”

So, I think again it’s a matter of time on those commercial projects that choose to open source. I think eventually the time will come when developers will come onboard. But it takes a little bit of time for that exact reason. Initially, what do we need to do? Don’t know.

Scott: Well, it sounds like you guys are doing it the right way. I mean, the other reason sometimes why companies open source something is just because they want to sunset it to some degree. They’re done with it. They’re tired of it. They don’t want to put people on maintaining it, so they just say, “Here, we open sourced it. Community – go take it from here.” Those ones usually just kind of die pretty much on the spot.

Robert: Well, for us there’s some interesting history here you guys should probably know that I maybe didn’t introduce early enough for you. MapGuide was first released as an Autodesk product in 1996-1997. About three years ago we took on a complete ground-up rewrite. We were about halfway through that effort that we decided to open source.

So, we basically have a legacy product there in MapGuide. We started a complete rewrite project. We actually completed that complete rewrite project. That’s what we open sourced. But, halfway through it we decided the right thing for the future of this product was to actually open source it and we intend to continue maintaining and being a huge part of the development community on that project, but at the same time getting some open source involvement.

Scott: Yeah, that seems to be key. It seems to be key when the company that’s open sourcing it is going to continue to invest in it – that really seems to be a key to success. You know, companies seem to open source for a lot of different reasons. Sometimes it’s because they feel like maybe they’re not in the best position to do a certain piece of work – like if it requires porting to a different platform or architecture – or they feel like the community will have great ideas that they won’t generate internally.

Sometimes companies do it as a market disruptor. They see that the market is saturated with closed source, proprietary solutions, and by being the first ones to open source they can radically change the market. For you guys, what was your motivation in going open source? Why did that make sense for this particular product?

Robert: Well, I’d be lying if I said market disruption wasn’t a huge factor, because it certainly was. What we recognized halfway through this project… I mean, again, it’s basically a piece of web mapping technology. It’s basically a web based GIS platform. So, what we realized about halfway through… We lifted our heads up and looked around. We saw that web-based GIS was growing in popularity by leaps and bounds and ESRI is pretty entrenched. They are a commercial competitor. All of our other competitors have products in this space as well.

Then you’ve got Microsoft. You’ve got Google and Google Maps. You have Yahoo! Maps. There’s a bit of a commoditization around web mapping that we discovered. What we really realized is that the revenue opportunities for us are not in the platform per se, but more in the solutions that we can deliver on the platform. Does that make sense?

Scott: Yeah. So, by open sourcing the platform and being still heavily invested in it, you’re in a great position to provide solutions on top of that platform that might be hard for your competitors to match.

Robert: Right, because we can influence the platform’s direction as well as direction of our solution providers. Again, we’re a company founded in establishing developer partnerships. It’s really partners that develop solutions on our product.

All of our partners now can influence the direction of the open source project by either: a) actively becoming involved in development on the platform; or, b) hiring companies that specialize in open source development to go enhance the platform in some ways that they never could when it was a closed source project.

Scott: So, how does it work? Has open sourcing it disrupted the market and provided a lot of the benefits that you guys were initially hoping it would?

Robert: It’s definitely been disruptive and that’s beneficial. The larger benefit for me, though, is the fact that our solution partners now have an opportunity to influence the new platform in ways they never have before. They get to see things coming early. Typically we release the open source version two or three times a year, so they are getting features faster and they can start to work with them.

On top of that, they can influence things. The idea of this whole AJAX front end is a great example of that. It’s where a solutions development partner for us basically came in and developed a whole new client framework that’s going back into the platform.

Scott: Have you guys at this point had to contend with anybody who has wanted to do a really strong fork of the source? How have you prevented that or dealt with that?

Robert: I don’t see anybody wanting to do a fork. It’s LGPL licensed. That helps a lot in that respect.
My crystal opinion on that, by the way, is that if you have a project that has a good community behind it, that is a cohesive community, I don’t think you’ll see a fork.

Scott: Yeah, it seems like there’s definitely a high bar. Forking is the nuclear option. For somebody to fork and have it get traction, they have to be making at least the same amount of investment in it as you guys are. That would be a pretty high risk thing to take on.

Robert: Right.

Scott: It seems like as long as the company is really meeting the needs of the community and supporting the community in the right way, it doesn’t really happen. But Netscape is an example of where it did and the outcome of that was Firefox. People really felt like the main sponsor was really not taking it in the right direction and they had a better idea.

Robert: Exactly. Let’s be honest – that’s one of the reasons that the project steering committee, who really do have control, is actually four community people and only three employees. We can always veto something because it only takes one vote to veto. But, in my experience so far, working with this committee in the last 18 months, I don’t see us ever actually using a veto.

Scott: Great.

Robert: In communities you can suddenly get a rift with the company that wants to maintain control, or sometimes you get a rift in the development team itself. Drupal is a good example of that. They had a rift in the middle of the development team, and that resulted in the fork of Drupal. I don’t think that’s a common thing.

Scott: Right.

Robert: If you have a good, well-oiled community, I don’t think that’s likely to happen in a company that’s open.

Scott: Great.

Scott: This was fantastic. This was really informative from our perspective. It’s nice to be able to talk to somebody who’s got so much experience in both realms and can provide some objective insight about their experiences on both sides. So, this was really great for me.

Sean: Yes, same here. And Bob, we’ve extended this courtesy to everybody. We’re just curious if there’s anything else you want to get on the record or another question you’d like to propose, or things along those lines before we wrap up?

Robert: No, I think the only thing we really didn’t get to was one of your questions earlier about QA and documentation; I didn’t touch on the documentation piece. I would say that’s one of our biggest weaknesses with open source right now was that we really don’t have a solution around documentation in the community right now.
That’s something that I think will still take time to formulate. Everybody is busy and nobody has time to write documentation. That’s actually my experience as one of the issues with open source projects.

Scott: Unless you’ve got a sponsor who staffs that, it’s just not the most fun stuff to work on.

Robert: Right. So for now, most of the open source documentation has been staffed by Autodesk. But that’s getting harder and harder for us to continue. So, that’s one of the places that I am, if anything, a little concerned about for the future; the documentation and trying to get a way to continue to staff it.

Scott: I don’t think that’s unique. Part of the problem is the people who really, really work on it, don’t really need the documentation. So, they don’t necessarily feel the need as much as the people who aren’t as active in the community itself.

Robert: Right. But the problem is that it’s hard to get new users on board when you don’t have good documentation. That’s really the challenge.

Scott: Right.

Sean: Well, Bob, I really want to thank you for taking the time. I couldn’t agree more that it was a great interview. It was really, really nice perspective we got.

Bookmark this:

Interviewee: Marc Miller

In this interview, we spoke with Marc Miller about his views on the current state of open source software.  Marc works for Advanced Micro Devices (AMD), and in January, Marc took on a role as the open source software evangelist in the AMD Developer Outreach organization enabling Linux kernel and application developers to develop optimized code using both AMD and 3rd party tools and resources. In his role as a software Alliance Manager for AMD 2001-2006, Mr. Miller played a significant role in developing a Linux marketing strategy with a focus on integration of AMD technology with software tools developed by the open source community and industry partners. Throughout his career at AMD, Marc has been a key contact for open source developers wishing to work with AMD, and has been an open source ambassador for AMD, helping to coordinate outbound and inbound communication between AMD and Linux developers.

In this interview Marc talks about:

• Challenges that open source companies have when competing with proprietary software
• Advantages of open source
• Choice also has disadvantages
• Is open-source anti-capitalism?
• The PR value of being open-source
• Open-source is less idealistic and more pragmatic today
• The future is blending open-source and proprietary software

Richard: First off, I have to ask what you think about the idea of Microsoft funding a blog that looks into the differences in delivery model between open source and closed source. Do you think it’s possible that that blog would be unbiased? I mean, we’re doing our best to make it that way, but I’m wondering if you think we can do it.

Marc: There’s a lot of suspicion right now in the open source community. In the aftermath of the Novell-Microsoft pact, it appears that nobody quite understands what Microsoft is up to and what their game plan is. Certainly the community is extremely skeptical about anything that Novell or Microsoft does at this point based on past experiences, for example with the Samba and Mono communities.

That said, while a blog, focused on open source and sponsored by Microsoft would be treated with skepticism by certain segments of the open source community, there would also be people interested in reading it. Everybody wants to understand Microsoft’s angle on Linux.

Richard: I figured that. I figured there’s going to be healthy skepticism.

Sean: When you speak, Marc, about what the angle is, are you asking what Microsoft’s angle is with Novell?

Marc: Yeah, and the way you look at it will depend on who you are. If you’re a large company like AMD that values both their Microsoft and Novell relationships, we’d be looking at that blog for opportunities where our customers can benefit from the marriage of the two.

One of the key things about the Novell and Microsoft pact was Steve Ballmer getting up on a box and saying that this is to protect Novell’s customers from being sued for use of Microsoft patented IP. That has a lot of people very frightened.

For example, people will be looking in those blogs for what the caution areas are. What products might I get burned on if a court decides Microsoft’s claims on software IP are legally defendable?

Richard: That’s interesting. So, with that in mind, I have a few basic open source questions that I hope you’re willing to address and give us your opinion. One is, what are the challenges an open source company faces when competing with closed source companies?

Marc: Very often the open source companies are trying to profit from something that’s not the source code itself.

Take a look at someone like Canonical. All of their software development efforts have been to get Ubuntu into the hands of as many users as possible. They make Ubuntu Linux available at no charge, under open source licensing mechanisms. Their profit model is currently based on support services. For someone like Canonical it’s a question of does the open source community develop products where the user might be willing to pay something for a service like support?

Richard: So, it seems to me that the biggest asset a software company has is the source code. What traditionally has been called trade secrets, intellectual property, or secret sauce.

Marc: In the closed source case, yes.

Richard: It’s an interesting idea to make money by giving your main product away.

Marc: The GPL completely allows for an open source program to be sold for profit. But it also allows multiple copies of that to be made and redistributed without royalties. So, it’s a real interesting area. Since it has been difficult to turn a profit from just selling a GPL application, traditionally most companies that back open source projects use the software itself as an enabler for whatever service or subscription that they plan to sell. In some cases these companies also sell a closed source product that compliments the open source projects.

Richard: So, what do you think are the most compelling advantages to open source over closed source? What is open source able to do much better than closed source can do, in your opinion?

Marc: Open source allows for freedom of innovation in the absence of a driving business case in a lot of cases. Take the Debian community, which has emphatically refused to adopt the same way of doing AMD64 architecture support the way that other Linux distributions have. To give you the ten-second synopsis, other Linux distributions support running 32-bit and 64-bit code in the same environment. Debian felt the way the directory structure was handled in other AMD64-supporting distributions was the wrong approach, and they felt they needed to adopt a structure that would prepare Debian Linux for future innovations. So they elected to support an approach that was different from what other Linux distributions were taking and refused the quick fix implemented for business reasons. I use this example to illustrate that, in the open source community, everybody has the freedom to innovate according to their own interest and ethics, business justification or not.  I respect the Debian community for holding to their values. In the closed source space it’s a much more restricted audience; the freedom to do what a particular developer feels is the right thing just isn’t there. People are still confined by ethics, but those can be overridden by corporate objectives and customer requirements.

For example, products sometimes are shipped before they’re truly ready just to stay on schedule. Debian is famous, if not notorious, for delaying a release until the software is really ready and no known critical bugs exist. If a bug is missed prior to release, open source methods allow opportunities for the community to fix the bug after the fact.

In open source software, anyone can disagree with a corporate direction and then redistribute their own version which has things written the way that they think it ought to be. This promotes a freedom to innovate that goes beyond what you can often find in the closed source model.

Richard: How does open source keep from having endless slightly different distros?  Is it Darwinian? The things that really work and get picked up live on, and those that don’t just sort of peter out?

Marc: The biggest strength and the biggest weakness of open source is the freedom of choice. Michael Dell has been quoted in the press prior to the May 1st announcement that they were going to start shipping Ubuntu. Each time he’s quoted about responding to when are you going to start shipping Linux, when are you going start supporting the community, he says, "I’m fine with supporting the community. Which one? Which one do I pick?"

Every Linux distribution has a different philosophy. Look at Red Hat; they have had a very long history of preserving backward compatibility and ensuring stability of the products such that when their enterprise customer upgrades, they’re not going to have to recompile their kernel and reinstall a bunch of apps.

Look at the SUSE case; they try to take a much more balanced approach to include innovation in cases where it might be a minor inconvenience, but they believe that the customer will appreciate the value that they get out of the product for that minor inconvenience.

Now look at Debian; I once suggested that Debian try to encourage enterprise ISVs like Oracle to integrate their software with Debian Linux as a way to expand their visibility and market share. A Debian developer responded, "Who cares about market share? What we care about is open source." Everyone on that discussion list thread agreed there’s no reason to make Debian more compatible with Oracle databases because Oracle isn’t open source. Debian’s mission is an open source revolution across all software segments.

So each Linux distribution, besides having a different code base, has a different philosophy behind it. And that creates a lot of choices for the user and that’s a big strength, but that choice complicates the decision too. There are two main product lines in the Windows release – a desktop/workstation line, and a server line of products, and one can even argue that those two product lines are more similar than dissimilar as they leverage many of the same operating system components. There are 359 open source operating system distributions according to Distrowatch, most of which are Linux.

Richard: It seems to me in doing research on this subject, I run into an awful lot of open source advocates who seem to be open source advocates more because of the anti-capitalism sort of flavor to it, at least in some corners of open source.

That seems to be a common voice, although it doesn’t seem to be a common motivation of all of the open source projects I’ve looked at. It looks like there’s an awful lot of people writing open source trying to figure out a way to make money on it. They believe in open source for maybe collaborative reasons, maybe for development cost reasons or whatever, but they want to make a profit.

But this anti-capitalist message seems to be sort of pervasive in one from or another through almost everything I’ve read. Do you think that that hurts the open source community in the eyes of people who are agnostic in terms of development paradigm? Do you think some people are turned off to open source because they see it as sort of anti-business, and they aren’t?

Marc: The attitudes of the open source community have changed over the years. If you look at the Linux community of the late ’90s, you’ll find that’s very much the case, to the point when we launched x86-64.org I asked that all of the corporate logos be removed from the website. Since then we’ve brought them back because the open source developers realized that without a corporate motivation and corporate sponsors that there’s a lot that they’re prevented from doing. Many open source applications are hosted and mirrored on either boxes in a corporation’s data center or using corporate sponsorship money. Linux Symposium wouldn’t happen without corporate sponsorship. Linux-supporting companies look after the interests of the open source communities in standards bodies and industry forums that developers don’t usually have access to.

Also, a lot of corporate sponsors pay developers to do what they would do anyway. And those developers have become grateful to those corporations in a way that has lessened the anti-capitalism trend.

Sean: Let me ask you a follow-up to that, and I know exactly why Richard called it an anti-capitalism bent, but do you see that as the most appropriate phrasing for it? I mean that’s kind of the quick draw phrase that everybody uses, but is it anti-capitalism or is it anti something else?

Marc: I’d say it started off as anti-capitalism because the feeling was that most corporations A) produced strictly closed source software and B) didn’t understand the open source model well enough to be any good to what people like Richard Stallman were trying to promote. So, it was an anti-capitalism setup.

But as I say, modern-day, most of these well known developers are sponsored by a business. So, a lot of that trend has died down.

Sean: So what do you think it’s kind of evolved into? Is there still kind of an anti- bent to the whole community? I mean is it kind of anti-control, or anti-throttling of innovation, or what would you stick there?

Marc: The Linux community is a lot more pragmatic about software IP than they were 15 years ago but long term, open source enthusiasts still are driving toward elimination of software as a proprietary product. It’s a freedom of knowledge movement, and much in the same way that you can go to Wikipedia and read about any topic that someone has contributed knowledge to. In the software case, that freedom of information is creating software instead of encyclopedia articles, and the content creates a new market. You can still make money off software creations, without making money from the software itself. Perhaps that’s services for the software, perhaps it’s a subscription to access premium content.

Open source enthusiasts feel that they are resisting controls over innovation and strive to obtain freedom to innovate in a way that’s beneficial to the customer, regardless of business case. It’s not anti-business but does challenge many existing business models.

Richard: In other soft product areas, like music, publishing, and stuff like that, you have a war between the copyright holders and those who think that stuff should be free. Do you think that closed source companies have a right to protect their source, or do you think somehow they’re subverting the greater good?

Marc: My opinion is that they have every right to do with their products whatever they need to do. If the software author also supports an open source movement, that’s a great thing but the open source model is not for everyone. In Eric Raymond’s "Cathedral and the Bazaar," there’s a whole section about reasons to open source, and that if your product does not meet all those criteria, then maybe you are better off keeping it closed. Often a solution stack is incomplete without certain pieces of software that happen to be proprietary.

For example, people buy a solution stack that includes Oracle and SAP because they derive value from the closed source applications, not because it’s built on top of an open source infrastructure. The open source operating systems (such as Linux) offer great features too, but today, most OS licensing and services revenue is driven by the application workload inherent in the solution stack the OS was bundled with, not necessarily the OS or benefits of open source.

Richard: So, you don’t think open source is in opposition to closed source, but rather just a different paradigm?

Marc: Yes.

Richard: OK. That’s cool; that’s interesting to know. I was trying to run down project-development leads on OpenOffice.org, and I think a little bit over half of them are employed by Sun. I’m curious, speaking non-altruistically, what does a company gain by sponsoring open source?

Marc: Besides the points I’ve already covered, they gain favoritism as an open-source-friendly company, that developers will even develop for their closed products because they believe they are contributing to the greater-good effort.

Richard: So, it’s sort of developer PR at some level? I’m sure not all of it, but there’s some, "We’re good guys, look, we’re sponsoring."

Marc: From a business perspective, that’s right – that’s what many of them see. The ability to leverage innovations from the community is also attractive, but the contributions aren’t guaranteed in the same way that the developer PR is.

Richard: That’s interesting.

Sean: For about seven years I’ve been involved in deep, technical efforts around evangelism, and I’ve certainly seen closed source companies realize and economic gain from sponsoring and supporting a community.

For a company that’s got a mainly closed source model, and they embrace open source model on the side, do you have any kind of anecdotes or examples of places where the company looked back and saw gain?

Marc: Probably the closest thing that I can think of is Adobe’s decision to allow projects like OpenOffice.org to include PDF generating tools, and their decision to make PDF a completely open specification that they encourage other people to put into their products. This has benefited Adobe’s PDF software products, even though Adobe’s products are closed. So, by creating an open standard, they’ve enlarged the market for themselves, and by supporting open source efforts, they’ve gotten a lot of mileage in the market that they sell into.

Richard: Last week Adobe open-sourced Flex, which was a free product. What, do you suppose, they gain by open sourcing it? Do they gain lower development costs, because they’re going to get free collaboration?

Marc: Well, I’m not very familiar with Flex, but suppose that they did the same thing with Acrobat Reader, which is another free product that they give away. They have already developed the market to the point that PDF is ubiquitous. If you want to make a document that is guaranteed to be readable on just about any platform out there, whether it’s x86, Apple, or a smart phone, you generate it in PDF.

If they decided to make an open source Acrobat Reader, that allows a lot of projects to tap into it and borrow code from it; it promotes the whole integration story of plugins and addons that go beyond the original plugin interface for the application. By giving their developer community the freedom to innovate, the development direction would be managed by the community.

You asked about lower development costs. The cost only goes down measurably when the community contributes something that the “copyleft” holder was going to pay for. Open source software lets a lot more innovators in the door to help with software development, but sometimes additional resources have to be found to do the work that no one contributes for free. The point that the open source community is trying to make is that open source innovators are shared among many open source projects. It would be expensive to recruit each of those developers under contract, and even more expensive to maintain that software over a long period of time.

Indeed, many projects have benefited from the open source model. Open source is not a guarantee of success any more than having great technology. There has to be demand for it, and from the right audience or no one will show up to do the work.

Richard: I see. A couple weeks ago, I ran into the CSI company that was started by Stuart Cohen. It’s sort of open source. The paradigm is that CSI hires developers, and managers as part of a geographically centered development teams. Companies who want a particular piece of software pay for CSI to manage that effort.

The idea is you might have half a dozen companies who want to have some software package. They pay CSI to develop it, and they also may make some of the developers on their staff available to the management of CSI to contribute to that project.

However, the thing that interested me the most is that was that once the software package is developed, it may not be re-licensed as open source. Rather, it’s going to be the decision of the contributing companies how to re-license, or even whether to re-license it. The presumption, I guess, is that they’ll go ahead and let it be available through some sort of open source model, where it can be modified and redistributed without cost, but they don’t have to.

It looks like a blending of open source and closed source. You’ve got this shared cost and collaboration of open source, but you don’t necessarily have the other main underpinning of open source, which is that the source code is available to everybody.

Do you think that that’s where we’re going, into different sorts of paradigms that blend proprietary and open source methodologies, and do you think that’s good or bad?

Marc: Though they would prefer everything to be open source, open source developers are beginning to accept that there are certain things that are difficult for a company to expose without getting them into potential legal trouble. They don’t like it, but they’re understanding. Overall, I think it’s a good thing.

Richard: So, it sounds like open source is drifting away from highly idealistic to more pragmatic; still holding onto their ideals, but being more pragmatic about attainability in certain situations.

Marc: There are some exceptions, but for the bulk of the open source community, yes, they’re becoming a lot more pragmatic about things. They still prefer to hold onto certain ideals, and continue to direct everything in that direction, but most developers are fairly pragmatic about it.

Richard: So, I had a couple of interviews with some guys at Microsoft focused on ensuring security for Microsoft products: Michael Howard and James Whittaker. Both of them, separately, said that the biggest advantage that closed source has, in almost every area—architecture, implementation, quality assurance, and documentation—is the ability to mandate a process. Microsoft has the ability to say, "This is part of the job, and you have to do it," whether it’s unit testing, adhering to coding standards, security reviews, or whatever. The implication was that would be hard to put together with open source because all you have is the code, you aren’t certain of the steps the developer went through to produce that code.

Is there good adherence to process in open source projects? Is it something that’s improving? Can you comment on that area, in general?

Marc: The open source community has code control procedures and models of their own. For example, in the LinuxBIOS/OpenBIOS project, you can submit a patch to the mailing list only once you’ve gotten one of the key maintainers to review and approve the code. So, there are still very much the rules for developing that code and processes to be followed. Some of these methods probably emerged from source code control models originally used in closed source settings, but in the spirit of open source, why change or reinvent what’s already working well?

The rules for submission seem to vary a lot for each project. You can have quite a range of different quality control mechanisms. But, usually, there is a quality control mechanism there.

Richard: Yes, there has to be. You can’t release software unless you have some kind of quality control. It seems like you have to have some sort of gatekeeper over the code base, or it’s going to deteriorate in a hurry.

Marc: Without someone moderating the process, you end up with wars of someone changing a bit of code, and another person changing it back. Endless back and forth. Code control also guides the direction of development so that community goals can be met.

Richard: Yes. So, what to you see on the horizon? What do you think is in the near future, generally, in open source?

Marc: A lot more of, as you put it, the blending areas particularly as Linux becomes more widely used in different server and desktop environments. For example, in multi media areas we’re seeing a lot of companies that create set top boxes that use a combination of closed source and open source software in embedded devices. It’s an area where businesses are realizing there’s money to be made with the combination of open source software and closed source IP.

Virtualization is a big area of expansion. A lot of companies want to make money off of virtualization, sometimes for security, and sometimes it’s to increase the range of compatibility across operating system environments. There are both closed and open source options, such as KVM, VMWare, and Xen. Each innovates using different philosophies. So, this is an area that I’m watching with interest.

Device drivers are also an area of much attention. The Linux kernel community has been very emphatic that they will not accept any closed source code into the Linux kernel. And yet the way that you develop drivers for Linux is to include it in the kernel.  For example, the FCC has given 802.11 wireless networking devices a certain range of frequencies that may be used. Many of those devices are essentially programmable radios. The hardware vendors don’t want to open source that because that allows someone to then misuse the product and gain access to frequencies that the FCC wouldn’t approve of. Sometimes in order to reveal specifications, licensed IP used in the hardware design has to be revealed.

The drivers I’ve just described can never be part of the Linux kernel, and yet Linux users want to be able to use all these devices. In recent months, just really January of this year, certain key people in the Linux community are starting to realize that this prevents a lot of hardware companies from participating in the way that they would like to, and that though hardware vendors have nothing against open source models, for one reason or another they simply cannot release their code. I’ve been watching a number of efforts to make it easier for hardware developers to produce Linux drivers. I hope this topic is discussed during the next USENIX Kernel Summit and that the maintainers find a way to prevent Linux from being stifled further on the basis of compatibility problems with new technologies.

Richard: Do you see Linux making inroads into the desktop market? Do you think that they can gain a significant part of that?

Marc: Client computing models are changing from traditional desktop and laptops, and with these changes there are new opportunities for Linux. Different types of desktop applications or use models also offer an opportunity for Linux.

I think in time we will see Linux emerge in the same way – having strengths in certain types of client devices. I don’t know what that strength is going to be yet for Linux but many people in the community are exploring what it should be. Today, most ISVs only want to deal with one desktop operating system, and that’s Windows.

Today there are a number of ways, including virtualization, that you can run Windows apps in a Linux environment. As that improves, I think we’ll see a lot of ISVs soften to that idea. And, as they look at some of the things that we’re developing for Linux, they may decide that it’s better to develop a Linux app that will also run under Windows than it is the other way around.

As the Novell-Microsoft Interoperability Lab continues to grow their efforts I can definitely envision Microsoft having to run Linux apps inside their environments much the same way that we have those today in Linux.

Richard: It seems to me that products like OpenOffice.org and other open source products that are becoming strong, mature applications in the traditional desktop applications space have to work to mitigate Microsoft’s grip on that market. In the past, when Windows first started to take over the desktop, it was because that’s where all the applications were. You could put Linux or something else on there, OS2 or whatever, but all the mature applications were Microsoft applications, and they had to run on Windows. And I think that’s sort of how they got that grip on the desktop market. It seems like the first step toward making that a more competitive market is through the applications phase.

Marc: Yeah, I agree with that.

Scott: It looks like the TCO battle has been fought to a stalemate.  Microsoft has Microsoft sponsored reports saying they have lower TCO.  On the Linux side, there are reports sponsored by Linux supporters saying that Linux has lower TCO.  Do the communities find any of these reports credible?  Have you come across any truly independent TCO reports that you’ve thought were especially good?  Is it even reasonable to expect that the finding of some TCO report will be borne out in the specifics of your organization?

Marc: In the independent studies I’ve read, per machine, the costs are pretty much the same for a supported machine if you compare Windows to a truly equivalent Linux implementation, but this fails to take into account the workload processing power costs. In the open source model, I have visibility into what code is slowing down my e.g. mail server, exclude the things I don’t need to have running, and make the machine more efficient. Without that optimization, more machines would be required for the same workload regardless of the operating system in use, which adds to the cost.

Richard: Marc.  Thanks for taking the time to chat with us.

Marc: Thank you.

Bookmark this: