Interviewee: Justin Erenkrantz

In this second part of a two part interview with Justin Erenkrantz we talked to him about:

• How the Apache project ensures good collaboration.
• The Apache Foundation’s philosphy of having no single person as the leader.
• Apache’s security committee.
• The process of removing someone from a position of responsibility within the Apache Foundation.
• What would make someone want to be part of the Apache Foundation’s group of projects.

Scott Swigart:  In this part of the interview, we wanted to dig into some of the tenets, if you would call it that, of the Apache way. And the first of those of course is collaborative software development.

So talk a little bit, if you would, about how Apache does collaborative software development. I’m sure some things are very traditional and similar to the way that other open source projects might do it, and there are probably things that also might just be a little bit unique to Apache. So how do you try to insure good collaboration?

Justin Erenkrantz:  So the main center point of all of the collaborative software development that we do in Apache is the mailing list. That’s where pretty much everything happens. As one of the guys mentioned before, the maxim has been: "If it didn’t happen on the mailing list, it didn’t happen." And that generally tends to be true.

Basically, if you follow the mailing list for a particular project, then we’re expecting that you should know what’s going on within the project. Within that, those are pretty much all public lists. Everybody can subscribe, even people who are committers, people who are just users, people who just work on another project that may consume the Web server or maybe PHP modules or something like that.

It’s pretty much an open forum. Anybody can voice their ideas. Generally though, just the way things work, most of the traffic tends to be from the core developers who are active at that time. The traffic patterns of the list change, and you get an idea of how much discussion. You generally see peaks and valleys for the mailing list discussion. Things get really heated or things are just chugging along and there’s not much traffic on the list.

So that’s where all the discussion should happen. And then of course there is the source code repository. And in Apache we always had the thought of having shared repository. A lot of projects now are starting to ‑‑ Git and Mercurial and all of these distributed version control systems to be centralized version control systems.

And that’s something that, within Apache, that goes against what our thoughts are, because we want to all be agreeing on, "This is the Apache version."

Scott:  Sure, no problem.

Justin:  So there’s no leader within the ASF. There’s no person, if you look at say, Linux and you say, "This is Linux’ tree," or this is Andrew’s tree or this is Alan’s tree. Instead there is just the Apache tree. So there’s really no concept of, "This is Justin’s tree," or someone else.

Scott:  Is the feeling then that essentially there should just be intensive discussion before something gets checked in? So when you’ve reached a point of consensus, it should get checked in rather than the way other projects work where different things get checked into different people’s trees. And then when it’s time to build a release, you have to pull stuff from these different sources to figure out what’s going to be in the release and what isn’t.

Justin:  Right. So, generally, what tends to happen is there are two states a tree can be in within Apache. One of them is "commit then review." And then there is "review then commit." And you’ll see some projects differ on particular trees.

For example for the HTTP Server, the trunk ‑‑ which is the main development ‑‑ is usually always under the "commit then review," which basically means that anybody who has commit access can feel free to go and make changes and they basically have the benefit of the doubt that the change is going to be good. And there’s generally an implied threshold: that if you’re going to make a really big change, go discuss it on the list. But if it’s a minor change or adding a little feature, that’s probably not going to be controversial, go ahead and commit that to trunk.

But for our stable releases, generally things that have already been released and we’re doing maintenance on those, are under the "review then commit" model. So that’s going to be RTC, and that means that any change to those trees has to be pre‑approved. That means you need to get three binding votes from other committers to say, "Yes, this is good change and no one has vetoed it." Technically you would use a file called STATUS, just a plain text file that some projects will use, that basically tracks all of the things that are under discussion to be back‑ported or added into this tree.

Scott:  Got you. Like any open source project, and this one is democratic, there is a vote to commit. Well, there is a vote if it is a release product. If it’s not a release product and people later decide it wasn’t a good thing to do, it can be reverted out.

And you mentioned before that part of your governance is that out of these 60‑something maintainers, one of them can essentially veto a change if they want to. So talk a little bit about how conflict resolution usually works. For example, when you have people who are—and I understand it doesn’t happen often—adamant one way and another person who’s adamant a different way. How do you see it play out that those eventually get resolved and things move forward?

Justin:  Generally what happens is, like before, the veto just tends to be a last resort. So let’s just say that someone makes a change to the trunk and I don’t like it. I might just say, "You know, we should talk about this change," or, "Here’s the problem with this." Generally, the person who has committed that says, "Oh, yeah. Here’s why I did it this way," and comes up with an explanation, and then through a process on the mailing list, figures out and resolves those conflicts.

That’s what you tend to see happen. And it’s all tends to be, for the most part, "I’m sorry, I forgot about that particular corner case or that." And everything tends to get resolved very naturally.

The veto tends to be when someone says, "No. I have to do it this way," and someone else says, "No, that’s wrong." That’s basically when the community is at risk of breaking down. But generally it doesn’t get to that point. Everybody is, "We’re all going to go in this direction; this is the right direction for us to go in. We want to add this feature. Let’s work through whatever issue you may have about this particular commit."

Scott:  One last thing before we move on to the next point. Other than the fact that everybody has their own private tree, is there anything else about the collaborative nature that you think is somewhat unique to Apache?

Justin:  We tend to do a roll call before the release. So at this point there’s been a review of everything. But then let’s just say that I want to release the next Apache 2.4 or whatever. Then basically what I will do as a release manager is say, "OK, I’m marking this as 2.4," and I produce all the artifacts. I produce the tarballs, [inaudible]generated files, whatever. And then I send it to the list and say, "Hey, is everybody happy with that?" And then that goes to a voting process.

There’s review at all stages, but there’s also a review at the point where you do a release, and you have to get, at least, three people to approve a release. One thing that’s different is that it’s not possible to veto a release. It’s strictly majority rule on the release.

Scott:  So in other words, there is veto capability on the individual check‑ins. But, as you said, when it comes down to doing a release it is a majority rule vote.

Justin:  Yes. You’ll tend to see someone voting ‘no’ on a release if it doesn’t work on Linux or something, and generally you’ll see it get stalled and it then gets fixed. But there have been a couple of cases when we did the release even though we knew that it didn’t work on a particular platform, and so we made a release note. But the veto does not apply to releases.

Scott:  Interesting. So moving on to licensing, one of the key things with Apache is the commercial‑friendly standard license. Talk a little bit about what that means.

Justin:  Basically within Apache, we like to have a big tent where everybody can come in and play with us. We think that the community that we developed within Apache is going to be the motivation for you to stay involved. For example, within the HTTP Server community, you have all these experts and Web servers, and if you’re part of this community, you get the benefits from them. And so there’s an incentive to play within the community, so that I don’t have to hire five guys and do a whole team; I can leverage the other people within the community.

But in turn, all those people who are part of the community say, "Whatever you want to do with the code is fine. We’re not going to get hung up if you make it a commercial product or an open source project. We created it and it served our needs, and if it serves your needs, that’s great."

Scott:  What springs to mind are things like GPL. Am I seeing it right in that Apache is more commercial‑friendly than GPL, V2 or V3?

Justin:  There are companies built around GPL licensed software. But what we tend to see are two classes of GPL products: In one, there is a real community, maybe within Linux, and they’re all happy to make all their changes available to everybody, and that’s a very good community. The other community you tend to see has a single stakeholder that has a prevailing interest in the GPL product, and they basically have an unfair share.

You can see this with some of the GPL projects that require copyright assignment. In order to participate, you license your changes in the GPL and you have to give a copyright assignment to the principal stakeholder. Now they are then free to release the commercial closed source based on your work because they have the copyright or whatever legal mechanism. There is an imbalance there when you look at those two.

Generally when you think about the GPL, you’re divided, with broad strokes, into those two groups. This is a real community but the other groups are aware and want to be clear about which one has a dominant role, and that’s one thing within Apache we don’t like to see. As our projects go through incubation and get added to the Foundation, one of the things we do is make sure that the community is diverse. In fact, there is not a single dominant stakeholder that can direct the project in any untoward way.

Scott:  Right. I can make changes to it, distribute it as part of a commercial product, and I would not be required to contribute those changes back to Apache. But under a GPL license, any modifications made require you to make the source code available. You cannot have closed source proprietary extensions or modifications of it. Any modifications you make, you have to open source and it has to be under the same license. So that’s the key differentiator?

Justin:  Yeah, our philosophy is that the community is what’s going to bring you and keep you there, and that’s why you’re going to stick around. If you released a commercial product around one of our projects it’s going to be to your benefit, to basically keep your commercial project as close to whatever we’re releasing.

Scott:  Right.

Justin:  You can pick up all the bug fixes and whatever improvements; you get those as a free rider. But in a sense, you are contributing whatever changes you’re making voluntarily back into the greater community.

Scott:  Yeah, that makes sense. Do you have examples of companies that have used different Apache projects because of the commercial‑friendly licensing, where they probably wouldn’t have it if the license weren’t so commercial‑friendly? Is that a topic that comes up?

Justin:  Absolutely. You see companies like IBM that release their versions of the Apache Web server or Geronimo under different names, but in the core, they are Apache projects. We’ll see that even with smaller companies such as Covalent that does commercial support. Basically, they added in a couple of extra things that provide support to their users.

One thing that the Apache community really does not focus on providing is 7/24 support. Covalent goes in with their business model and provides the support and training around these particular Apache projects. You will see businesses like JBoss using Tomcat. So you see all of these commercial companies using things that are Apache projects under the covers.

Scott:  Right, so that freedom has led people to be a lot more creative about how they structure their business. They have a lot more options in how they participate with the different Apache projects, how they contribute back and how they structure their own products. What is the relationship between the Apache Software Foundation and the Free Software Foundation? Is there any or are those fairly separate endeavors?

Justin:  There is no formal relationship. I’ve never had a conversation with Richard Stallman, but I’ve had conversations with Bradley Kuhn who used to be, at that time, the Executive Director of the Free Software Foundation. So there’s an informal get‑together of foundations to compare notes, and that’s generally a very good thing. How do we keep our ears open to what Mozilla’s doing? If they’re doing this new technique, then we can give them a call and ask, "What are you doing? We’d like to follow on it."

One thing we’ve been doing with the Eclipse is a joint conference. There’s going to be a conference in Asia that’s now scheduled for 2008. So it’s a way for us to get the communities talking to each other.

Scott:  Sure. So basically, if I can summarize, you guys get together around joint events and joint things where it makes sense. You share information because you’re all part of the open source community. Philosophically you may agree to disagree in terms of the details of licensing, commercial friendly, and that kind of stuff.

Justin:  Well and you ought to be using more…projects have different circumstances. Apache‑‑we have a very vocal membership and we have this and you compare that to let’s say the Mozilla which has a completely different governance structure. But if you look at Brian Behlendorf, he’s been on the Mozilla board for a very long time and he was one of the founders of Apache.

Scott:  Gotcha.

Justin:  So you have this intermingling of the communities. So someone like Brian Behlendorf who was brought in through the Mozilla and says here’s how we did things within Apache, and here’s his expertise and his experience that he got, he can share that with the other people within Mozilla.

Scott:  Gotcha, gotcha.

Sean:  Let me refer to one of the other tenants of the Apache way. I’m curious about this just because I was thinking about the conversation we were having. To state the obvious, you’re focused on producing software. I notice that you have a security committee that—if I’m reading it right and for lack of a better phrase—provides a service to all of the projects that are part of the foundation. And it looks like those projects can turn to the security committee and ask security related questions or possibly look for guidance from them, regardless of whether they’re Tomcat or some other piece of the foundation? Is that accurate or is that not accurate?

Justin:  It’s somewhat so. We have a security team, which I believe is currently a Board committee. But basically what they’re responsible for doing is ensuring our security at Apache.org mailing address gets responded to. And these are generally people who are very security savvy.

But there tends to be some people from Tomcat, from the HTTP Web server, from a higher profile project on this internal mailing list. So let’s say that, to give you an example, let’s say there was a security vulnerability in Derby and they could parse to those reps and say, "Hey, we have a security vulnerability. What do we do?" And so there’s expertise and, "OK. Here’s what you do. Here is your administrative contact. Make sure your mailing list… Go talk to…" Kind of a shared resource. But we’re not getting the focus on producing the fixes for the project but it’ll be "OK, here’s the responsible disclosure policy and an attribution policy." So that’s generally what their role is.

Sean:   Are they providing fairly prescriptive guidance but just not down to the ‘I’m going to change you’re code’ level because they don’t know the individual projects at that level? Would that group essentially be the center for discussions around a security development lifecycle for the Apache Software Foundation? And an attempt to pull those best practices together?

Justin:  Yeah. I think basically our project concern…we have something. What’s the process? What do we do? And that’s as an advisory role. OK, here’s the process and the procedures to follow.

Sean:  But it’s purely advisory, right? I mean one of the projects where they feel that their code is "secure enough", or they’ve looked at it long enough or they feel that they’ve handled it.  Then the advisory committee comes back and says, "Well, we really think you could take a look at this again." That’s where the communication would stop and it would be up to the individual project whether they want to take that under advisement or not.

Justin:  Yeah. I think so. I think record security, can you maybe at that point write back to your original reporter and say, "We looked at it and we don’t feel there’s a security vulnerability here." That may be…that has happened where we look at things and we say, "No. This is not an issue." But generally, really the security team is more of a reactive. So they’re not proactively performing security analysis on our code or anything like that.

Scott:  I just want to clarify. It sounds like they have a little bit of an all‑up policy for somebody sending an email to that address; somebody reports what they perceive as vulnerability or reports some kind of issue. They do a little air traffic control. They route it to the project.

Justin:  Exactly, exactly.

Scott:  There’s a general process that the different projects would follow. Basically that sort of happens and that’s one of the things that the security group advises the other projects on. Well this is generally a ‘way we do it’ sort of thing.

Sean:  Let’s go to a different piece of the Apache way, the emphasis on a technical‑based interaction. One of the things that I find fascinating about open‑source projects is the way that they exorcise community members that maybe aren’t following those rules.

[laughter]

Sean:  Because we got some interesting responses when we talked to people about it. It’s like, fine. We understand that everybody is an adult. We understand that everybody will try to handle themselves in an appropriate manner. But if anybody’s worked on a software project they know that not everybody does, right? So…

[laughter]

Sean:  Considering that you can learn a lot from a story…if you’ve got a story or two about either fully pulling the ejection handle on somebody that would be interesting to hear? Or a scenario where it just took serious counseling to get somebody pointed in the right direction.

I would be curious to see how you guys handle that. You obviously have procedures in place. But at times you have to go beyond those with some amount of intervention and I’m just curious how that played out.

Justin:  Yeah. So there’s one case that comes into mind but I’m trying to reserve the right to figure out how much of this has been disclosed.

Sean:  Yeah, Sure

Justin:  So I’ll tell the story and then leave people’s names out of it but I have to go back and see how much of this has been told. So recently within one of our larger, well‑known projects, there was a bout, to use the word, between two committers. And they basically ended up vetoing each other on everything. It’s like no, no, no and tempers got flared. And it got into a very unhealthy situation. They’re two very strong‑willed individuals.

And basically what happened is the PMC, so it was the PMC responsible for this particular activity, basically had to step in and say, "OK; we need to come up with some policy or come up with some new rules to get everybody back to ground zero. So it wasn’t a matter of ejecting anybody. It was never really an option that…basically what happened was they said, "Here are the ground rules. Here is…if you’re going to go do this you have to follow this set of rules. If you’re going to go do that, you’ve got to go follow this set of rules." Basically the community agreed to say we’re going to go and we’re going to voluntarily adopt these rules. But as a settlement process; lots of flames and a lot of innocent people getting…

Sean:  Right.

Justin:  …accused of things and that process. The other one that’s in our history was a project called Avalon. And this is one that’s definitely well known so this won’t be any issue about this one. Avalon was a container framework. And what happened was two individuals just did not get along. And they were ending up in what we would call a commit war, where they would basically be reverting each other’s changes as soon as they came in. And it is just this whole really poisonous environment and basically in that case, the community wasn’t able to deal with it. And so basically what happened there was they fractured.

And so that one of the individuals went off and he took his code and you know, we wished him luck and said have a nice life and he went off and then it was kind of some other people came along and they did a project called Excalibur, which was basically the remnants of this whole Avalon project. You will just see if you look at the mailing list traffic, you will just see this giant peak and then this sudden nothingness because the project got shut down because no one could play well with each other.

Sean:  Right.

Justin:  By the way what it is interesting is some of the veterans of Avalon, they really got involved in the greater Apache community, one of our new directors this year. He was in the middle of all of this, but during that whole experience, he was one of the people trying to keep things level and stuck around and this year he got elected to the board of directors.

Sean:  Since Apache is a large foundation, I’m curious about a different point. If you were a closed source company and you feel you are short on testers or short of security experts, you simply go to HR and put out a requisition and hopefully some good folks come back.

So has the foundation had to answer requests from projects where they say, "OK, look, we think we are geniuses on nine out of 10 of the things we need to do, but this one thing we really need people to help." How does the foundation help with staffing up a project in this type of case?

Justin:  It is more bottom‑up than that I think in the sense of the culture that we have. You will see some overlap between the HTTP Server committers and the Tomcat committers because you will see that ‑‑ sometimes the Tomcat committers came over and they say, "Hey, we need some help with HTTP." Well lucky us, we have some of the world’s foremost experts in HTTP server, and that basically got them within the communities.

So I think our community’s diverse enough to, "Hey I am looking for a person who knows SQL." OK, I am going to go on the Derby mailing list and say, "Hey, I need some help with SQL" or for something for build systems, I’ll go to Ant community. And there will be some of the people who are the foremost experts in that. That’s actually one thing of having such a large diverse community is that you can pretty much find someone who understands something about something somewhere within the Foundation.

Sean:   I figure that’s one of the advantages. You’ve got a massive talent base but at the same time it is segmented into the project, so the Foundation can help orient a little bit of that knowledge of where the talent base is.

Justin:  Yeah and as I said if you look out the social graph and it is a weird mix of people who are committers on Cocoon, maybe committers on Mina, then maybe on Gump and so you will see that developers themselves, the committers aren’t necessarily staying in their silo, there are some who do, but there are also just as many who will go to other communities and work on other projects.

Sean:   Well I have couple more questions but Scott can go ahead. I want to give you an opportunity to jump back in.

Scott:  So talk a little bit about standards because one of the other tenets or pieces of philosophy is faithful implementation of standards. Talk a little bit about what that means for Apache?

Justin:  Right. So this was initially when we started off there was HTTP and there was the IETF standards and that was when you have editor of the HTTP standard is one of the people behind the code base, there is the knowledge is going both ways in a sense is that we are that able to influence the standards process. But at the same point we also have some of us involved with the standard process and feeding those changes back and to the development and supporting those standards.

But since then, the initial, you see our participation within some of the key web server specifications, then probably most importantly our participation in the Java Community Process and that is, as you know, we have so many Java projects and that so many of our projects are implementing some JSR specification and our involvement within the JCP has been to ensure that we can implement the specifications and we have projects have representation on these expert groups that device these standards.

Scott:  Right. So it isn’t just like the standard shows up and then you figure out how to implement it. There is this two‑way street where you are shaping the standard because you guys have such a big, real world implementation. And meanwhile, the standard is telling what you guys do because they have their own stakeholders, but they are considering your recommendations and you want to conform to what they eventually approve.

Justin:  Correct.

Scott:  Yeah, I don’t know. I don’t have anything else specific. Sean, do you?

Sean:  No, not right now. I think this led us into some new stuff and from our end, we really enjoyed chatting about it. Justin, do you have anything you’d want to add or things you think we should address overall based on the theme of where we were going?

Justin:  No, I mean you did a good job of asking me the questions.

Scott:  I guess there is one final question. When somebody is starting or has an open source project, they can pick the license they want  they can do what they want for their community, things like that. What makes people want to be, in your mind, an Apache project? What is the draw I guess?

Justin:  Well, I think from my perspective, the draw is we handle a lot of the mundane governing structures and all this and the infrastructure and the licenses. And that is all essentially managed and I think you see a lot of open source projects like, "Oh we need to go get a foundation."

And that’s a lot of overhead, there is a lot of overhead to create a corporation, handle donations and handle essential infrastructure and that is what our goal at the Foundation at the broadest level is to provide support. So that these people who are working on all these different projects all they have to worry about is doing code. They don’t have to worry about, "Oh I need to go and buy a new server, how we are doing to deal with this donation or this tax policy." We try to deal with all of that. So I think there is a critical mass that works in our favor.

Scott:  Right, right and let them focus on the piece of it that they really enjoy, which is whatever this project is that they have come up with. They have a passion, like you said, for not having to worry about all the housekeeping stuff.

Bookmark this:

Interviewee: Justin Erenkrantz

In this interview with Justin Erenkrantz we talked to him about:

• The Apache Foundation.
• Project Management Committees and the Apache Foundation.
• Some of the reasons Jason feels Apache has been so successful.
• What other open source projects might look to Apache for in terms of inspiration due to Apache’s longevity.
• The Apache Incubator and it’s role as part of the Apache Software Foundation.
• What is the on horizon for the Apache Web Server.

Justin:  So my name is Justin Erenkrantz. I’m currently president of the Apache Software Foundation (ASF), also on the board of directors. And I’ve been a contributor to the Apache HTTP Server, the Apache Portable Runtime, and Subversion, and some other projects for quite a while now.

Scott:  Talk a little bit about Apache and talk about how it’s built.

Justin:  The Foundation as a whole has over 50 different projects. There’s the Web server, Tomcat, SpamAssassin, Geronimo. There’s a whole variety of projects. So there’s an overall Foundation, with committers for each project. They’re relatively isolated. The fact that I have access and I work on HTTP Server doesn’t mean that I have access to, say, Maven.

Each project kind of gets its own merit. Our culture is a meritocracy, and so we expect people to show up on the public mailing lists and start contributing, and eventually they’ll be recognized. Eventually they’ll get a vote, and this vote will allow them to be able to commit (code). Across the Foundation, we have about almost 1,600 committers that can commit to some of the 50‑something projects. And with that, you get to make changes.

You also get something called a veto, which is something that we can probably talk about a little bit later. That’s one of the core governance structures that we have. There’s also a thing called the Project Management Committees, and those are the groups that are responsible for each one of those projects.

Above that is the board of directors. And in comparison with some of the other open source organizations, our board at Apache doesn’t get involved with technical details. We’re not going to get in and say, "Oh, you need to change this variable name." That’s not at all what we do. We’re just there to make sure the organization is running, make sure that everybody’s happy and getting along. You won’t see a director really getting involved in a technical discussion, unless they’re a part of that project to begin with.

Scott:  Does the board provide more of a steering functionality, then?

Justin:  No, not even that. We’re almost completely hands‑off. There are times when personality clashes happen in each project, and try to mediate situations. Again, we’re not going to make technical decisions.
There are some things that are centralized. So one of my responsibilities, as president, is I’m responsible for the day‑to‑day operations of the Foundation. We have a Subversion server. We have issue tracking. We have websites. All of that is centrally managed by our infrastructure, too.

Scott:  One of the things you mentioned was votes and vetoes as the way the project is governed. So expand on that a little bit. How does that come into play?

Justin:  For example, there’s the HTTP Server project. Within that, it’s governed by a Project Management Committee, a PMC. Looking at your past interviews, other people have mentioned this. This kind of structure has replicated itself on how we have it at Apache.

These are groups of committers who are responsible for the project. On HTTP Server, there are maybe about 60 to 70 people on the PMC. Every single one of those people has what we call a binding vote. The votes are used in two main ways.

The first way is for release. Within Apache, across any of the projects — this is one of the hard and fast rules — there must be three binding votes before it can be released. That means we must find three people out of 60 people on HTTP Server to say, "Yep, this is a good release. We’re going to put the Apache brand on it. It’s going to be the ‘Apache HTTP Server’." That’s the release part of the vote.

The other part is a veto. If even one of those 60 people say, "This change is bad, I’m going to veto it," that means the change doesn’t make it in.

One of the ASF’s founders, Roy Fielding, refers to it as a kind of a shot gun. It’s kind of like, "OK, I’m done discussing this with you. You’re not listening to reason. Veto. Stop. We’re not going any further."

And it’s usually a last resort. Vetos are uncommon. They’re not something that happens every week or every month. Generally, if there does happen to be a lot of vetoes, it means that people aren’t willing to compromise. So, that may be something where maybe the Board of Directors might say, "Hey, you know, we might keep a close eye on it. Is there anything you need to talk about, do you need any help to resolve this…"

But, generally vetoes are relatively rare but they do give a power to the 60 people to say, "You know what, there’s not going to be any change that I disagree with, there’s not going to be anything where I say ‘Oh my God, I can’t live with this change being made’." So that’s an enormous power and it’s given to the members of the PMC.

Sean:  We found these nuclear options in place across many projects, but they don’t get used very often. It that involvement in open source takes considerable time and you do it because you believe in moving the project forward. It seems like going into this, people know that the only way to make progress is by consensus. No one person is just going to get their way. And if they really want to, they have another option, which is to go fork the source. But that has extremely high barriers also.

Justin:  Yeah, absolutely. But as I said, generally the vetoes tend to be very rare. It’s almost like mutually assured destruction. That’s the point. One thing that has happened a couple of times over the 10 or 11 years of the HTTP Server Project, is where there were vetoes on both sides. "We’ve got to do it this way. No, we’ve got to do it that way". There was this huge flame war for a couple of weeks and then they finally said, "You know what, we’re not going to agree". They even had a telephone call. They were doing everything, it was just this big mess. And eventually they said, "You know what, we’re just going to leave it to a vote. We can’t agree, we need to move forwards, this is blocking us. OK, we’ll go ahead and resolve this, whatever way the vote may turn out".

Scott:  Talk a little about the history of Apache. How did it get its start and what are some of the major evolutions it’s had getting to the present point?

Justin:  Apache has its roots back in the early days of the World Wide Web. The story begins with NCSA Web server from Illinois Urbana‑Champaign. They were running the NCSA Web server. Eventually, a lot of people left to go to Netscape. The NCSA code eventually got abandoned, more or less. I think there were nine people who found each other on Usenet and said, "Hey, I have a patch for NCSA. OK. Why don’t we start trading patches?"

They got together and they started to exchange patches and start coming up with a new version of this NCSA service. They started taking it in this new direction. They started saying "OK. Maybe we should get another group going." So they founded something called the Apache Group. It was an informal thing. They did that for about four or five years, starting in ‘93 or ‘92 (started in Feb ’95) Eventually they got to the point where other people said, "Hey, we like what you are doing."

By this point Apache had already gone through, [inaudible] a Web server, and made it up to version 1.2. They created The Apache Software Foundation in 1999, and started doing things besides just a Web server.

That was the start of the Apache Software Foundation. The early initial project was the Web server, and that is still what a lot of people think Apache is. Now you have close to 60 projects.

Scott:  Focusing on the Apache Web Server, what are some of the secrets of its success?

Justin:  For the web server, more than anything, it’s been the way we designed and supported all the standards. And it’s free. That was the tagline that Roy Fielding had on his website for a long time, "Apache, the best web server money can’t buy."

One thing that really speaks well of our community has been the lack of forks. The community embraces anybody who shows up. The project has evolved and widened, from just the Web server itself, where people have wanted to do new things. People wanted to do an FTP server, a mail server. It can do all those things today. The community has been characterized by being willing to be open to just about anything.

Scott:  It seems like open source projects that are modular do better because working on the core of an open source project might have a high bar. If it’s modular, you can write modules without going through the scrutiny of submitting code to the core. Modules give you a way to get your feet wet and participate. At the same time you need a really healthy community too. It is the personalities and way the governance structure is set up around it. It has to be really healthy as well. Just to say it back in my own words, those two things seem like they came together with Apache…

Justin:  Yeah, and if you look at one of the key evolution points between the original NCSA server and Apache, it’s when (early Apache developer) Robert Thau modularized the whole thing one weekend. By and large, most of what he did 10, 12, 13 years ago is still present in the code base and technical architecture. By modularizing, he did a really good job of cleaning up the earlier NCSA code base.

Scott:  So what is version 2 all about?

Justin:  Version 2 was all about threading and portability. With Apache 1.3 they added Win32, Netware, and OS/2 support.

Version 2 started out with a number of internal forks. One of them looked at Netscape’s portability runtime. Other developers did their own portability library, implementing the same function on three platforms and hiding the implementation details."

There ended up being a licensing dispute with the Netscape/Mozilla guys that prohibited the Apache guys from using the NSPR runtime. That spawned the Apache Portable Runtime project. That’s a lot of different projects now, but if you look at why it happened, it had to do with the licensing issue. If you look at the Foundation now, I think one of the things we are well known for is the terms of our licensing.  It’s a key differentiator from, say, the Free Software Foundation.

Scott:  Right, right.

Sean:  Apache’s been around for a really long time, and it’s obviously seen as one of the more successful open source projects, to say the least. What do you think other open source projects look to Apache for in terms of inspiration when they’re starting up?

Justin:  I think by and large, what you see most people copying are the governance structures and the licensing. Those are two things projects have been copying. I think you can see that in Eclipse: they almost use some of the exact same terminology.

Scott:  You still there?

Sean:  One of the things you mentioned earlier that has always been intriguing to me, was portability. I can see that it’s really important for Apache to be portable between different Linux flavors, and maybe even be portable to embedded devices and things like that. How important is it, from a practical standpoint, that Apache runs on more than just Linux?

Justin:  Extremely important. We have contributors who are only interested in supporting a NetWare or Windows or OS/2, even BeOS in the past. It’s where we’ve gotten some of the diversity of the community. It’s a hook to get people into the community. "Here’s a little something I know about, I know my operating system, and I’ll contribute this patch. Hey, there’s something else that may not be platform‑specific."

Scott:  Does Apache take the standpoint that it should run equally well across operating systems? OpenOffice, for example, wants to be pretty much the same OpenOffice regardless of where it’s running. Does Apache run differently depending on…

Justin:  Absolutely differently. Basically, our approach is in whatever platforms people want to maintain, that’s what gets supported. By and large, on the Apache HTTP Server, we have one guy who does the Win32, and it’s been his baby for many, many years. There are other people who contributed a little bit to the Win32, but he’s this one person who had been the individual who is responsible for it.

It’s not a dictate that, "Oh we have to support that." If someone is interested in supporting an OS, great! We’re not going to stop them, but it’s not going to be a mission statement, that we have to support all these platforms equally.

Actually, if you do look at our HTTP server mission statement, it says, "Apache HTTP Server Project is an effort to develop and maintain an open source HTTP server for modern operating systems including Unix and Windows NT." So, it’s in our mission statement, but the only reason it’s there is because we have the contributors to provide that support.

Scott:  One other thing that varies from project to project is where the code comes from. If you take a look at MySQL, pretty much everybody working on it works for the MySQL company. If you look at other things like the Linux kernel, a lot of that comes from corporate developers: IBM, Red Hat and a lot of people. Do you have a sense for where the Apache code comes from? How much of it is from corporate‑sponsored developers versus the proverbial guy‑in‑his‑garage?

Justin:  I think it comes from a wide number of sources. What you will see is that contributors remain the same even when they move from job to job. That’s definitely been the case within the HTTP Server, that’s been the case for some of these older projects as well. One day they may be working for IBM, the next day they may be working for Red Hat, and then they may be working for some other company. They may be working for Google, maybe doing it on the side, that’s what you tend to see. Some of these contributors may have started out working at Sun or HP, then they move but they’re still working on it. They still contribute to the project.

Scott:  There are certain people who look at open source and they think it’s all written by people in their garages, contributing. Other people look at it and say, "It’s all written by people working for corporations." How important do you think big corporate sponsorship is to a project like Apache, and does that also create certain challenges for the project?

Justin:  It’s a balance. You see some people who are getting paid to work on it. They work on it all day during normal business hours.

Then you see people who are the exact opposite, who may be working as a system administrator or something else, and they only time that they can work on it is on the weekends. So you see the overlap.

One of the key things in Apache, another quote from Roy Fielding is, "If it didn’t happen on the mailing lists, it didn’t happen." All of the discussions, all of the decisions, have to be made on our published mailing list. That allows people who may be in different time zones, or different work schedules, to coordinate through this mailing list.

They can read it during the day when they’re at work, during the night when they’re at home, whatever works for them. That way, decisions aren’t made in a face‑to‑face meeting, or a call, or an IRC, all the decisions have to happen on a mailing list.

Scott:  So IBM, just to take a big company name, can’t get something into Apache just because they’re IBM and they want it. If one person out of sixty people vetoes it, it doesn’t really matter how badly a big company wanted certain code in, it’s not going in.

Justin:  That’s right. The other aspect of it, the thing Apache has been addressing the last couple of years, is how new projects come into Apache through something called the ASF Incubator. This is about how they operate as an Apache project. They have to get all of the legal paperwork in place, so we can say, "Yes, we can release this under the common Apache license." That’s how we are trying to get new projects, and that’s why you’re seeing growth in the number of our projects, because incubator keeps spinning out new projects.

It’s always a concern that in order to graduate from incubator and become a full‑fledged project, you have to have diversity. Basically, you can’t have any one company dominate the project.

The rule we use, that you see pop‑up again and again in Apache, is the rule of three. There must be at least three committers that are diverse. The discussion that is going on right now is, "What is the definition of diverse?" An example: "Well, I work for IBM, and I work on this project full‑time, but there’s another guy from a completely different division who isn’t getting paid to do this who’s also working on it." Should that be counted as a separate individual? That is a discussion now. Some of these companies are so big, it’s like the old joke of, "Oh, you’re from London!  You must know so-and-so."

Scott:  In open source projects what gets checked in is the source code. With Apache, it looks like there’s this thing called the Apache HTTP Test Project.  Is that essentially like a test suite for Apache?

Justin:  Yes. Yes it is.

Scott:  OK. And what’s that focused on? Is that mainly functional testing?

Justin:  Yeah, it’s basically a Perl‑driven test suite, originally from the Mod Perl guys. They had this whole Apache test tool kit that they used as a kind of smoke test. And we said, "Hey, we’d like to take that." And we extended it from there.

Generally, what you’ll see is you’ll see people will use that as a kind of smoke test before they do a release. We talked earlier before about that you need to have three plus one in order to do a release. But we haven’t said anything about how people make up their minds, and say, "Yes, release this." And so generally what people do is they run tests on their favorite platform.

One of the things that we did with 2.0, and still do to some extent is "eat our own dogfood."  In the early days in of the 2.0 series for the Apache HTTP Server, we would say, "OK, we have a release candidate. We’re going to put it up on Apache.org. We’re going to go run it for 72 hours and it can’t crash."

Basically that was kind of another way of doing the acceptance testing. Saying, "OK, we can run it on a site that gets this much traffic. It didn’t crash so it’s probably going to be OK for you."

Scott: Are there tests specifically looking for vulnerabilities like buffer overruns, or is that really outside of the scope…

Justin:  In the past when there’s been some type of buffer overflow, or some type of CDE vulnerability, generally you write and check in a test to make sure it won’t show back up in regression.  I think that basically depends on if we can come up with an easy reproducible test case.

But I think you won’t see some test cases there that are typically for the vulnerabilities.

Scott:  A while back we talked to Michael Howard, who’s a security guru at Microsoft.  There’s a lot of things they do, but one in particular was banning certain APIs like strcpy because they were just inherently vulnerable.

Justin:  Yeah. Basically we do some things and put them in Apache 2.0 with the APR path (code base). [sp]
If somebody actually tries to call these functions, it’s going to expand onto, "Why are you trying to do this?" There have been some cases where we put it in the file to say, "Don’t do this. Don’t call this." Or say, "Oh you’re going to call this? Well then we’re going redirect you to a safer version."

But, we don’t have the flexibility of say a Microsoft, and say, "Oh we want to have this new security API in the operating system." That’s not something that we have influence on.

We generally have to look into the constraints of the operating system and work with that.

Scott:  Sure.

Justin:  Over the past few years, I think there have only been one or two cases where there were remote root exploits, and that speaks well for us.

Scott:  When people are posting patches, are the security implications discussed on the mailing list?

Justin:  Oh, absolutely. You’ll get people saying, "Hey there’s something with this vulnerability, or this will break this or that."  So, yeah there’s this constant vigilance for the security.

Scott:  Are there a lot of security-related tests that are put in proactively? I hear about things like "fuzz testing" and other proactive ways to probe the surface area for vulnerabilities. Does that kind of thing happen….

Justin:  There are security product providers, Coverity is the one that pops to my mind. They’ll say, "Hey, we ran our tool on your code and here’s a report of vulnerabilities." We take a look at the reports and say, "Thank you very much." And then analyze them ourselves.

But it’s really triggered by what the committers are interested in. We’ll see committers who are very interested in conformance to the protocol specs. We’ll see people who are interested in security, people who are interested in performance, etc.

But we don’t tell the committers what to be interested in from the top‑down. It’s more like, "John is interested in security so he’s really focused on tying up all the security issues."

Scott:  I’ve bumped into companies like Coverity that use open source to market their tools because open-source provides a large, free code base they can throw at their tool.

Justin:  Absolutely. Yeah, I remember when we first looked at Coverity, the amount of false positives were generally high.  We’d look at the code and determine, "No, there isn’t a vulnerability there. What the tool is reporting can’t actually happen." There were maybe a handful of actual things that we said, "Yes this is an issue".

They may not have been as severe as what the tool was claiming, but we said, "OK, we’ll clean this up."

Scott: What do you see on the horizon for the Apache Web server?

Justin:  The IETF is forming a new working group to do an editorial revision of the HTTP stack. Their work might lead to the next generation of the HTTP protocol. And I think that is something that we will be very much involved with.

Scott:  What about in the incubator? 

Justin:  Every Board meeting we’re graduating things like ServiceMix which is an ESB and component suite based on the Java Business Interface. One that is going to be new, probably the next board meeting, is a standard C++ library, which we’re getting from RogueWave. There’s things like Abdera, which is an Atom feed. We’re seeing a lot of things like ActiveMQ which is event‑based messaging.

What you probably see a lot in Apache are low‑level infrastructure type things. You’re not going to see things like, say, OpenOffice. You’re going to see things that people can pick and choose to build larger applications. I think that’s what our niche really is.

Scott:  What’s your sense for what happens to an Apache Web server release between the time you’re done with it and it gets distributed by a Red Hat, Oracle, Solaris , etc?

Justin:  We have contributors from Red Hat. A lot of the people who are doing distros and ensuring that it gets in front of the users are involved in our community. Our philosophy has been why are you making this huge patch set for this particular product? Get it upstream, get it back to us, we want to take it. I think generally, for the most part, you will see that there isn’t a lot of variation when it gets into the distribution because these people have been working with us.

Sean:  Let me do a follow‑up on that. Some say the strength of a closed source project is that the company may be able to provide more of an integrated stack. You take a look at something like Microsoft ships and they might say, "You should use this because it’s an integrated stack, and there’s a single vendor you go to for support." Obviously there are some projects that are tightly coupled together, such as Suversion and Apache. What would you say to that from the open source side?

Justin:  Right.

Sean:  It seems like in open source, the communities are not isolated.  There’s a fair amount of core maintainer communication that’s going on.

Justin:  Yeah, I think you see that. I think that’s why you see a number of committers in multiple communities. As you get used to it, you start to follow the dependency chain, and you get into those communities and say, "Hey, I just broke this for you over here, but here’s the patch to fix it." You tend to see a lot of that happening.

Sean:  Are there things you see in other open source projects that look interesting and might influence Apache?

Justin:  I like what Ubuntu has been doing, where they say, "We’re doing a release every six months. (no matter what), we’re going to have a release." That’s a very hard thing to do. That requires some of the dynamics that Canonical has with their contributors. That’s something that I think they do really well.

Generally our philosophy has been, “we’re releasing when it’s ready”, and some think that’s a good philosophy. You don’t want to promise something, but then you think, "Well it’s been so long since the last version." There are all these changes that sit there and keep getting improved upon. But if you have the regular release cycles, I think that’s a good thing.

Scott:  Justin, we’re out of time, but this has been a great conversation.  Thanks for taking the time to chat with us.

Bookmark this: