Interview with Justin Erenkrantz – President – Apache Software Foundation

Interview with Justin Erenkrantz – President – Apache Software Foundation – Part II

Interviewers: Scott Swigart and Sean Campbell

In this second part of a two part interview with Justin Erenkrantz we talked to him about:

Scott Swigart: In this part of the interview, we wanted to dig into some of the tenets, if you would call it that, of the Apache way. And the first of those of course is collaborative software development.

So talk a little bit, if you would, about how Apache does collaborative software development. I’m sure some things are very traditional and similar to the way that other open source projects might do it, and there are probably things that also might just be a little bit unique to Apache. So how do you try to insure good collaboration?

Justin Erenkrantz: So the main center point of all of the collaborative software development that we do in Apache is the mailing list. That’s where pretty much everything happens. As one of the guys mentioned before, the maxim has been: "If it didn’t happen on the mailing list, it didn’t happen." And that generally tends to be true.

Basically, if you follow the mailing list for a particular project, then we’re expecting that you should know what’s going on within the project. Within that, those are pretty much all public lists. Everybody can subscribe, even people who are committers, people who are just users, people who just work on another project that may consume the Web server or maybe PHP modules or something like that.

It’s pretty much an open forum. Anybody can voice their ideas. Generally though, just the way things work, most of the traffic tends to be from the core developers who are active at that time. The traffic patterns of the list change, and you get an idea of how much discussion. You generally see peaks and valleys for the mailing list discussion. Things get really heated or things are just chugging along and there’s not much traffic on the list.

So that’s where all the discussion should happen. And then of course there is the source code repository. And in Apache we always had the thought of having shared repository. A lot of projects now are starting to ‑‑ Git and Mercurial and all of these distributed version control systems to be centralized version control systems.

And that’s something that, within Apache, that goes against what our thoughts are, because we want to all be agreeing on, "This is the Apache version."

Scott: Sure, no problem.

Justin: So there’s no leader within the ASF. There’s no person, if you look at say, Linux and you say, "This is Linux’ tree," or this is Andrew’s tree or this is Alan’s tree. Instead there is just the Apache tree. So there’s really no concept of, "This is Justin’s tree," or someone else.

Scott: Is the feeling then that essentially there should just be intensive discussion before something gets checked in? So when you’ve reached a point of consensus, it should get checked in rather than the way other projects work where different things get checked into different people’s trees. And then when it’s time to build a release, you have to pull stuff from these different sources to figure out what’s going to be in the release and what isn’t.

Justin: Right. So, generally, what tends to happen is there are two states a tree can be in within Apache. One of them is "commit then review." And then there is "review then commit." And you’ll see some projects differ on particular trees.

For example for the HTTP Server, the trunk ‑‑ which is the main development ‑‑ is usually always under the "commit then review," which basically means that anybody who has commit access can feel free to go and make changes and they basically have the benefit of the doubt that the change is going to be good. And there’s generally an implied threshold: that if you’re going to make a really big change, go discuss it on the list. But if it’s a minor change or adding a little feature, that’s probably not going to be controversial, go ahead and commit that to trunk.

But for our stable releases, generally things that have already been released and we’re doing maintenance on those, are under the "review then commit" model. So that’s going to be RTC, and that means that any change to those trees has to be pre‑approved. That means you need to get three binding votes from other committers to say, "Yes, this is good change and no one has vetoed it." Technically you would use a file called STATUS, just a plain text file that some projects will use, that basically tracks all of the things that are under discussion to be back‑ported or added into this tree.

Scott: Got you. Like any open source project, and this one is democratic, there is a vote to commit. Well, there is a vote if it is a release product. If it’s not a release product and people later decide it wasn’t a good thing to do, it can be reverted out.

And you mentioned before that part of your governance is that out of these 60‑something maintainers, one of them can essentially veto a change if they want to. So talk a little bit about how conflict resolution usually works. For example, when you have people who are—and I understand it doesn’t happen often—adamant one way and another person who’s adamant a different way. How do you see it play out that those eventually get resolved and things move forward?

Justin: Generally what happens is, like before, the veto just tends to be a last resort. So let’s just say that someone makes a change to the trunk and I don’t like it. I might just say, "You know, we should talk about this change," or, "Here’s the problem with this." Generally, the person who has committed that says, "Oh, yeah. Here’s why I did it this way," and comes up with an explanation, and then through a process on the mailing list, figures out and resolves those conflicts.

That’s what you tend to see happen. And it’s all tends to be, for the most part, "I’m sorry, I forgot about that particular corner case or that." And everything tends to get resolved very naturally.

The veto tends to be when someone says, "No. I have to do it this way," and someone else says, "No, that’s wrong." That’s basically when the community is at risk of breaking down. But generally it doesn’t get to that point. Everybody is, "We’re all going to go in this direction; this is the right direction for us to go in. We want to add this feature. Let’s work through whatever issue you may have about this particular commit."

Scott: One last thing before we move on to the next point. Other than the fact that everybody has their own private tree, is there anything else about the collaborative nature that you think is somewhat unique to Apache?

Justin: We tend to do a roll call before the release. So at this point there’s been a review of everything. But then let’s just say that I want to release the next Apache 2.4 or whatever. Then basically what I will do as a release manager is say, "OK, I’m marking this as 2.4," and I produce all the artifacts. I produce the tarballs, [inaudible]generated files, whatever. And then I send it to the list and say, "Hey, is everybody happy with that?" And then that goes to a voting process.

There’s review at all stages, but there’s also a review at the point where you do a release, and you have to get, at least, three people to approve a release. One thing that’s different is that it’s not possible to veto a release. It’s strictly majority rule on the release.

Scott: So in other words, there is veto capability on the individual check‑ins. But, as you said, when it comes down to doing a release it is a majority rule vote.

Justin: Yes. You’ll tend to see someone voting ‘no’ on a release if it doesn’t work on Linux or something, and generally you’ll see it get stalled and it then gets fixed. But there have been a couple of cases when we did the release even though we knew that it didn’t work on a particular platform, and so we made a release note. But the veto does not apply to releases.

Scott: Interesting. So moving on to licensing, one of the key things with Apache is the commercial‑friendly standard license. Talk a little bit about what that means.

Justin: Basically within Apache, we like to have a big tent where everybody can come in and play with us. We think that the community that we developed within Apache is going to be the motivation for you to stay involved. For example, within the HTTP Server community, you have all these experts and Web servers, and if you’re part of this community, you get the benefits from them. And so there’s an incentive to play within the community, so that I don’t have to hire five guys and do a whole team; I can leverage the other people within the community.

But in turn, all those people who are part of the community say, "Whatever you want to do with the code is fine. We’re not going to get hung up if you make it a commercial product or an open source project. We created it and it served our needs, and if it serves your needs, that’s great."

Scott: What springs to mind are things like GPL. Am I seeing it right in that Apache is more commercial‑friendly than GPL, V2 or V3?

Justin: There are companies built around GPL licensed software. But what we tend to see are two classes of GPL products: In one, there is a real community, maybe within Linux, and they’re all happy to make all their changes available to everybody, and that’s a very good community. The other community you tend to see has a single stakeholder that has a prevailing interest in the GPL product, and they basically have an unfair share.

You can see this with some of the GPL projects that require copyright assignment. In order to participate, you license your changes in the GPL and you have to give a copyright assignment to the principal stakeholder. Now they are then free to release the commercial closed source based on your work because they have the copyright or whatever legal mechanism. There is an imbalance there when you look at those two.

Generally when you think about the GPL, you’re divided, with broad strokes, into those two groups. This is a real community but the other groups are aware and want to be clear about which one has a dominant role, and that’s one thing within Apache we don’t like to see. As our projects go through incubation and get added to the Foundation, one of the things we do is make sure that the community is diverse. In fact, there is not a single dominant stakeholder that can direct the project in any untoward way.

Scott: Right. I can make changes to it, distribute it as part of a commercial product, and I would not be required to contribute those changes back to Apache. But under a GPL license, any modifications made require you to make the source code available. You cannot have closed source proprietary extensions or modifications of it. Any modifications you make, you have to open source and it has to be under the same license. So that’s the key differentiator?

Justin: Yeah, our philosophy is that the community is what’s going to bring you and keep you there, and that’s why you’re going to stick around. If you released a commercial product around one of our projects it’s going to be to your benefit, to basically keep your commercial project as close to whatever we’re releasing.

Scott: Right.

Justin: You can pick up all the bug fixes and whatever improvements; you get those as a free rider. But in a sense, you are contributing whatever changes you’re making voluntarily back into the greater community.

Scott: Yeah, that makes sense. Do you have examples of companies that have used different Apache projects because of the commercial‑friendly licensing, where they probably wouldn’t have it if the license weren’t so commercial‑friendly? Is that a topic that comes up?

Justin: Absolutely. You see companies like IBM that release their versions of the Apache Web server or Geronimo under different names, but in the core, they are Apache projects. We’ll see that even with smaller companies such as Covalent that does commercial support. Basically, they added in a couple of extra things that provide support to their users.

One thing that the Apache community really does not focus on providing is 7/24 support. Covalent goes in with their business model and provides the support and training around these particular Apache projects. You will see businesses like JBoss using Tomcat. So you see all of these commercial companies using things that are Apache projects under the covers.

Scott: Right, so that freedom has led people to be a lot more creative about how they structure their business. They have a lot more options in how they participate with the different Apache projects, how they contribute back and how they structure their own products. What is the relationship between the Apache Software Foundation and the Free Software Foundation? Is there any or are those fairly separate endeavors?

Justin: There is no formal relationship. I’ve never had a conversation with Richard Stallman, but I’ve had conversations with Bradley Kuhn who used to be, at that time, the Executive Director of the Free Software Foundation. So there’s an informal get‑together of foundations to compare notes, and that’s generally a very good thing. How do we keep our ears open to what Mozilla’s doing? If they’re doing this new technique, then we can give them a call and ask, "What are you doing? We’d like to follow on it."

One thing we’ve been doing with the Eclipse is a joint conference. There’s going to be a conference in Asia that’s now scheduled for 2008. So it’s a way for us to get the communities talking to each other.

Scott: Sure. So basically, if I can summarize, you guys get together around joint events and joint things where it makes sense. You share information because you’re all part of the open source community. Philosophically you may agree to disagree in terms of the details of licensing, commercial friendly, and that kind of stuff.

Justin: Well and you ought to be using more…projects have different circumstances. Apache‑‑we have a very vocal membership and we have this and you compare that to let’s say the Mozilla which has a completely different governance structure. But if you look at Brian Behlendorf, he’s been on the Mozilla board for a very long time and he was one of the founders of Apache.

Scott: Gotcha.

Justin: So you have this intermingling of the communities. So someone like Brian Behlendorf who was brought in through the Mozilla and says here’s how we did things within Apache, and here’s his expertise and his experience that he got, he can share that with the other people within Mozilla.

Scott: Gotcha, gotcha.

Sean: Let me refer to one of the other tenants of the Apache way. I’m curious about this just because I was thinking about the conversation we were having. To state the obvious, you’re focused on producing software. I notice that you have a security committee that—if I’m reading it right and for lack of a better phrase—provides a service to all of the projects that are part of the foundation. And it looks like those projects can turn to the security committee and ask security related questions or possibly look for guidance from them, regardless of whether they’re Tomcat or some other piece of the foundation? Is that accurate or is that not accurate?

Justin: It’s somewhat so. We have a security team, which I believe is currently a Board committee. But basically what they’re responsible for doing is ensuring our security at Apache.org mailing address gets responded to. And these are generally people who are very security savvy.

But there tends to be some people from Tomcat, from the HTTP Web server, from a higher profile project on this internal mailing list. So let’s say that, to give you an example, let’s say there was a security vulnerability in Derby and they could parse to those reps and say, "Hey, we have a security vulnerability. What do we do?" And so there’s expertise and, "OK. Here’s what you do. Here is your administrative contact. Make sure your mailing list… Go talk to…" Kind of a shared resource. But we’re not getting the focus on producing the fixes for the project but it’ll be "OK, here’s the responsible disclosure policy and an attribution policy." So that’s generally what their role is.

Sean: Are they providing fairly prescriptive guidance but just not down to the ‘I’m going to change you’re code’ level because they don’t know the individual projects at that level? Would that group essentially be the center for discussions around a security development lifecycle for the Apache Software Foundation? And an attempt to pull those best practices together?

Justin: Yeah. I think basically our project concern…we have something. What’s the process? What do we do? And that’s as an advisory role. OK, here’s the process and the procedures to follow.

Sean: But it’s purely advisory, right? I mean one of the projects where they feel that their code is "secure enough", or they’ve looked at it long enough or they feel that they’ve handled it. Then the advisory committee comes back and says, "Well, we really think you could take a look at this again." That’s where the communication would stop and it would be up to the individual project whether they want to take that under advisement or not.

Justin: Yeah. I think so. I think record security, can you maybe at that point write back to your original reporter and say, "We looked at it and we don’t feel there’s a security vulnerability here." That may be…that has happened where we look at things and we say, "No. This is not an issue." But generally, really the security team is more of a reactive. So they’re not proactively performing security analysis on our code or anything like that.

Scott: I just want to clarify. It sounds like they have a little bit of an all‑up policy for somebody sending an email to that address; somebody reports what they perceive as vulnerability or reports some kind of issue. They do a little air traffic control. They route it to the project.

Justin: Exactly, exactly.

Scott: There’s a general process that the different projects would follow. Basically that sort of happens and that’s one of the things that the security group advises the other projects on. Well this is generally a ‘way we do it’ sort of thing.

Sean: Let’s go to a different piece of the Apache way, the emphasis on a technical‑based interaction. One of the things that I find fascinating about open‑source projects is the way that they exorcise community members that maybe aren’t following those rules.

[laughter]

Sean: Because we got some interesting responses when we talked to people about it. It’s like, fine. We understand that everybody is an adult. We understand that everybody will try to handle themselves in an appropriate manner. But if anybody’s worked on a software project they know that not everybody does, right? So…

[laughter]

Sean: Considering that you can learn a lot from a story…if you’ve got a story or two about either fully pulling the ejection handle on somebody that would be interesting to hear? Or a scenario where it just took serious counseling to get somebody pointed in the right direction.

I would be curious to see how you guys handle that. You obviously have procedures in place. But at times you have to go beyond those with some amount of intervention and I’m just curious how that played out.

Justin: Yeah. So there’s one case that comes into mind but I’m trying to reserve the right to figure out how much of this has been disclosed.

Sean: Yeah, Sure

Justin: So I’ll tell the story and then leave people’s names out of it but I have to go back and see how much of this has been told. So recently within one of our larger, well‑known projects, there was a bout, to use the word, between two committers. And they basically ended up vetoing each other on everything. It’s like no, no, no and tempers got flared. And it got into a very unhealthy situation. They’re two very strong‑willed individuals.

And basically what happened is the PMC, so it was the PMC responsible for this particular activity, basically had to step in and say, "OK; we need to come up with some policy or come up with some new rules to get everybody back to ground zero. So it wasn’t a matter of ejecting anybody. It was never really an option that…basically what happened was they said, "Here are the ground rules. Here is…if you’re going to go do this you have to follow this set of rules. If you’re going to go do that, you’ve got to go follow this set of rules." Basically the community agreed to say we’re going to go and we’re going to voluntarily adopt these rules. But as a settlement process; lots of flames and a lot of innocent people getting…

Sean: Right.

Justin: …accused of things and that process. The other one that’s in our history was a project called Avalon. And this is one that’s definitely well known so this won’t be any issue about this one. Avalon was a container framework. And what happened was two individuals just did not get along. And they were ending up in what we would call a commit war, where they would basically be reverting each other’s changes as soon as they came in. And it is just this whole really poisonous environment and basically in that case, the community wasn’t able to deal with it. And so basically what happened there was they fractured.

And so that one of the individuals went off and he took his code and you know, we wished him luck and said have a nice life and he went off and then it was kind of some other people came along and they did a project called Excalibur, which was basically the remnants of this whole Avalon project. You will just see if you look at the mailing list traffic, you will just see this giant peak and then this sudden nothingness because the project got shut down because no one could play well with each other.

Sean: Right.

Justin: By the way what it is interesting is some of the veterans of Avalon, they really got involved in the greater Apache community, one of our new directors this year. He was in the middle of all of this, but during that whole experience, he was one of the people trying to keep things level and stuck around and this year he got elected to the board of directors.

Sean: Since Apache is a large foundation, I’m curious about a different point. If you were a closed source company and you feel you are short on testers or short of security experts, you simply go to HR and put out a requisition and hopefully some good folks come back.

So has the foundation had to answer requests from projects where they say, "OK, look, we think we are geniuses on nine out of 10 of the things we need to do, but this one thing we really need people to help." How does the foundation help with staffing up a project in this type of case?

Justin: It is more bottom‑up than that I think in the sense of the culture that we have. You will see some overlap between the HTTP Server committers and the Tomcat committers because you will see that ‑‑ sometimes the Tomcat committers came over and they say, "Hey, we need some help with HTTP." Well lucky us, we have some of the world’s foremost experts in HTTP server, and that basically got them within the communities.

So I think our community’s diverse enough to, "Hey I am looking for a person who knows SQL." OK, I am going to go on the Derby mailing list and say, "Hey, I need some help with SQL" or for something for build systems, I’ll go to Ant community. And there will be some of the people who are the foremost experts in that. That’s actually one thing of having such a large diverse community is that you can pretty much find someone who understands something about something somewhere within the Foundation.

Sean: I figure that’s one of the advantages. You’ve got a massive talent base but at the same time it is segmented into the project, so the Foundation can help orient a little bit of that knowledge of where the talent base is.

Justin: Yeah and as I said if you look out the social graph and it is a weird mix of people who are committers on Cocoon, maybe committers on Mina, then maybe on Gump and so you will see that developers themselves, the committers aren’t necessarily staying in their silo, there are some who do, but there are also just as many who will go to other communities and work on other projects.

Sean: Well I have couple more questions but Scott can go ahead. I want to give you an opportunity to jump back in.

Scott: So talk a little bit about standards because one of the other tenets or pieces of philosophy is faithful implementation of standards. Talk a little bit about what that means for Apache?

Justin: Right. So this was initially when we started off there was HTTP and there was the IETF standards and that was when you have editor of the HTTP standard is one of the people behind the code base, there is the knowledge is going both ways in a sense is that we are that able to influence the standards process. But at the same point we also have some of us involved with the standard process and feeding those changes back and to the development and supporting those standards.

But since then, the initial, you see our participation within some of the key web server specifications, then probably most importantly our participation in the Java Community Process and that is, as you know, we have so many Java projects and that so many of our projects are implementing some JSR specification and our involvement within the JCP has been to ensure that we can implement the specifications and we have projects have representation on these expert groups that device these standards.

Scott: Right. So it isn’t just like the standard shows up and then you figure out how to implement it. There is this two‑way street where you are shaping the standard because you guys have such a big, real world implementation. And meanwhile, the standard is telling what you guys do because they have their own stakeholders, but they are considering your recommendations and you want to conform to what they eventually approve.

Justin: Correct.

Scott: Yeah, I don’t know. I don’t have anything else specific. Sean, do you?

Sean: No, not right now. I think this led us into some new stuff and from our end, we really enjoyed chatting about it. Justin, do you have anything you’d want to add or things you think we should address overall based on the theme of where we were going?

Justin: No, I mean you did a good job of asking me the questions.

Scott: I guess there is one final question. When somebody is starting or has an open source project, they can pick the license they want they can do what they want for their community, things like that. What makes people want to be, in your mind, an Apache project? What is the draw I guess?

Justin: Well, I think from my perspective, the draw is we handle a lot of the mundane governing structures and all this and the infrastructure and the licenses. And that is all essentially managed and I think you see a lot of open source projects like, "Oh we need to go get a foundation."

And that’s a lot of overhead, there is a lot of overhead to create a corporation, handle donations and handle essential infrastructure and that is what our goal at the Foundation at the broadest level is to provide support. So that these people who are working on all these different projects all they have to worry about is doing code. They don’t have to worry about, "Oh I need to go and buy a new server, how we are doing to deal with this donation or this tax policy." We try to deal with all of that. So I think there is a critical mass that works in our favor.

Scott: Right, right and let them focus on the piece of it that they really enjoy, which is whatever this project is that they have come up with. They have a passion, like you said, for not having to worry about all the housekeeping stuff.

Bookmark this: