How Software is Built » collaboration

Interview with Stormy Peters – GNOME Foundation

campsean — Mon, 05 Jan 2009 02:38:26 +0000

Interviewers: Scott Swigart and Sean Campbell

In this interview we talk with Stormy. In specific, we talk about:

Sean Campbell: Stormy, to get us started, can you give a little bit of your background?

Stormy Peters: Sure. I got into open source software when I was managing the desktop team at HP and I had a team of engineers in India working on the CDE desktop. All we had time to do was fix the defects that our customers were calling in.

This was about 1999, and Linux was becoming really popular. I realized that the Linux desktop did everything our customers needed, as well as other features like OpenOffice, and I thought, “Well, why don’t we just use that one?”

It turned out to be a very easy technical process to port the GNOME desktop to HP-UX. The main issues we had to contend with were non-technical. When we proposed our plans to management, their main concern, initially, was that we were going to accidently copyleft HP-UX.

Pretty soon, I’d worked myself into a new job creating the open source program office for HP with the Open Source Review Board, and policy and strategy. Then I ended up helping HP customers and partners, and I eventually left to go to OpenLogic.

OpenLogic is a software startup that helps Fortune Global 5000 companies use open source software. I helped set up a plan with the community to support the projects those companies were using, as well as to help OpenLogic’s customers set up open source software policies of their own.

Then, back in April, I was at the Linux Collaboration Summit in Austin, and a couple of the GNOME guys approached me to work with them. I thought about it, and I realized that it would be really fun–a great mission and passionate people to work with.

I really like working closer with the community, and the GNOME folks are great people. One thing led to another, and now I’m Executive Director of the GNOME Foundation.

Scott Swigart: I think a lot of our readers will be pretty familiar with GNOME, but in your own words, could you describe the project and tell us a bit more about your specific involvement in it?

Stormy: The GNOME project is very large; you can think of it as an umbrella project with a couple of hundred modules in it. If you use Linux, you probably use GNOME, which is the desktop for Linux with all the windows and the menus and look and feel that you see when you log into Linux. KDE is the other Linux desktop environment, but in most of the Linux distributions, GNOME is the default.

GNOME includes a lot more than just on the desktop environment for Linux. It’s also a development environment that developers can use to build applications that run on Linux, OpenSolaris, or other Unix platforms. It’s also got all sorts of things from web browsers and media players to the ability to play TV or do instant messaging. A huge array of projects fall under the umbrella of GNOME.

Scott: It strikes me that people tend to think of GNOME as just the shell, but it is really a set of other stand-alone, independent applications, as well. Talk a little bit about that relationship between GNOME the shell and all those other projects, as well as what it means for something to be under that GNOME umbrella.

Stormy: GNOME started out as the desktop. That was its primary focus, but the GNOME developers started to find needs like being able to read their mail on Linux, so they wrote Evolution, which is a mail reader. Then they decided they needed an open source browser, so they wrote a browser, Epiphany.

So you see, GNOME’s growth beyond being just a desktop is because the community of GNOME developers continues to define new requirements and opportunities. It’s even a set of applications that run on cell phones and handheld devices called GNOME Mobile.

I asked the question, “What does it mean to be a GNOME project, and how does a project become one?” when I started, and to be honest, it’s still not terribly clear to me.

Scott: [laughs]

Stormy: I’m waiting to see the next project that becomes a GNOME project, to observe how that happens. There are more than 100 projects in there, and I think the core identity of what it means to be a GNOME project is that it participates in the community.

People who work on those projects work a lot with the other GNOME developers, and they follow GNOME values and interface guidelines. GNOME is really good at making sure that everything we develop is accessible, internationalized, and usable by everybody.

Scott: It seems like a fair analogy to say that in some ways GNOME is almost like a distro. How do you manage the necessity of making sure that 100 projects are all ready on the same day? Does GNOME just freeze it at some point and say that whatever projects are ready to be included when a new GNOME release comes out are included, and the others aren’t?

Stormy: You touched on something that GNOME has done quite well in the past, which has obviously been picked up by all the Linux distributions. The Linux distributions are now large companies with lots of customers, and they need regular releases.

For the past few years, GNOME has been releasing regularly on a six-month cycle, which is pretty unusual for an open source software project that’s run by volunteers. The release team decides which applications are ready to be included and what should be included in each release. They make that call, and then they write up release notes that they put out on the web.

Scott: I understand that you can run KDE apps on GNOME and vice-versa. How does that work, if, say, I code a chat client against the GNOME API and you want to use it on KDE? How does KDE support that?

Stormy: When you’re building the client, you get the GNOME libraries, you’ll use the GNOME look and feel, and you’ll use the GNOME tools. When you finish, your application will actually look like it’s in GNOME.

That said, someone running KDE could always download it. They would probably have to download the GNOME libraries, too, and they could run it on KDE without any problem. There’s quite a bit of that that goes on–people using KDE with GNOME apps and vice versa.

Scott: Talk a little bit about what’s in the GNOME core that application developers depend on. Is it libraries for windowing and that kind of basic stuff? What do they actually code into that’s in GNOME?

Stormy: If you were developing on GNOME, you would have access to all the libraries that control things on the desktop: everything from the desktop panel and the menus on the top to how things come up and shut down. It’s the whole user interface for Linux.

Scott: You mentioned a couple of characteristics that give GNOME its identity. One thing you mentioned was internationalization and accessibility, as well as having a six-month release schedule that’s very deterministic. Can you characterize a few other things that you feel really identify the culture of GNOME?

Stormy: The community has key values. It’s really funny, because there are a lot of GNOME stickers that are covered in hearts and slogans about “GNOME love”. It’s a very close-knit community that really focuses on making sure that what they develop is not only beautiful, but it works well, is easy to use, and can reach out to people everywhere.

There’s been a lot of work put into GNOME by individuals and companies to make sure it’s accessible to people that might not be able to see or hear, as well as to people that just might be getting older and can’t see as well.

There’s been a lot of work to make sure it’s internationalized. We’re actually having our first conference in Asia next weekend. I guess to characterize GNOME, I would say that it’s a very caring, very inclusive community that has a lot of fun, but really cares that their product is easy to use and effective for the world.

Sean: Within that framework, could you lay out for us what you see yourself doing with the GNOME project, and what opportunities and challenges lie ahead?

Stormy: My first opportunity and challenge is figuring out exactly where to spend my time. I think of my job as having five parts. One is to be the eyes and ears for GNOME.

Part of it is just to be the person that people can come to, not so much as a representative of the community, but as the interface for the community–a single point of contact as well as someone who attends conferences and does interviews like this one to promote awareness of GNOME.

Another is our sponsors. The GNOME Foundation is funded by donations from volunteers as well as large donations from our corporate sponsors. Part of my job is finding new ones and working closely with existing ones to make sure that their relationship with GNOME is a good one.

Along with that comes marketing. We have a marketing team, but part of my job is to figure out what we want to do with GNOME marketing and help set up the infrastructure so that volunteers can help work on that.

Lastly, it’s just making sure the day-to-day stuff happens. I officially report to the Board of Directors, which is made up of seven volunteers who are elected once a year. These guys are giving their time, and they do a lot of work.

They get frustrated that they don’t have more time in the day, and that once in a while something drops. One of the reasons they hired me is to help get all those things done that they don’t quite have time for, but that they really think are important.

Sean: How do you effectively market GNOME? What are you trying to achieve to give it more visibility? Is there a role for trying to take market share away from the competitor, which in this case I suppose would be KDE, in a sense?

Are you trying to bring more people to the project, or more sub-projects? What do you see as the end goal of marketing more effectively?

Stormy: That’s a really good question, because I think it’s easy to get caught up in how to do it, rather than why we’re doing it. Our goal definitely is not to take market share from KDE.

If we wanted to take market share away from anyone, it would probably be Windows. The community has a really strong value around open source software. They should have free and open source solutions.

Our end goal is to get more people using GNOME and to get more people contributing to GNOME. Probably more than a thousand people work on GNOME. We have 400 individuals who have been officially recognized as contributors and have applied for membership to the GNOME Foundation, but we still have way more than we can do.

I think our end goal for marketing is to have more people see the value of GNOME and use it, and to get more people to contribute.

Sean: In a certain subset of users, it seems that Ubuntu has tried to position themselves as the de facto choice for people who may be considering using Linux. At the same time, they have a tendency to fuzz the bounding box between their efforts and upstream projects. I’m not trying to advance the standard argument that “They don’t contribute upstream.” While I think that’s an interesting argument, I also have the sense that it’s not as legitimate as people sometimes make it out to be.

I am thinking more about the way they package and present their offering, which can tend to take things from other projects and almost leave the impression that it really was their work output. I’m not suggesting that it’s malicious, but I do think that some users get a mistaken impression about the boundaries of the project.

Do you feel that they’re doing a service to GNOME, since they’ve standardized on it? Is there a tension between whether it’s maybe a good short term plan for them to fuzz those distinctions, but maybe not a good one in the long term? Or do you think it’s just good overall?

Stormy: I think that whether it’s good or bad, how they are doing it is often controversial. That said, Canonical is a sponsor of the GNOME Foundation and they have members that are present in the community. I think they are trying to give credit to the community, and in fact, they have asked me for a quote for their next press release.

They have other projects within the distribution. That said, I think they’re definitely trying to present it as one thing to the end user, I imagine probably with the idea that it’s really confusing to end users to get 39 projects instead of one.

I think it’s a balancing act that they’re working out between building a composite offering–which by all means, open source projects want them to do–and giving credit to the community and participating upstream.

Sean: I agree. I think it’s a delicate balancing act. After all, Windows doesn’t ask you to think about which volume shadow copy version is embedded in it, or make it part of the decision making process to actually purchase the product.

Stormy: One of the issues that we have is that all of the distributions take GNOME and then they brand it. They brand their distribution with their logos and their brand. We’re totally OK with that, but in the process it removes all of the GNOME logos.

We have considered rebranding guidelines that are sort of analogous to the “Intel Inside” sticker. For example, we might say that the distros can replace all of the logos and all of the splash screens except for one or two specific ones, which we would ask that they leave in place to retain some of the GNOME identity.

Sean: That’s an interesting problem–how do you retain your marketing and your branding with an open source project? It’s an extension of the standard discussion of how an open source project should avoid getting cannibalized, so to speak, by somebody taking your efforts and moving ahead on them without you.

Your idea of the Intel Inside sticker makes excellent sense. It’s not in the way, but yet you always know it’s an Intel laptop because of it.

Stormy: Right, and maybe the top menu has the little GNOME foot logo.

Sean: Exactly. Do you think that the community will be open to those efforts over time to help grow strong projects, or do you feel that there’s always going to be a tension in that area?

Stormy: Well, I think we’ll work it out. I haven’t floated the idea of leaving the GNOME logos in certain places past the distributions officially, so I don’t know what their response will be yet. I think they would be open to it, as long as it didn’t detract from the overall user experience. Of course, we don’t want to detract from the overall user experience either.

Sean: Right. It’s not like you’re going to have a ten-second Flash animation come up saying “You’re using GNOME!” [laughs]

Notwithstanding the fact that I don’t want to try to position you as competitors, I think it’s obligatory that we ask you what you see as the particular strengths of GNOME over KDE.

Stormy: First, it’s important to affirm that we’re collaborating with KDE, rather than competing with them. For example, they have an annual conference called the Akademy, and we have an annual conference called GUADEC. We’re actually going to co-locate with them next year so our developers can meet each other and collaborate in areas that make sense so that we’re building on open source and not reinventing the wheel every time.

We do have different focuses and different looks and feels, and our communities have slightly different values. We both value free and open source software, but the focus is on different areas, which enables us to reach different people. We are certainly looking to collaborate more with them, not compete more with them.

Scott: What do you feel GNOME’s strengths are compared to Windows?

Stormy: That’s a good question. Our obvious weakness is that everyone knows how to use Windows, and when you’re used to something, it’s hard to switch.

That said, I think GNOME has a lot of strengths in the ability to customize your own look and feel. You have lots of skins and lots of different applications you can apply. A lot of developers play with things they think are really cool. When I use GNOME, it just feels more friendly, if that makes sense.

You also get updates much more frequently, which some would say is bad, but when there’s something wrong, somebody’s working to fix it.

Scott: Looking at it from the outside, it’s obviously a great benefit to people who are packaging distros that they’re able to customize it and use GNOME as this great foundation for a broader user experience.

Another thing that I hear a lot is that if a user has a problem or a question, they can get on a mailing list and potentially be talking to the person who wrote the code. You don’t typically have access to that type of scenario with closed-source proprietary software.

Stormy: It also comes with a lot more than Windows does in terms of applications, like music players, CD players, instant messaging, and financial calculation programs that save you cash. You don’t have to pay extra for Outlook, for example, because it comes with Evolution.

Scott: Talk a little bit about the interaction between users and the people working on the project. Many open source projects are “by developers, for developers,” or in the case of projects like Apache, many are by developers for very technical people.

With GNOME, on the other hand, you have a far less technical user base, especially as Linux-based devices like netbooks proliferate. It’s potentially hard for a really technical developer to put themselves in the mindset of a non-technical user.

What do you see as helping to raise the awareness on both sides and to facilitate communication between those highly technical folks and non-technical users?

Stormy: I think the first thing is that the GNOME community really wants to hear from users. In the annual conference we have called GUADEC (the GNOME Users and Developers Conference), we try to attract end users, although that’s always a challenge.

We’ve also been talking about doing usability studies. We’ve done some in the past, and we’re hoping that our corporate sponsors can help us figure out how to do more. We’ve also talked about doing case studies.

The City of Largo uses GNOME. Dave Richards, who’s an administrator, came to our user experience test this week and talked about why they like GNOME and what they have troubles with. He shared his experience of using GNOME in a work environment full of not necessarily technical people.

I’ve heard his talks quoted from several people already. People really paid attention to what he had to say, and the GNOME developers as a whole are very interested in talking to end users.

One obstacle is simply the tools that users versus developers use. When users have questions, they usually end up on forums. When developers look for feedback or ask questions, they go to mailing lists. It’s really a question of how best to find each other and to talk to each other.

Scott: Right–users aren’t necessarily versed in Bugzilla. Let me flip it around, then. GNOME obviously interacts directly with the users, but the other side of that is interacting with the distros. In a perfect world, how does the relationship between a project like GNOME and a distro work?

Obviously, they need to cater to their users, and they want a certain experience. They’ve got a certain culture, and they’re probably doing certain customizations, which might involve the stuff that you can just configure in GNOME, or maybe it even involves forking it in a small way.

How does that relationship work in a perfect world between the distros who are getting questions posted on their forums and an upstream project like GNOME, and what are some of the places where occasionally the wheels come off?

Stormy: GNOME’s actually a lot closer to the distributions than they are to our end users, because the distributions are the ones that take GNOME to the end users.

Almost all of the major distributions are sponsors of GNOME, and they sit on the GNOME Foundation Advisory Board, so we get their feedback at the foundation level.

Also, many of the distributions employ GNOME developers. A lot of the GNOME contributors get paychecks from the distributions and are involved in their plans at work.

There’s a strong relationship, but there is also room for improvement in the way things work. For example, a lot of times GNOME releases and then the distributions pick it up. They had some of their developers working on GNOME before it released, actually contributing stuff, but some distributions make changes downstream, and we would like to see more of that work happen upstream.

On the other hand, that’s often really just a function of their business model. If they have a customer that asks for something or bugs that need to be fixed, they fix it first and then eventually it works its way upstream.

Scott: That’s similar to what we heard from Shuttleworth over at Canonical, who basically said they were going to work harder at doing more stuff upstream.

Stormy: They just hired a team of designers, and I’m hopeful that those designers will work with our upstream developers so that the developers get design feedback before they actually write a lot of code.

Sean: Many discussions happen around good design and good usability, and a lot of people say that exceptional design typically comes from a flash of inspiration from an individual, like Steve Jobs saying, “I shall birth the iPhone.”

What do you say about this notion that inspiration is led by a benevolent dictator, given that your user interface is essentially driven by a community? How do you get a mass community to come up with those flashes of inspiration, given that a certain segment of people says the only time that ever happens is when one person leads the charge?

Stormy: I’m still learning how it all works. GNOME has human interface guidelines. We have a designer’s mailing list and an IRC chatroom where developers can go ask questions, and hopefully there are designers online who will give them feedback right away.

We also put a lot of focus on the design, like the user experience hackfest that’s going on this week. We’ve gotten together a whole group of GNOME developers in the same place to talk about the user experience, how we can make it better, what areas are issues, and what our vision should be for the next couple of years.

I think it happens not so much by one person working by themselves, but more from a lot of conversations happening between people.

Scott: What’s really interesting from your perspective that we haven’t asked about?

Stormy: Two things. First, I knew before that open source software projects happen by a lot of volunteers doing work, but I didn’t really realize how much work they do. I’m tremendously impressed by how much volunteers contribute to and run the project.

The volunteer Board of Directors keeps the whole organization running. They answer tens of emails a week about funding, strategy, and operations. GUADEC, our annual conference that has between 300 and 500 people every year, is completely run by volunteers.

An amazing amount of good work comes out of these volunteers, many of whom have other full time jobs. I had a sense of that before, but I am much more aware of it now, and I’m even more impressed.

My second addition is that we haven’t talked much about GNOME Mobile. Open source software has been in the mobile industry for a while now, but it’s really starting to pick up, which I think is pretty exciting.

I went to the Maemo Summit a couple of weekends ago in Berlin. There was Linux and open source GNOME running on Nokia tablets. Some of the projects that people showed were just phenomenal. There’s everything from medical stuff, to music stuff, to note taking with a pen in a way that you could search and scroll really easily. Everyone did their presentations from their tablets using all open source software on Maemo.

Everyone was waiting for Linux to be on the desktop, and now with netbooks–with the Eee PC, the Mini and those type of devices–Linux is an important presence there that will only grow. I think the next movement we’ll see is with cell phones and devices, and there’s a lot of interesting work to be done there in the open source community.

Scott: We interviewed a person who was on Mandriva recently who said something that really resonates with what you just said. He suggested that Linux is never going to take over the desktop the way people think of it, in terms of going to Dell and buying a desktop computer with Linux instead of Windows.

He suggested instead that the desktop isn’t really going to be what people have thought of it being. It’s going to be a lot more about this merger between netbooks, mobile devices, and that kind of stuff. We’re talking about different form factors and different use cases, aiming more for where people are going than where they’ve been for the last decade.

If you don’t mind, expand on that a little bit from your own perspective.

Stormy: GNOME is absolutely working on another front. We have the traditional desktop, whether desktop is an overused word or not. But we also have projects like the online desktop, which is trying to merge online applications in the cloud with your desktop. We have the GNOME desktop on Nokia tablets, cell phones, netbooks, and other devices.

GNOME technology is used on a device used in classrooms that can test things like temperature, altitude, or pressure. Instead of having a whole computer, they just have a little test device that gives them everything they need right there.

I think computing is definitely changing. It used to be that you carried your laptop with you everywhere, and now with a lot of the smartphones, I see businesspeople actually leaving their laptops behind completely. They’re working entirely from their tablet, their iPhone, or their cellphone.

Scott: It seems like people are working on stuff that’s going to be really interesting to general consumers a year or two from now. Those of us who are used to carrying a laptop and sitting at a desktop aren’t necessarily thinking of some of those scenarios that are enabled by a mobile phone and high speed network connectivity, whether it’s 3G, 4G, stuff up in the cloud, or whatever.

Walk us through a few stories of the world of the future that you know people are working on.

Stormy: I don’t have any “predicting the future” stories to walk you through, but I can give you my feel for where it will go. When I’m walking around, I get really frustrated when someone mentions something that I want to Google but I can’t.

Or, we’ll be at lunch, and I’ll be trying to tell you about someone I really think you need to meet with, but I can’t just pull it up on my phone. I could if I had my laptop, but I don’t.

I think a lot of people feel like, when they’re walking around not connected, half of them is missing. I think that in the future, it will be much easier to stay connected to all of your knowledge all the time.

Scott: A lot of devices that try to solve these problems have done so in really awful ways, and so people don’t really use them. For instance, a mobile phone manufacturer might want you to record a voice memo for yourself, but that isn’t really what you want.

The iPhone has made some really significant strides in solving those issues. It’s the first time in a long time I looked at technology and thought, “That’s just magic; I don’t know how that works.”

For example, there’s an app that samples a song playing on the radio and gives you the name of the song, the name of the artist, a link to download it from iTunes, and maybe a link to a YouTube video of the song in performance.

Stormy: Right, and I know I wanted that five or ten years ago. I’d be driving around in the car and I’d be like, “What song is that?”

Scott: The things that are the most interesting to me are where two completely independent technologies converge, and all of a sudden whole new things are possible that weren’t really possible before.

The iPhone’s multi-touch screen is relatively large, but still a relatively small form factor for the total device. You don’t feel like you have to carry it under your arm; you can just drop it in your pocket. When you combine that with high-speed connectivity, all of a sudden there’s a lot of stuff that becomes feasible.

What do you make of Android versus Limo, versus whatever some of the other distros are that are trying to scale down to handheld devices?

Stormy: From the GNOME perspective, most of them use a little bit of GNOME technology, and our focus is on working with all of them to provide the most open platform that we can.

We have a set of technologies that we call GNOME Mobile for the whole range of those different devices. I went to the Open Source and Mobile conference in Berlin a couple of weeks ago, and I was really struck by the range of open source knowledge in the people there, from very little knowledge by some people to very high levels in others.

I sat next to three people from a very large provider company. They didn’t know what GNOME was and they weren’t Linux users, but there they were at an open source conference. The carriers and operators are now beginning to get into open source because of movements like Symbian and Android.

Whereas before, they really didn’t have to worry about the software because the device manufacturers took care of it, open source and new initiatives are enlarging the ecosystem to include the carriers, the operators, the device manufacturers, and the software companies.

They’re all playing in the open source software space, and I think that that means there’s a lot of opportunity to enhance and evolve the mobile model.

Scott: So many people are focused on where we’ve been, but what you’re talking about is really where we’re going.

Where do you see the difference? Like you said, it used to be that only the handset manufacturer had to worry about the software, and now it’s the carrier. What are you hearing in terms of, “As a carrier we’re really interested in this” whereas, “As a handset manufacturer we’re really interested in this”?

Are they interested in the same things? Is there a lot of overlap? Or are they really looking at the issue from two completely different perspectives in terms of the software that they would be interested in?

Stormy: I haven’t talked to enough carriers to be sure, but my sense is that they’re not sure exactly what they’re interested in yet.

What I think is interesting is that they’ll both have to focus more on the end user. It used to be that the software came with your phone. You almost picked the brand of your phone based on whether or not you liked their software. I remember switching from one phone to another and being really upset that the menus were different.

Now it’s not just the menus, but there are all sorts of applications I can run on my phone. Those applications aren’t dependent on either the cell phone company or the operator. Now those applications are in a space where they both have to collaborate, and end users can participate as well.

I think that the vast majority of cell phone users will never care about that, but some set of users will care that they’ll have much more choice on what applications they put on their phone. The operators and the cell phone manufacturers will work more closely with end users, and I think we’ll see much better applications. The cell phone companies have to guess at what we want.

Scott: It seems likely that the device manufacturers are going to care mostly about having a basic OS and shell, and some basic apps on the device itself. On the other hand, carriers are going to try to think of a lot of value added services that are only available through them as a carrier.

Whether it’s speech activated search like Sprint has, or streaming television to the phone or whatever, you get it because you’re part of that carrier’s network. They’re pushed into writing end-user application software that ties into their network in a way that they never really had to, or necessarily had the capability to do before.

In that light, it is very empowering to the user if they can just go to the package manager on the phone and say “What can I get?” This is basically a Linux device, and there’s a bunch of stuff that they can just pull down, independent of what the phone manufacturer or the carrier says.

Stormy: Right, and that kind of empowerment of users is a key benefit of open source.

Scott: I really appreciate you taking some time to chat with us.

Stormy: It was interesting as always.

Bookmark this:

Interview with Justin Erenkrantz – President – Apache Software Foundation – Part II

campsean — Thu, 31 Jan 2008 16:00:02 +0000

Interviewers: Scott Swigart and Sean Campbell

Interviewee: Justin Erenkrantz

In this second part of a two part interview with Justin Erenkrantz we talked to him about:

Scott Swigart: In this part of the interview, we wanted to dig into some of the tenets, if you would call it that, of the Apache way. And the first of those of course is collaborative software development.

So talk a little bit, if you would, about how Apache does collaborative software development. I’m sure some things are very traditional and similar to the way that other open source projects might do it, and there are probably things that also might just be a little bit unique to Apache. So how do you try to insure good collaboration?

Justin Erenkrantz: So the main center point of all of the collaborative software development that we do in Apache is the mailing list. That’s where pretty much everything happens. As one of the guys mentioned before, the maxim has been: "If it didn’t happen on the mailing list, it didn’t happen." And that generally tends to be true.

Basically, if you follow the mailing list for a particular project, then we’re expecting that you should know what’s going on within the project. Within that, those are pretty much all public lists. Everybody can subscribe, even people who are committers, people who are just users, people who just work on another project that may consume the Web server or maybe PHP modules or something like that.

It’s pretty much an open forum. Anybody can voice their ideas. Generally though, just the way things work, most of the traffic tends to be from the core developers who are active at that time. The traffic patterns of the list change, and you get an idea of how much discussion. You generally see peaks and valleys for the mailing list discussion. Things get really heated or things are just chugging along and there’s not much traffic on the list.

So that’s where all the discussion should happen. And then of course there is the source code repository. And in Apache we always had the thought of having shared repository. A lot of projects now are starting to ‑‑ Git and Mercurial and all of these distributed version control systems to be centralized version control systems.

And that’s something that, within Apache, that goes against what our thoughts are, because we want to all be agreeing on, "This is the Apache version."

Scott: Sure, no problem.

Justin: So there’s no leader within the ASF. There’s no person, if you look at say, Linux and you say, "This is Linux’ tree," or this is Andrew’s tree or this is Alan’s tree. Instead there is just the Apache tree. So there’s really no concept of, "This is Justin’s tree," or someone else.

Scott: Is the feeling then that essentially there should just be intensive discussion before something gets checked in? So when you’ve reached a point of consensus, it should get checked in rather than the way other projects work where different things get checked into different people’s trees. And then when it’s time to build a release, you have to pull stuff from these different sources to figure out what’s going to be in the release and what isn’t.

Justin: Right. So, generally, what tends to happen is there are two states a tree can be in within Apache. One of them is "commit then review." And then there is "review then commit." And you’ll see some projects differ on particular trees.

For example for the HTTP Server, the trunk ‑‑ which is the main development ‑‑ is usually always under the "commit then review," which basically means that anybody who has commit access can feel free to go and make changes and they basically have the benefit of the doubt that the change is going to be good. And there’s generally an implied threshold: that if you’re going to make a really big change, go discuss it on the list. But if it’s a minor change or adding a little feature, that’s probably not going to be controversial, go ahead and commit that to trunk.

But for our stable releases, generally things that have already been released and we’re doing maintenance on those, are under the "review then commit" model. So that’s going to be RTC, and that means that any change to those trees has to be pre‑approved. That means you need to get three binding votes from other committers to say, "Yes, this is good change and no one has vetoed it." Technically you would use a file called STATUS, just a plain text file that some projects will use, that basically tracks all of the things that are under discussion to be back‑ported or added into this tree.

Scott: Got you. Like any open source project, and this one is democratic, there is a vote to commit. Well, there is a vote if it is a release product. If it’s not a release product and people later decide it wasn’t a good thing to do, it can be reverted out.

And you mentioned before that part of your governance is that out of these 60‑something maintainers, one of them can essentially veto a change if they want to. So talk a little bit about how conflict resolution usually works. For example, when you have people who are—and I understand it doesn’t happen often—adamant one way and another person who’s adamant a different way. How do you see it play out that those eventually get resolved and things move forward?

Justin: Generally what happens is, like before, the veto just tends to be a last resort. So let’s just say that someone makes a change to the trunk and I don’t like it. I might just say, "You know, we should talk about this change," or, "Here’s the problem with this." Generally, the person who has committed that says, "Oh, yeah. Here’s why I did it this way," and comes up with an explanation, and then through a process on the mailing list, figures out and resolves those conflicts.

That’s what you tend to see happen. And it’s all tends to be, for the most part, "I’m sorry, I forgot about that particular corner case or that." And everything tends to get resolved very naturally.

The veto tends to be when someone says, "No. I have to do it this way," and someone else says, "No, that’s wrong." That’s basically when the community is at risk of breaking down. But generally it doesn’t get to that point. Everybody is, "We’re all going to go in this direction; this is the right direction for us to go in. We want to add this feature. Let’s work through whatever issue you may have about this particular commit."

Scott: One last thing before we move on to the next point. Other than the fact that everybody has their own private tree, is there anything else about the collaborative nature that you think is somewhat unique to Apache?

Justin: We tend to do a roll call before the release. So at this point there’s been a review of everything. But then let’s just say that I want to release the next Apache 2.4 or whatever. Then basically what I will do as a release manager is say, "OK, I’m marking this as 2.4," and I produce all the artifacts. I produce the tarballs, [inaudible]generated files, whatever. And then I send it to the list and say, "Hey, is everybody happy with that?" And then that goes to a voting process.

There’s review at all stages, but there’s also a review at the point where you do a release, and you have to get, at least, three people to approve a release. One thing that’s different is that it’s not possible to veto a release. It’s strictly majority rule on the release.

Scott: So in other words, there is veto capability on the individual check‑ins. But, as you said, when it comes down to doing a release it is a majority rule vote.

Justin: Yes. You’ll tend to see someone voting ‘no’ on a release if it doesn’t work on Linux or something, and generally you’ll see it get stalled and it then gets fixed. But there have been a couple of cases when we did the release even though we knew that it didn’t work on a particular platform, and so we made a release note. But the veto does not apply to releases.

Scott: Interesting. So moving on to licensing, one of the key things with Apache is the commercial‑friendly standard license. Talk a little bit about what that means.

Justin: Basically within Apache, we like to have a big tent where everybody can come in and play with us. We think that the community that we developed within Apache is going to be the motivation for you to stay involved. For example, within the HTTP Server community, you have all these experts and Web servers, and if you’re part of this community, you get the benefits from them. And so there’s an incentive to play within the community, so that I don’t have to hire five guys and do a whole team; I can leverage the other people within the community.

But in turn, all those people who are part of the community say, "Whatever you want to do with the code is fine. We’re not going to get hung up if you make it a commercial product or an open source project. We created it and it served our needs, and if it serves your needs, that’s great."

Scott: What springs to mind are things like GPL. Am I seeing it right in that Apache is more commercial‑friendly than GPL, V2 or V3?

Justin: There are companies built around GPL licensed software. But what we tend to see are two classes of GPL products: In one, there is a real community, maybe within Linux, and they’re all happy to make all their changes available to everybody, and that’s a very good community. The other community you tend to see has a single stakeholder that has a prevailing interest in the GPL product, and they basically have an unfair share.

You can see this with some of the GPL projects that require copyright assignment. In order to participate, you license your changes in the GPL and you have to give a copyright assignment to the principal stakeholder. Now they are then free to release the commercial closed source based on your work because they have the copyright or whatever legal mechanism. There is an imbalance there when you look at those two.

Generally when you think about the GPL, you’re divided, with broad strokes, into those two groups. This is a real community but the other groups are aware and want to be clear about which one has a dominant role, and that’s one thing within Apache we don’t like to see. As our projects go through incubation and get added to the Foundation, one of the things we do is make sure that the community is diverse. In fact, there is not a single dominant stakeholder that can direct the project in any untoward way.

Scott: Right. I can make changes to it, distribute it as part of a commercial product, and I would not be required to contribute those changes back to Apache. But under a GPL license, any modifications made require you to make the source code available. You cannot have closed source proprietary extensions or modifications of it. Any modifications you make, you have to open source and it has to be under the same license. So that’s the key differentiator?

Justin: Yeah, our philosophy is that the community is what’s going to bring you and keep you there, and that’s why you’re going to stick around. If you released a commercial product around one of our projects it’s going to be to your benefit, to basically keep your commercial project as close to whatever we’re releasing.

Scott: Right.

Justin: You can pick up all the bug fixes and whatever improvements; you get those as a free rider. But in a sense, you are contributing whatever changes you’re making voluntarily back into the greater community.

Scott: Yeah, that makes sense. Do you have examples of companies that have used different Apache projects because of the commercial‑friendly licensing, where they probably wouldn’t have it if the license weren’t so commercial‑friendly? Is that a topic that comes up?

Justin: Absolutely. You see companies like IBM that release their versions of the Apache Web server or Geronimo under different names, but in the core, they are Apache projects. We’ll see that even with smaller companies such as Covalent that does commercial support. Basically, they added in a couple of extra things that provide support to their users.

One thing that the Apache community really does not focus on providing is 7/24 support. Covalent goes in with their business model and provides the support and training around these particular Apache projects. You will see businesses like JBoss using Tomcat. So you see all of these commercial companies using things that are Apache projects under the covers.

Scott: Right, so that freedom has led people to be a lot more creative about how they structure their business. They have a lot more options in how they participate with the different Apache projects, how they contribute back and how they structure their own products. What is the relationship between the Apache Software Foundation and the Free Software Foundation? Is there any or are those fairly separate endeavors?

Justin: There is no formal relationship. I’ve never had a conversation with Richard Stallman, but I’ve had conversations with Bradley Kuhn who used to be, at that time, the Executive Director of the Free Software Foundation. So there’s an informal get‑together of foundations to compare notes, and that’s generally a very good thing. How do we keep our ears open to what Mozilla’s doing? If they’re doing this new technique, then we can give them a call and ask, "What are you doing? We’d like to follow on it."

One thing we’ve been doing with the Eclipse is a joint conference. There’s going to be a conference in Asia that’s now scheduled for 2008. So it’s a way for us to get the communities talking to each other.

Scott: Sure. So basically, if I can summarize, you guys get together around joint events and joint things where it makes sense. You share information because you’re all part of the open source community. Philosophically you may agree to disagree in terms of the details of licensing, commercial friendly, and that kind of stuff.

Justin: Well and you ought to be using more…projects have different circumstances. Apache‑‑we have a very vocal membership and we have this and you compare that to let’s say the Mozilla which has a completely different governance structure. But if you look at Brian Behlendorf, he’s been on the Mozilla board for a very long time and he was one of the founders of Apache.

Scott: Gotcha.

Justin: So you have this intermingling of the communities. So someone like Brian Behlendorf who was brought in through the Mozilla and says here’s how we did things within Apache, and here’s his expertise and his experience that he got, he can share that with the other people within Mozilla.

Scott: Gotcha, gotcha.

Sean: Let me refer to one of the other tenants of the Apache way. I’m curious about this just because I was thinking about the conversation we were having. To state the obvious, you’re focused on producing software. I notice that you have a security committee that—if I’m reading it right and for lack of a better phrase—provides a service to all of the projects that are part of the foundation. And it looks like those projects can turn to the security committee and ask security related questions or possibly look for guidance from them, regardless of whether they’re Tomcat or some other piece of the foundation? Is that accurate or is that not accurate?

Justin: It’s somewhat so. We have a security team, which I believe is currently a Board committee. But basically what they’re responsible for doing is ensuring our security at Apache.org mailing address gets responded to. And these are generally people who are very security savvy.

But there tends to be some people from Tomcat, from the HTTP Web server, from a higher profile project on this internal mailing list. So let’s say that, to give you an example, let’s say there was a security vulnerability in Derby and they could parse to those reps and say, "Hey, we have a security vulnerability. What do we do?" And so there’s expertise and, "OK. Here’s what you do. Here is your administrative contact. Make sure your mailing list… Go talk to…" Kind of a shared resource. But we’re not getting the focus on producing the fixes for the project but it’ll be "OK, here’s the responsible disclosure policy and an attribution policy." So that’s generally what their role is.

Sean: Are they providing fairly prescriptive guidance but just not down to the ‘I’m going to change you’re code’ level because they don’t know the individual projects at that level? Would that group essentially be the center for discussions around a security development lifecycle for the Apache Software Foundation? And an attempt to pull those best practices together?

Justin: Yeah. I think basically our project concern…we have something. What’s the process? What do we do? And that’s as an advisory role. OK, here’s the process and the procedures to follow.

Sean: But it’s purely advisory, right? I mean one of the projects where they feel that their code is "secure enough", or they’ve looked at it long enough or they feel that they’ve handled it. Then the advisory committee comes back and says, "Well, we really think you could take a look at this again." That’s where the communication would stop and it would be up to the individual project whether they want to take that under advisement or not.

Justin: Yeah. I think so. I think record security, can you maybe at that point write back to your original reporter and say, "We looked at it and we don’t feel there’s a security vulnerability here." That may be…that has happened where we look at things and we say, "No. This is not an issue." But generally, really the security team is more of a reactive. So they’re not proactively performing security analysis on our code or anything like that.

Scott: I just want to clarify. It sounds like they have a little bit of an all‑up policy for somebody sending an email to that address; somebody reports what they perceive as vulnerability or reports some kind of issue. They do a little air traffic control. They route it to the project.

Justin: Exactly, exactly.

Scott: There’s a general process that the different projects would follow. Basically that sort of happens and that’s one of the things that the security group advises the other projects on. Well this is generally a ‘way we do it’ sort of thing.

Sean: Let’s go to a different piece of the Apache way, the emphasis on a technical‑based interaction. One of the things that I find fascinating about open‑source projects is the way that they exorcise community members that maybe aren’t following those rules.

[laughter]

Sean: Because we got some interesting responses when we talked to people about it. It’s like, fine. We understand that everybody is an adult. We understand that everybody will try to handle themselves in an appropriate manner. But if anybody’s worked on a software project they know that not everybody does, right? So…

[laughter]

Sean: Considering that you can learn a lot from a story…if you’ve got a story or two about either fully pulling the ejection handle on somebody that would be interesting to hear? Or a scenario where it just took serious counseling to get somebody pointed in the right direction.

I would be curious to see how you guys handle that. You obviously have procedures in place. But at times you have to go beyond those with some amount of intervention and I’m just curious how that played out.

Justin: Yeah. So there’s one case that comes into mind but I’m trying to reserve the right to figure out how much of this has been disclosed.

Sean: Yeah, Sure

Justin: So I’ll tell the story and then leave people’s names out of it but I have to go back and see how much of this has been told. So recently within one of our larger, well‑known projects, there was a bout, to use the word, between two committers. And they basically ended up vetoing each other on everything. It’s like no, no, no and tempers got flared. And it got into a very unhealthy situation. They’re two very strong‑willed individuals.

And basically what happened is the PMC, so it was the PMC responsible for this particular activity, basically had to step in and say, "OK; we need to come up with some policy or come up with some new rules to get everybody back to ground zero. So it wasn’t a matter of ejecting anybody. It was never really an option that…basically what happened was they said, "Here are the ground rules. Here is…if you’re going to go do this you have to follow this set of rules. If you’re going to go do that, you’ve got to go follow this set of rules." Basically the community agreed to say we’re going to go and we’re going to voluntarily adopt these rules. But as a settlement process; lots of flames and a lot of innocent people getting…

Sean: Right.

Justin: …accused of things and that process. The other one that’s in our history was a project called Avalon. And this is one that’s definitely well known so this won’t be any issue about this one. Avalon was a container framework. And what happened was two individuals just did not get along. And they were ending up in what we would call a commit war, where they would basically be reverting each other’s changes as soon as they came in. And it is just this whole really poisonous environment and basically in that case, the community wasn’t able to deal with it. And so basically what happened there was they fractured.

And so that one of the individuals went off and he took his code and you know, we wished him luck and said have a nice life and he went off and then it was kind of some other people came along and they did a project called Excalibur, which was basically the remnants of this whole Avalon project. You will just see if you look at the mailing list traffic, you will just see this giant peak and then this sudden nothingness because the project got shut down because no one could play well with each other.

Sean: Right.

Justin: By the way what it is interesting is some of the veterans of Avalon, they really got involved in the greater Apache community, one of our new directors this year. He was in the middle of all of this, but during that whole experience, he was one of the people trying to keep things level and stuck around and this year he got elected to the board of directors.

Sean: Since Apache is a large foundation, I’m curious about a different point. If you were a closed source company and you feel you are short on testers or short of security experts, you simply go to HR and put out a requisition and hopefully some good folks come back.

So has the foundation had to answer requests from projects where they say, "OK, look, we think we are geniuses on nine out of 10 of the things we need to do, but this one thing we really need people to help." How does the foundation help with staffing up a project in this type of case?

Justin: It is more bottom‑up than that I think in the sense of the culture that we have. You will see some overlap between the HTTP Server committers and the Tomcat committers because you will see that ‑‑ sometimes the Tomcat committers came over and they say, "Hey, we need some help with HTTP." Well lucky us, we have some of the world’s foremost experts in HTTP server, and that basically got them within the communities.

So I think our community’s diverse enough to, "Hey I am looking for a person who knows SQL." OK, I am going to go on the Derby mailing list and say, "Hey, I need some help with SQL" or for something for build systems, I’ll go to Ant community. And there will be some of the people who are the foremost experts in that. That’s actually one thing of having such a large diverse community is that you can pretty much find someone who understands something about something somewhere within the Foundation.

Sean: I figure that’s one of the advantages. You’ve got a massive talent base but at the same time it is segmented into the project, so the Foundation can help orient a little bit of that knowledge of where the talent base is.

Justin: Yeah and as I said if you look out the social graph and it is a weird mix of people who are committers on Cocoon, maybe committers on Mina, then maybe on Gump and so you will see that developers themselves, the committers aren’t necessarily staying in their silo, there are some who do, but there are also just as many who will go to other communities and work on other projects.

Sean: Well I have couple more questions but Scott can go ahead. I want to give you an opportunity to jump back in.

Scott: So talk a little bit about standards because one of the other tenets or pieces of philosophy is faithful implementation of standards. Talk a little bit about what that means for Apache?

Justin: Right. So this was initially when we started off there was HTTP and there was the IETF standards and that was when you have editor of the HTTP standard is one of the people behind the code base, there is the knowledge is going both ways in a sense is that we are that able to influence the standards process. But at the same point we also have some of us involved with the standard process and feeding those changes back and to the development and supporting those standards.

But since then, the initial, you see our participation within some of the key web server specifications, then probably most importantly our participation in the Java Community Process and that is, as you know, we have so many Java projects and that so many of our projects are implementing some JSR specification and our involvement within the JCP has been to ensure that we can implement the specifications and we have projects have representation on these expert groups that device these standards.

Scott: Right. So it isn’t just like the standard shows up and then you figure out how to implement it. There is this two‑way street where you are shaping the standard because you guys have such a big, real world implementation. And meanwhile, the standard is telling what you guys do because they have their own stakeholders, but they are considering your recommendations and you want to conform to what they eventually approve.

Justin: Correct.

Scott: Yeah, I don’t know. I don’t have anything else specific. Sean, do you?

Sean: No, not right now. I think this led us into some new stuff and from our end, we really enjoyed chatting about it. Justin, do you have anything you’d want to add or things you think we should address overall based on the theme of where we were going?

Justin: No, I mean you did a good job of asking me the questions.

Scott: I guess there is one final question. When somebody is starting or has an open source project, they can pick the license they want they can do what they want for their community, things like that. What makes people want to be, in your mind, an Apache project? What is the draw I guess?

Justin: Well, I think from my perspective, the draw is we handle a lot of the mundane governing structures and all this and the infrastructure and the licenses. And that is all essentially managed and I think you see a lot of open source projects like, "Oh we need to go get a foundation."

And that’s a lot of overhead, there is a lot of overhead to create a corporation, handle donations and handle essential infrastructure and that is what our goal at the Foundation at the broadest level is to provide support. So that these people who are working on all these different projects all they have to worry about is doing code. They don’t have to worry about, "Oh I need to go and buy a new server, how we are doing to deal with this donation or this tax policy." We try to deal with all of that. So I think there is a critical mass that works in our favor.

Scott: Right, right and let them focus on the piece of it that they really enjoy, which is whatever this project is that they have come up with. They have a passion, like you said, for not having to worry about all the housekeeping stuff.

Bookmark this: