Interviewee: Jim Grisanzio.

In this interview we talk with Jim. In specific, we talk about:

• The process of opening a huge codebase to the public
• The unique place of OpenSolaris in the open source community
• Lessons learned from earlier open source projects at Sun and elsewhere
• Trading lessons learned back and forth with the Linux community
• Regional differences in the role of open source technology in Asia
• The reasoning behind Sun’s products and technologies that compete with each other

Scott Swigart: Jim, tell us about yourself and your role at Sun to get us started.

Jim Grisanzio: I’ve been at Sun for about eight years now, in multiple roles, mostly focusing on project management in one way or another. You can read my bio at blogs.sun.com/jimgris/page/bio.

The first three and a half years or so, I was in a communications role at the corporate level, doing lots of outreach to the media and analysts, focusing mostly on Sun’s software businesses, including Java and various open source projects.

I started to drift over to work more with the developers and engineers, because they were closest to the action and some of them are excellent spokespeople. It was very interesting to go into the core engineering operation of the company and get those people in front of the media and the analysts, both for competitive undertakings and just for plain informational efforts.

I enjoyed interacting with those very technical people a lot, and as OpenSolaris project was created, I decided that I really wanted to dedicate myself to it. So then I was faced with the task of moving from marketing into Solaris kernel engineering. That’s a tough move.

Scott: [laughs]

Jim: It took a lot of interviews, tapping those very same engineers I had been working with previously. I eventually came up with the idea of focusing on communications among the community. Communities are large numbers of people, and they’re distributed globally.

To get a project started, there’s an important role to be filled in focusing on that and that alone. You already have plenty of engineers, but you don’t have plenty of community people.

So, that’s how we crafted it, initially, and it worked out very, very well. I came over from the marketing side to the engineering side, I started off in a project manager role, as the community manager. I’ve been on the project ever since.

I was actually on the first team that opened up all of Solaris. We were charged with creating the program itself. Now, four years later, everything is distributed and it’s a very, very different place.

Scott: Linux is on an ascent; you’ve got IBM with AIX, you’ve got HP with HP-UX, you’ve got Sun with Solaris. And Sun made a fairly unique decision to take this proprietary Linux and open it up; in fact, Sun is obviously doing that across the board with all if its products–making Java open, and even making hardware specs open.

How do you even start on that? I’m not sure people realize the complexity associated with open sourcing a product, and you even hear things like, “Microsoft should just open-source Windows.” I think what people don’t necessarily realize is that with proprietary software, you probably license technologies from other companies. There’s a whole bunch of encumbrances in the software that prevent you from just posting the code to the web the next day.

Plus, as you mentioned, you want to build a community around it. You want there to be a way that people can be involved. So, how do you even start down that path of taking something as big as a proprietary UNIX operating system like Solaris and getting it to the point where you are now, four years later?

Jim: Well, you mentioned Microsoft. Even if they wanted to open Windows, it would take years to do it, just as it’s taken years for us. And that’s what you do. You sit down, and you map out a multi-year strategy, knowing full well that things will change along the way.

Now, the story of how we opened Solaris has never really been told, although bits and pieces have been. I’ve been sort of tinkering around the idea of writing a book around this, because I’ve been involved in most of it; certainly not all of it, but I’ve been around long enough to know all the characters.

Multiple things happened at the same time. Some executives and engineers had been wanting to open Solaris for a long time. We opened it in June of 2005, although there was an entire year before that when our team was preparing, so that brings you to about June of 2004.

There were at least two or three years prior to that when there were discussions among various kernel engineers kicking around the idea. So, that’s late Solaris 8, early Solaris nine timeframe. That’s a long time ago now.

There were multiple ways this came together from the top and also from the bottom and from sideways, as well. And, you’re right that we didn’t own all of what’s in Solaris. A lot of the code, like what’s in Solaris 10, is brand new, written by a bunch of 30 year olds, whereas the older stuff has been around for a very, very long time.

We had a very serious challenge in the licensing and development methodology and business models around this. And there were many, many competing ideas on how to do it; how to license it, what the development methodology would be, what the business model would be, and it took years to figure all that out. Quite frankly, in fact, we’re still figuring it out.

We already have some experience at this now, but no company of our size has released all three of its core products at the source level. Sparc, Java, and Solaris are the three core products of the company, and to open those up has been quite a learning experience.

There was no model to look at, really, so it meant spending a couple years going through the code and finding the license agreements. This stuff goes way back, and we spent lots of time digging in old file cabinets.

We also had to find a license that would be workable with the many, many copyrights inside the code base, so we had to be able to mix licenses, essentially. We needed to figure out a development methodology that would work with our base–the old Solaris market–as well as what we wanted to build. And even then, we still had to address the process of opening everything.

When we actually opened in June, 2005, we didn’t open up everything. We opened up most of it–it was mostly the kernels at that point–but Solaris is designed with huge chunks of code called consolidations. The unique kernel is one, storage is another, and install is another, and you get these huge, huge chunks of code that come together inside of a distribution. We couldn’t open it all at the same time. Heck, we were just worried about making sure this thing actually could build on the first round, because we could not open all the bits as source.

There are a few things that we had to keep closed because we didn’t own them. It was kind of minor stuff compared to the scale of the source that we actually owned, and some of the stuff that we had to keep closed initially we’ve subsequently opened.

I would say the entire first year and a half was a constant wave of opening things, after the first release, and very few people realize that. There were 20 or 25 releases of code after the initial kernel, and most of it’s open now. I don’t think we’ve opened up anything recently, except little projects here and there that are continuing to open.

It took us a couple of years to do, after we figured out the licensing scheme and everything. This has left us vulnerable, in some sense, from a business perspective, because we didn’t have the benefit of, say, something like Linux that started very small. Linux started from an individual, then a group of people, and then a large group of people, and then companies got involved, but it started way off in the corner.

Four years ago, Sun was under competitive attack. Everyone was hitting us hard, and from a business perspective, things were challenging. We were going to open up these huge code bases while everybody watched, so every single mistake we made was a headline.

One of the things we did early on was to go to the open source community, and we had multiple conversations with the leadership. We said, “OK. We’re going to do this.” And they didn’t believe us initially, by the way. “How do we do it? What are your ideas? What are your suggestions?” We had multiple conversations, and we had lots of agreements and disagreements.

For about a year, we spoke to many, many people. That culminated in a big meeting at OSCON, in 2004, I guess. We had about 30 open source people in there from OSCON, various leaders, a couple of our core kernel engineers, and the rest of our team. We just talked and fleshed it out: “What are some things that we could do that would be really, really stupid? And what are some things that we could do that would be really, really cool?”

Scott: One of the things that is interesting when I look at OpenSolaris is that the open source community and the Linux community, to some degree, don’t really know exactly what box to put it in. On one hand, they’re happy because it’s open source. On the other hand, some of them had gotten used to thinking of Sun as a proprietary UNIX enemy. They may think it’s a pretty cool distro, but still, it’s not a Linux kernel.

I sense people being a little bit conflicted about OpenSolaris, and at the same time, it’s gotten some pretty good reviews.

What do you think? Maybe from a technical level, what are some of the advantages that OpenSolaris has by not running the Linux kernel? And at the same time, what are some of the challenges that OpenSolaris has because it’s not running the Linux kernel?

Jim: Well, from a technical perspective, you really have to speak with the engineers, but people have definitely been conflicted. There’s no question about that, and there’s no way around it.

Solaris grew from the open source community, though. It grew from BSD. And so, it has a long heritage of open source. And in the interim, while we were closed, then Linux grew, and it grew into a very large community.

One of the reasons people have been conflicted is that we have been confusing. Sun has been a huge contributor to open source over the years, but we as a company are sometimes confusing in our statements, in terms of being supportive or not.

Scott: Well, it’s hard, if you’ve got 10,000 people, to just have everybody suddenly kind of see the light and be on the same page. I mean, turning the company means individually turning everybody in it. And that doesn’t always happen simultaneously or smoothly.

Jim: Absolutely, and Sun’s culture is such that things are not just all of a sudden ordered in one way and everybody moves. This is an engineering company, and it’s always been very innovative in that respect. People have a lot of freedom to talk, to flesh things out, to argue, and to debate. Whenever you go into a meeting in this place, there are always multiple competing views, and that’s very, very cool.

Having said that, though, we have had the legitimate problem that we’ve had to open it up in phases, so you’ve only seen little snippets at a time, and it’s been growing over the years. When we first launched, we didn’t have a distribution. It was just source code. Now we have the OpenSolaris binary distribution, but that’s very recent–only six months ago.

Before then, it was just source code. We had a binary in the sense that we offered Sun’s Solaris Developer Edition and Solaris 10 itself, but those were not branded OpenSolaris. Our offering has come together in stages, so people have never seen the whole thing at once, and so it’s been confusing.

The situation is becoming clearer now, though, because of the distribution. People are noticing that it’s something they can download, use, and contribute to, and we had to do a few years of work in order to get here.

From the perspective of technical advantages, we got a lot of engineering credibility for Solaris 10. There was a big upgrade between 9 and 10, with things like DTrace, Zones, and ZFS. If you ask engineers and other technical people, they would point those things out as being competitive with Linux. In terms of OpenSolaris, those things get most of the attention.

ZFS probably is the most important new technology on the whole project, from an open source perspective. It’s a new file system, and it has also been ported over to Mac OS. I also just read recently that there’s a port started for DTrace, over to Linux.

These are the bits of technology where many people would say we have competitive advantage over Linux, although some engineers will argue against that. The Linux community has a vast community and developer base, and that’s what we’re trying to create. That’s where they’re mature and we’re young.

I also I like to have conversations with other communities. I’m always hanging out with the Linux community here in Tokyo. It’s an international community here, and they’re quite open to having conversations from the technical perspective and the community perspective; the same thing’s happening in China.

The OpenSolaris community and the Linux community are getting together over there and having conversations. They’re talking about where we can collaborate with ideas and things like that. So, it’s a little bit competitive and it’s a little bit of cooperative.

Sean Campbell: Let me ask you a question in a different area. Leaving aside any product comparisons about whether you’d rather run VirtualBox versus Workstation versus something from Parallels, did those guys learn anything from the OpenSolaris team since the acquisition? Have you guys tried to communicate with each other?

Obviously, you have a long history with getting things out there under open source, and lots of lessons learned. As Sun brings in groups like MySQL or VirtualBox, what kind of cross-pollination goes on there?

Jim: A lot. I’m not familiar with the guys at VirtualBox necessarily, but for any acquisition, there are many, many meetings at the executive, engineering, and marketing levels as we integrate the new company and the new technology.

We’ve had some of the MySQL guys come here to Tokyo, where I’m based right now. I got them in a conference room for an hour, and got them to talk to me about what they do, how they do it, and why they do it, from a community building perspective.

Sean: What were some of the highlights of that conversation? They have been successful at building a pretty strong community. What were some of the takeaways from that conversation?

Jim: Specifically with MySQL, they are essentially building a user community, and we are just actually creating our user community now. It’s brand new for us, and Solaris has been an enterprise operation. Really, really smart people use Solaris.

Well, I’m not really, really smart. I’m just a user who wants to be able to use Solaris, and it’s important for the team to be able to have conversations with people like me as well. That’s the new area we’re getting into, and MySQL has significant experience there. Obviously, they are technical users, but they’re somewhat different from the traditional Solaris focus on developers, kernel developers, and other highly technical people.

One of the things I found very interesting with MySQL is that their community leader is a VP reporting right to the CEO, instead of to marketing or engineering. When I came onto this project, I came from marketing to engineering. The argument that I used at the time is that community building is an engineering process.

I still largely believe that, because ultimately you have large groups of people hunkered around code, and that code is technical, rather than a marketing thing or a project management thing. The MySQL guys agree to some extent, but they say it’s not a complete view. They suggest that there are three separate buckets: marketing, community, and engineering.

Their argument is that the community guy should interact with marketing and engineering as equals. My focus was always to get the engineer out across the firewall working with people who are outside the company. That is the first principle, since without that you cannot build a community.

But, from a user perspective, if you could build a large user community, the engineers aren’t necessarily involved in 100% of that. They are far too technical, and that focus is going to make the community much, much smaller.

Now, the user community is gigantic. I’m interacting with them and people like me, but the actual core engineers are interacting with a much smaller group of people because they’re far more technical. So, that’s one thing I learned from them.

Through meeting various MySQL people, either in California or here in Japan, I do see a lot of cross-pollination of ideas. I don’t know that we have necessarily changed our strategic models, but we have integrated a lot of ideas that we see working for one another.

This whole concept of user identity is what’s changing right now in the OpenSolaris project, as we move from purely a development project into a multi-level community of users, developers, application programmers, and others. The timing is perfect for MySQL to come in, because this concept of users is ingrained in them.

Scott: Given that you guys just rolled out a distribution, are there other Linux projects that you look at and say “I really like what Ubuntu is doing around users,” or “I really like the way Red Hat or Novell positions themselves in the enterprise,” or “I like what Citrix is doing with their open source virtualization stack.”?

What stands out to you as you march ahead with a full distribution? Are there things out there that you’re looking at saying, “That would be cool if we could do that.”?

Jim: We have been impressed with the Ubuntu model, for sure. In fact, the whole Linux community impresses me, and I believe that there is a lot for us to learn there. I’ve always been a supporter of the Linux community–even before we opened Solaris–just because they have built a very large and thriving community, and it’s something that you can learn from.

With our new package manager, it now takes half an hour or 40 minutes to install Solaris. When I started in this project, Solaris 10 took three hours and I couldn’t do it.

[laughter]

I couldn’t do it, because it was not really meant at that time for people like me. It was for system admins and high-end developers. At that time, when we launched Solaris 10, we had just finished the port over to x86. It’s come a long way, and when you look at the distribution now, it’s far more focused.

I just upgraded last night to the latest build, and the interface is very nice. It installs cleanly, it finds all of your hardware, and general users can actually use it.

Previously, we had Sun’s Developer Express distribution, but the OpenSolaris distribution is a different model. It’s a much smaller footprint. You go out over the network, get your applications, and you upgrade. It’s not all in one big chunk. It’s also just a CD now instead of five or six DVDs.

There’s been a lot of learning, I think, from the Linux community, and I think there are a lot of things we can teach as well. Brian Cantrill is one of the creators of DTrace, and he offered in his blog to help the community with the port of DTrace over to Linux.

The other thing that’s happening now, is that as opposed to the first couple of years where we focused on opening the source and building a little niche community, we now have the early stages of a community, and we’re in the process of opening the entire Solaris development infrastructure. We want to take contributions into the distribution in the form of packages and things like that, just like you have with the Linux community.

We also want to open up the core infrastructure for kernel development, so now you’re talking about all the tools that we use inside: all the build tools and the SCM system. We have actually moved off of our old SCM system onto a new one, which we just completed this week, and that is a huge milestone. From there, we are looking ahead to being able to actually open up the whole thing outside.

The outside community people will have access to the same tools and processes that we have inside, which will help the distribution and the community building. The Linux community has an open model, and we want to have an open model as well, from a development perspective.

There are multiple levels of development, though: the kernel, applications and on up the stack. Of course, there are unique OpenSolaris features brought in here. From a user perspective, it’s pretty similar, but from a development perspective, you’re relying on the Solaris technology, which is built to run huge, highly scalable systems on.

Sean: Let me ask you a question in a completely different area. What do you think are some of the key differences in the way OpenSolaris and Linux are viewed, implemented, and embraced in Japan and China, compared to what you have seen in the US?

Jim: It’s a different world here, which is both negative and positive. I am speaking mostly for East Asia–that is, Korea, China, and Japan–as opposed to India.

Open source is very, very new here. There is lots of Linux here in Japan, but there is more Microsoft here than Linux. Microsoft is huge all over East Asia. When I came here, I thought I would see more Linux in terms of community people, but it is pretty small. One of the cool things about it, though is that it’s very international, and I can actually access the community because it is not only Japanese. It’s a very cool group of people too, by the way.

It’s pretty similar in China, as well, although maybe just a little bit smaller. I am going to be going over to China in a few weeks to have some conversations with people from the OpenSolaris and Linux communities.

Open source in general doesn’t have as big a profile here as it has in the West, and it’s not as well understood. There are huge language barriers, which is a tough issue, because there’s just not a lot of English here. When you go to places like India, there is a boatload of English, so the communication between India and the West is much more robust.

I was in Bangalore last year for FOSS India. The cool thing about that conference was that it focused on contributions. How are we Indians going to contribute to the Linux community? It was focused mostly on Linux and how India could be a net contributor into the community. You don’t hear that language a lot in East Asia. It is just not there yet.

I view that as a huge opportunity, and I am very excited about it. When you go to China, particularly recently, they are asking lots of very probing questions. The second thing that is happening in China is that the level of English skills is significantly increasing, very quickly.

I might be skewed in that view in the sense that I am interacting mostly with college kids, but they are really hungry, and they are assertive and smart.

I was there about six months ago, doing a talk at a university. There were 300 people in the room, which they think is small. I was going through my community stuff and they were asking lots of questions, about licensing, and the kernel, and all kinds of things.

Even though I insisted that I am not an engineer, and that I was not going to sit there and talk about bits and ones and zeros and stuff, they asked the questions anyway. I wound up promising to follow up with a lot of them; I’d get their card and say, “I’ll have to get a kernel guy to talk to you.” Before I got back to my hotel, I’d get an email reminder from them: “Did you follow up yet?”

Scott: [laughs]

Jim: I mean, this was really, really cool. I’ve established some relationships with these kids, because they’re very, very interested, and I’ve made some connections for them. My pitch to them is to get them involved in the main project. I’m not as interested as getting them involved only in China. That’s good, and it’s a first step, but I want them to get involved in the entire project, globally.

And that’s been challenging in East Asia because of the language barrier When I talk to Linux people in China and here in Japan, I get a similar story. We’re coming in a little bit after Linux, time-wise, but language issues are big, and the concept of really contributing, the way we look at it in the West, is not widely understood yet, but that’s changing, mostly in China right now, which is very cool.

This is a huge, green field opportunity, which is one of the reasons why I’m here. I have an interest in emerging markets, and so I’m going to spend a fair amount of time in the next few years in China. Because when we go to the universities, the interest is overwhelming.

The traffic patterns are changing a bit at our site; a lot of them are coming from East Asia now. Even in Japan, which is a pretty conservative and quiet place, the traffic from Tokyo is huge. It’s just that they express themselves differently here than the West, and it’s the same thing in China. Community building is a different thing here.

Scott: Let me ask you a question at a higher business level. Some people I talk to are a bit confused by Sun and all the competing technologies. You’ve got Solaris containers for virtualization, XVM, and VirtualBox. You’ve got your own hardware line, but you also do x86. And you’ve got your own OS, but you also support Linux. A lot of people just can’t figure all of that out.

It seems that what Sun has decided to do is maybe different than somebody like IBM or Microsoft. IBM tries to get you all the way into their WebSphere stack, and Microsoft wants to get you on Windows Server, and they’ll want you using Exchange, SharePoint, and SQL Server.

It seems like Sun’s just making sure that the door on the conversation is never really all the way closed. If Sun’s talking to somebody, and they say, “Hey, we think Solaris is a fantastic operating system,” and somebody comes back and says, “You know what? I really prefer Linux,” Sun can turn around and say, “You know what? We love Linux too. We just think you should run it on our hardware.”

And then somebody says, “Well, we’re going to go with Dell servers.” Sun can come along and say, “Well, but you’re going to need storage. And have you seen our MySQL database?” and this kind of stuff.

Am I onto something there, strategy-wise?

Jim: I wouldn’t quibble with anything you’ve said, really, although you have to understand that the company is going through a pretty big transition, so there are lots of parts around. People don’t say IBM is confusing, even though they have a whole lot more operating systems than we have.

Scott: [laughs]

Well, they just call it all WebSphere, and then it sounds like it’s all the same.

Jim: Well, yeah. [laughs]

Scott: Still, it seems like Sun intentionally keeps competing products in the portfolio and lets the marketplace kind of figure it out, and as long as there’s demand for both, Sun is going to just keep supporting both.

Jim: Definitely. I’m not on the product side, but it just makes sense to keep products in the line if there’s a market for them. There’s a huge market for SPARC, and we’re growing in other areas with x86. They don’t necessarily conflict, even if some of them overlap.

One point that you hit on is that we want the conversation channels open. Jonathan Schwartz says this externally, but he says it internally as well, and he’s very consistent with the message that when you open your code, and you move your engineers and your infrastructure outside for your development, you’re engaging in a fiercely substantial amount of conversations.

We do this with our blogs. We have something like 5,000 blogs now–some huge, obnoxious number. We have all these communities where thousands of Sun engineers are now outside, and we have hundreds of e-mail lists on OpenSolaris alone, and we’re engaged with tens of thousands of people.

There are multiple levels to this conversation, and it gives our salespeople access to different levels of organizations, and that is the key, from a development perspective.

When you move from development into business, I am interested in this level of conversation: how does that transform into business opportunity for the company? When we started the project, we did a pilot program, which I primarily ran.

For about a year, we opened the code to a small group of people from outside the company who came in under NDA. It was only under NDA because we hadn’t licensed yet. This was my first experience with this issue of multiple conversations within our customer base.

I don’t ordinarily speak to Sun customers on Wall Street. They’re kind of high-end guys, but when we started the pilot program, we went to our customer base and our various communities, and we said, “Who wants to come in to help us start?”

I eventually found some people on Wall Street–sys admins and developers and such–and I brought them in, and then we added some guys in the defense industry and the education industry and whatever market Solaris happened to be in.

That was a very interesting experience for me, and also for Sun, to realize that you can have multiple conversation channels into a customer base, starting from the development side. This is all very, very cool, because information flowed freely, and I think there’s more than enough evidence now that these conversations will lead to more engagements and opportunities for Sun to be able to sell things.

Scott: There are so many different ways now that Sun has to start a conversation and keep a conversation going with the customer, around virtualization and different ways of doing it, and around Solaris, Linux, hardware, storage, and databases.

I suppose that’s why it makes sense for Sun to have created a strategy where the doors never really close.

Jim: Exactly. I was talking to a customer early on, maybe four years ago now, and they said, “You know, you’re not really into virtualization.” Well, now they’ve got some virtualization to pick from.

[laughter]

It’s interesting because I remember that conversation with this guy, and he was very genuine, and we had a good conversation. And I also remember all the conversations internally, and huge plans to get into virtualization. It was already part of Solaris 10; it just wasn’t well known.

A lot of my challenges every day center around electronic communication, because it takes a long time to build a community. It takes a long time to get messages out across cultures, languages, and geographies, especially while you’re transforming the whole company and opening your core product. There’s a lot going on.

Sean: At this point on the conversation, we usually like to turn the microphone over and ask whether there are any closing thoughts that you want to add.

Jim: The closing thought I can offer is that this really has been a multi-stage process, with a lot of figuring it out as we go, and a lot of looking back and figuring out the specifics.

Looking back on it now, after four years, we’re pretty much where we wanted to be. We’re behind on some things, and we’re behind moving some of the core infrastructure outside, but we’re way ahead on other things. We had no idea we would have a community of the size that we have right now. I think we’ve moved faster on the community front than we have actually on some of the infrastructure front.

Governance is going to be a big challenge for us, I think, as we move tools and engineers outside. They bring processes with them, so we have a governance model that we have to integrate into those processes. And we will have to decide whether those are the right processes for the open source community, or whether they are too focused on Sun.

I think that the focal point, to kind of tie these things together, is the OpenSolaris binary distribution, and the fact that it’s surprisingly good. We always felt that we would get good reviews from a technical perspective, because Solaris is so strong technically, but, people seem genuinely happy with it.

Sean: Thanks for taking the time to talk with us today. This has been great.

Jim: Thank you.

Bookmark this:

Interviewee: Alexandre Julliard.

In this interview we talk with Alexandre. In specific, we talk about:

• Developing an open source project in cadence with Windows
• Philosophies about how Linux should reach out to Windows users
• The leadership model used in the Wine project and deciding what goes in
• Deciding what applications to support and what to leave out
• The difficulties of working with the closed Windows API and Wine’s relationship with Microsoft

Sean Campbell: To start, please tell us a little bit about your background as it relates to Wine, your role as CTO with Crossover, and that sort of thing.

Alexandre Julliard: I’ve been working on Wine since the beginning of the project, 15 years ago, when I was still a student. For a number of years, I worked on Wine in my spare time, and then in 1999, I joined CodeWeavers to do Wine development full time. I’ve since become CTO of CodeWeavers where, essentially, all we do is build around Wine.

Sean: With a project of that longevity, what interesting evolutions have you seen, in areas like a community evolution or a usability evolution?

At the same time, as the Windows platform and the types of applications on it evolves, does that create a race condition of sorts, where you’re trying to catch up to everything that’s changing, and at the same time, adding new features to Wine?

Alexandre: Yes, mostly the evolution is driven by Microsoft and how Windows is evolving. And clearly, in that time frame, Windows has become much more complex. One of the major transitions we had to make was from 16 bit to 32 bit, which was very hard for Microsoft, and very hard for us, too, as we had, of course, the same problems.

The sheer size of the Windows API is, I would say, the major factor in the evolution of Wine. The first versions were small, self-contained binary code that just provided all the APIs. Then we had to make the code much more modular, to split things up, to define interfaces between the different areas of the code. We also had to start being very serious about regression tests. At the beginning, we were just hacking things until it worked. But, it became clear that it wasn’t scaling, and that we needed regression tests to be able to nail down the behaviors of the APIs, and make sure that when we add more stuff, we don’t break the old APIs.

That was a major factor in the evolution of Wine, and the fact that it’s really becoming stable is because of the huge test suite that we now have.

Sean: Given that you have to deal with the potential of really any Windows application running on Wine, you’ve got a massive set of applications, and I’m sure, lots of legacy applications that your users hope will still run, in cases where maybe the legacy application isn’t even really fully supported by Windows anymore.

But before I ask about that, I’m curious about what you are doing around 64 bit, given that you had a big evolution from 16 to 32. You take somebody who puts 64 bit Vista on their machine, and inevitably, iTunes will only kind of sort of work until they get it working, or there will be some kind of issue with their TiVo software package

Of course, those examples are just consumer grade stuff, but, there are also issues with drivers, with smart cards, and on and on. Have you started to see those issues creep up on the roadmap as something you’ll have to address? Or, is that just far enough out from your perspective that it’s still predominantly a 32 bit world?

Alexandre: Well, it’s clearly on the roadmap, but it’s not something we’ve had a problem with yet. There are simply no 64 bit applications that anybody wants to run that don’t have a 32 bit version. We don’t run drivers, so we don’t have the problem of the kernel being 64 bit. That’s not an issue for us; it’s just the applications.

Sean: That makes sense. Let’s talk a little bit about the QA process, then. Can you give us a sense of how many people are on the QA team? What’s the method of doing QA? How much user involvement versus core members from Crossover are involved in supporting Wine? Your relationship to Crossover and Wine seems a little bit analogous to the way Canonical is to Ubuntu, but maybe that’s a mischaracterization.

Alexandre: The relationship of CodeWeavers to Wine is maybe more like Red Hat to Linux, in the sense that we’re one of the contributors, but we don’t direct anything in the project. Obviously, I’m personally directing things in the project, but CodeWeavers doesn’t have a word in my choices with Wine maintenance.

Sean: That’s a good clarification.

Alexandre: As far as QA, we have several approaches. At the start, in Wine itself, we have a regression test suite, which basically performs unit tests of very specific Windows APIs. We test them one by one, call them with all sorts of parameters and test the return values and all the parts of the behavior of a single API. It is purely automated, so it is run after each change.

We also have higher level application level tests, and we do some of that at CodeWeavers. We have a team of three or four people who test the applications. We also have automated mechanisms, based on a tool called CXTest, which has been written by CodeWeavers specifically for Wine testing. For instance, it lets us run application installations in a fully automated way.

And then, we also rely simply on bug reports from the community. We have an application database on the Wine website where people can report what works, what doesn’t, and a public bug tracking system. We try to leverage the power of open source where all of these people are willing to help and everybody tests a different application and reports problems.

Scott Swigart: Linux is obviously becoming more popular, specifically on the desktop, which certainly must be good for you guys because of things like the ASUS netbook. It seems like there’s more demand for Linux desktops than ever. Certainly the vast majority of people are running Windows, but Linux is starting to penetrate into mainstream thinking.

I encounter two predominate thoughts. One is that we should make Linux as much like Windows as we can, because that’ll make it very easy for people to move from Windows over to Linux, and that’s the ultimate goal.

It seems like you guys are aligned with that school of thought. Wine and CodeWeavers exist because they recognize that some people are just going to have to run Windows apps as part of what they do, either at work or at home. Office is probably the most the common one that people want to run.

But there’s another camp that says, “The problem is Windows. We’re not going to solve it by making Linux more like Windows, and we shouldn’t we even go down that path.”

Do you ever encounter that point of view? What is your response to the idea that making Linux like Windows is a problem, rather than a solution?

Alexandre: It’s certainly something we hear a lot. Many people don’t like Wine because of that, because they think that Linux should get its own applications rather than using the Windows ones. I think that, with the number of Windows applications out there, it’s simply not realistic to hope that all of them could be brought to Linux right away.

The motivation for Wine has always been to make a transition possible. And once enough people get on the Linux desktop, then you create the demand for real Linux applications. I certainly agree that the goal shouldn’t be to make Linux just like Windows; there’s no point. Linux is different and should remain different, but there needs to be a way to enable people to switch, and that’s where we come in.

Sean: In a different area, you’ve been leading the Wine project for a long time–what has that position of leadership been like? In a lot of open projects, you see a certain amount of churn at the leadership level, or you see the project evolve into something else, which causes the leadership team to change. But you’ve had a very steady focus for a long time leading this community.

What kind of leadership style have you taken with the Wine community? Is it the benevolent dictator? Is it a very community-oriented decision making process? Give us a sense of what decision making looks like in the Wine community. Is it top down or bottom up driven? Because every community we’ve talked to about open search projects seems to be just a little bit different.

Alexandre: Well, it is clearly the dictator model.

Sean: [laughs]

Alexandre: I’m really making the decisions, and I think part of the reason people accept my decisions is because I’ve been around for so long. And I’m the only one who’s been around since the beginning, so I have an overview of the project that nobody else has.

I’m a big fan of the dictator model, not just because I’m the dictator in this case, but because I think it’s really the most efficient. It cuts down on discussions, on politics, on all that stuff that can cause a project to lose a lot of time and cause trouble between developers. I think it’s good to have a very clear way of making decisions and not to lose time over decisions, even if they may not always be correct.

Sean: That totally resonates with my entrepreneurial experience. Sometimes it’s better to make the decision that’s 85% right if you can evolve it later. You’ll at least have made a decision as opposed to just getting deadlocked and not making anything at all, especially if you can do 85% goodness while you’re waiting to fix the 15%.

On a different topic, what was the rationale behind the versioning scheme for Wine? Obviously, the project was in use for an extended period of time before it hit “version 1.0.” But yet, after some years of development, you felt it was time to go to 1.0.

Alexandre: The idea was to provide something that people can really use. Given the size of the Windows API, it just doesn’t make sense to release something like this only half implemented, because nothing works. Most applications really require large percentages of APIs to actually be working, so that’s why we didn’t want to create false hopes and release something that wouldn’t support anything beyond Solitaire.

Sean: So what led you to finally decide you’d reached the 1.0 milestone? Was is just that you felt you had covered enough of the Windows API, or was there some other instigator?

Alexandre: One part of it was support for enough applications that people find interesting today, which are not the same that people found interesting five years ago. There were also some questions of internal design and a number of things we wanted to clean up to make sure that we can grow from here without needing to re-architect things and break compatibility and things like that. So, we wanted to make sure that 1.0 was a good foundation for the future.

Sean: Tell me a little bit about the process of getting an application “supported,” although I realize that’s kind of a gray issue because certain things will work and certain things won’t? What’s the process around filtering out, “Well, that’s somebody’s pet application, but we don’t really feel a lot of people use it?”

Is that a fairly subjective judgment on your part? There are things like Office that are fairly easy to establish that everybody uses. But, there’s probably that second tier or third tier of applications that I imagine it’s a little hard to decide about, especially if it’s a vocal sub group that wants it supported.

Where do you draw the line between the core things you’re going to support and the things that you push off to the community?

Alexandre: Clearly, we need to go where the market goes and try to figure out what applications people want. But in general, it’s very hard to say, “I have this application and I want to make it work.” What we usually end up doing is to fix bugs all over the place, and then we look at 10 applications and we find out that three of them work and we decide that we aren’t going to support the others for now.

It’s really more reasoned by how well an application is already working than by a conscious decision of concentrating on one application instead of another one.

Sean: It sounds like sort of a mixture where there are certain things that you obviously want to work on, as well as other things where the decision is made by the amount of friction it takes to get it working and the amount of work you have to go through.

Alexandre: Yes. And the process of getting an application to work is fairly slow, so we depend a lot on the community and on people spending the time to make their favorite application work. Once they do that, it becomes an application that we support. And it is sometimes the case that one developer is the only one in the world who cares about this application, but that’s really enough to make it work and support it.

Scott: Talk a little bit about what some of the most important applications are. From my perspective, on the outside, it seems like Photoshop is something that comes up a lot, and both the older version and the current version of Office seem to be really high priorities for people. What do you see as the top five applications that everybody wants to run?

Alexandre: Office is clearly one of the major ones, though it has changed a bit. A few years ago, there was no discussion, everybody wanted Microsoft Office, and today, it’s a bit less so. People are happy with OpenOffice, and there is less pressure on Wine to support Microsoft Office. Photoshop is in high demand, as is Outlook (more so than Office as a whole), and games are fairly big too.

Scott: I noticed that there is a Crossover product specifically for games.

Alexandre: Yes, that’s fairly recent actually. We noticed that running their favorite games was something that people really wanted to do with Wine, and it turns out that most games are fairly easy to support, compared to something like Office. The range of APIs that games use is much smaller.

Scott: Outside of the 16 to 32 bit change, what was one of the hardest changes you had to go through?

Have you had the experience of waking up one morning to realize that Microsoft has released some update to “X” product, or a Service Pack, or a particular OS change that was particularly more painful? Can you describe for us one of the more challenging engineering things that you had to deal with as Windows was evolving?

Alexandre: The truth is that Microsoft cannot change things easily, because they are a slave to compatibility just as we are. We typically don’t suddenly find out that we need to support something that we hadn’t planned for.

The most problematic thing for us recently is that applications are starting to depend on features that are only present in recent Windows versions. For a long time, you couldn’t release a Windows application if it didn’t run on Windows 95. So we had Windows 95 compatibility and everything was fine. Now there are more and more applications that don’t bother with 95 compatibility, so we find that a lot of the architecture that was good enough for 95 support is not good enough for XP.

Sean: What do you think about the VMware Fusion and Parallels coherence and unity features, since this is starting to blur that line of what OS you’re running? People are starting to see the option, for example, of running XP on a virtual machine under Linux, and that’s where you load your Windows only apps.

What’s your opinion about actually running the app on Windows through something like Wine or Crossover versus using a virtual machine where your Windows apps go, even though you’re living in your Linux partition most of the time?

Alexandre: There are different aspects to that scenario. Clearly, no matter how much effort VMware and the others make, they cannot get the level of integration that we can achieve with Wine, where the applications can integrate with the Linux applications in all sorts of ways.

The other thing is that with Wine, you don’t need a Windows license, and you can be completely independent of Microsoft if that’s what you want. That’s something you will never get with VMware.

Scott: At the technical level, as you’re working to support an application, I’m guessing that you guys have implemented a large set of Win 32 APIs, so you start running the applications and see what does and doesn’t work, right?

How do you trace that, as an application’s running? What are some of the tools or diagnostics or whatever you use to figure out where it broke? Maybe it’s because the application hit an API that you haven’t implemented, or it hit an API that you’ve implemented, but it’s not 100% compatible. Talk a little bit about how that works.

Alexandre: The main difficulty we face when we run an application that doesn’t work is to find out why. We have some mechanisms, we have a lot of traces in the code. In particular, we have a way of tracing every single call that the application makes to the Windows APIs. We have an integrated Wine debugger in case of a crash and things like that.

Really, what it boils down to is, you look at the traces and you try to spot in the trace something that doesn’t look right. That’s pretty hard. You need a lot of experience to be able to find those needles in the haystack.

Once you have found something that doesn’t behave quite right, usually it’s trivial to fix. You just write a test case, see how it’s supposed to behave under Windows, and change the implementation. That’s really the easy part.

Scott: I guess that since you’ve always got the Windows API as a reference, it’s no mystery.

Alexandre: Actually, the Windows API is a mystery and that’s why you have to run all of these tests.

Scott: [laughs] Right.

Alexandre: And it’s the same way with documentation, it’s very incomplete, so the tests are the only way of finding out how it’s supposed to behave.

Scott: What kinds of interesting or surprising things have you learned about the Windows API? In some ways, you looked closer at the Windows APIs than just about any programmer who builds applications for Windows. As you have looked at it from your unique perspective, what’s been interesting or surprising about the Windows APIs themselves?

Alexandre: Well, I don’t know if it’s surprising, but the bad quality of the API is clearly the thing we take from that. Most APIs are buggy or don’t behave as recommended or just have some have corner cases that get triggered by some applications that nobody knows about, probably not even Microsoft.

People just write code and try it on Windows, and it works, and they don’t know why. Likewise, nobody knows why it doesn’t work on Wine, so we have to figure out why the code works on Windows. We actually find a lot of bugs in applications. Wine is a great debugging tool for applications.

Scott: [laughs]

Alexandre: The thing is that the Windows API actually goes to a lot of effort to hide the bugs. Whenever the application does something incorrectly, the Windows API tries to hide it instead of crashing or showing an error.

It’s good in a way that it doesn’t crash, of course, but it also prevents people from debugging their applications. So, we find a lot of bugs that if the Windows API were a bit more straightforward in what it accepts, the bugs wouldn’t have made it into the applications.

Sean: Have you seen any kind of evolutions over time, whether positive or negative? Have you seen evolutions in the API in terms of quality, or do you feel like you could sometimes get a sense of which way the wind is blowing with the Microsoft development process internally just by where you see them put emphasis on the API from release to release or service pack to service pack?

Alexandre: Yes. In the recent API, they have clearly made an effort to try to avoid security issues like buffer overflow and things like that. That’s clearly something they are now adding to the new API that they never cared about before.

Lately it seems they are more trying to spot bugs early and to crash when something is incorrect, instead of just hiding the bug. That’s definitely a positive evolution.

Scott: Have you seen Microsoft’s relationship to your project change at all over the years? I don’t think it’s any secret that Microsoft views Linux as a competitor, so they’re not really out to make stuff run better on Linux. What’s been your relationship, if any, to Microsoft, either as a company or individually–has anybody been at all supportive of the work that you’re doing?

Alexandre: Actually, we’ve not had any contact with Microsoft at all. We know that they are very aware of Wine, and they watch what we’re doing very closely, but we’ve never had any kind of official contact, either support or threats or anything. They just don’t talk to us.

Scott: They have just completely left you alone? They never sent the lawyers, but they never sent tech people to help you out either?

Alexandre: No.

Sean: We’ve reached the end of our time, but I wanted to thank you for taking the time to talk with us today.

Alexandre: OK. Thanks.

Bookmark this:

Interviewee: Joe Brockmeier.

In this interview we talk with Joe. In specific, we talk about:

• Where openSUSE fits into the desktop Linux landscape
• Relationships between openSUSE and SUSE Linux Enterprise Server and upstream projects
• The effect of commercial agreements on open source projects
• The future roadmap for openSUSE

• Bringing together technical and non-technical users in open source projects
• Coordinating testing and bug fixes among distros and upstream projects
• Design coherence among distributed developers

Sean Campbell: Joe, to get us started, could you give us a little bit of your background?

Joe Brockmeier: OK. About 1999, I started writing about Linux as a freelancer. That eventually morphed into a career writing about Linux and open source as a technology journalist, more or less full time, much of that time as a freelancer. Around 2005, I joined Linux.com as editorial director, and then last year, I was with “Linux Magazine” as editor in chief. My educational background is in journalism, and that’s when I discovered Linux–when I was in university.

But last fall, Novell came and asked if I’d be interested in applying for the community manager job, which at that time was called the chief Linux evangelist. Having followed the open source community for so long, I was really interested in being directly involved with the project, as opposed to just observing and writing about what open source communities were doing, so I joined Novell in February as the community manager and have been pretty much going non-stop since.

Back in August 2005, openSUSE got its start when SUSE announced that they were going to start a community distribution. Prior to that, SUSE Linux had been developed in house by SUSE before they were purchased by Novell, and while it was a very strong distro, there really wasn’t much community input into it. Novell decided to build strong community participation.

The project is going on about three years old now, and we are making pretty good progress in terms of getting the infrastructure in place for external contributions and having transparent development.

We just announced last week the 1.0 release of the openSUSE Build Service, which adds the ability for external contributors to make direct contributions to the distribution, as opposed to going through the gatekeepers, so to speak. And we released openSUSE 11.0 in mid-June, with a lot of improvements, particularly in the installer and the package management, and of course all the normal improvements that go along with a version upgrade, such as new versions of KDE and GNOME and things of that nature.

Sean: Tell us a bit about where you feel openSUSE sits in the landscape of desktop distributions. What do you think it’s exceedingly good at, and maybe some of the places where you see challenges or opportunities?

Joe: Generally, my metric for success on the desktop is how well it fits what people need. I don’t really spend a lot of time comparing it to other Linux distros, because I really think we all have the same mission, which is to get people using Linux. So I don’t view them as competition, so much as inspiration, if anything.

The audience we’re trying to address includes home office users and others who want a good, solid desktop operating system that’s as easy to use as possible.

I think openSUSE is exceedingly good at package management, being easy to use, offering a top-notch desktop experience in GNOME or KDE, and providing a wide range of the best free and open source software available.

Our challenge is reaching new users and encouraging more users to become contributors.

The opportunity, right now, is that a lot of people are put off by Vista and are looking around at their options. I want those folks to find openSUSE, and another challenge we have is to find ways to make the jump from the audience that’s Linux-aware to the vast number of potential users who haven’t heard of Linux or know very little about Linux.

Sean: Every distribution seems to have an area that they don’t go after, or that they maybe feel they don’t have as great a degree of fidelity in. Is there some area where you think that’s true for openSUSE, where it’s not particularly geared for?

Joe: I was at Sun CommunityOne, and we had a Linux distro panel that included me, Jono Bacon from Ubuntu, Karsten from Fedora, and Glynn Foster from Sun. We were talking about this.

I think the only major difference is that we have a slightly different philosophies. Fedora really wants to approach the desktop with only free software, and they don’t want to ship anything that is proprietary at all. On the other hand, if you look at the retail box and the DVD that we ship, we actually include a little bit of proprietary software like Adobe Flash that we think people who are coming from Windows are going to want.

As much as I don’t like the fact that you have to have a proprietary program to view a lot of web pages, I think that that’s reality. If my girlfriend’s kids use a computer, they’re going to go to web sites that require Flash, and they’re probably not going to be sold on the argument that since it’s not free software, they should just not go to those sites.

I think it’s more important to successfully get people to use a 98 percent free software desktop than it is to ship a 100 percent free software desktop that no one wants to use.

Sean: So maybe we have to be OK with the idea of shipping a proprietary 3D driver, because everybody has EVO integrated right now. We can either tell them to go spend seven days configuring it when they’re not really competent to, or we can just roll over for two percent of what’s going to be in the distro.

Do you feel that’s kind of what’s going to have to happen in the interim, and then eventually the hardware manufacturers and the other folks will ship Linux drivers, and then maybe that’s not an issue as much anymore?

Joe: Yeah. I think it’s a major chicken and the egg thing. You have to demonstrate that you have a certain percentage of the user base before you can expect companies to spend engineering time or be willing to do things differently than they have in the past.

I don’t really buy the argument made by a lot of the manufacturers that they have to have proprietary code to be competitive. On the other hand, I do think it’s necessary for us to make the argument to them that there is some benefit to them when they ship a free software driver.

And the first thing that we have to accomplish there is to demonstrate that there are enough users that it will make an appreciable difference in the number of units that they would sell.

Scott Swigart: What’s the relationship between openSUSE and Novell’s enterprise distros for the server and desktop?

Joe: Basically, openSUSE is the foundation of SUSE Linux enterprise. We just released 11.0, and now we’re working on 11.1, which will be the foundation of the SUSE Linux Enterprise Server and Desktop products. The appropriate technologies that we develop in openSUSE are carried over into the enterprise product.

Scott: Would it be accurate, then, to say that newer technology might show up there, and then a decision is made in terms of what’s ready for the SLES and SLED distros?

Joe: We do sort of incubate technologies within openSUSE, although we also work on things that are of benefit to the openSUSE community that may not really have an impact for our enterprise customers.

We are also careful to make sure that whatever we ship in openSUSE is usable and stable enough that our users are going to be satisfied with it and happy with it.

Scott: What are the usual things that you do when you’re putting together a distro? Where does the work of the upstream project stop and your work start?

Joe: In some cases, there really isn’t a great deal of difference. For example, we have a team that works on GNOME and we try to do as much work as possible within GNOME, so that there’s not that loss between the upstream project and the distribution.

To illustrate one of the problems historically between a distribution and an upstream project, say you have Fedora and Ubuntu and openSUSE and all these other different distros working on something. You get lossiness when one project decides to innovate in one area or add a few patches or whatever, but those changes don’t necessarily make it upstream, or they do make it upstream but after the main project has also started working on the same feature or problem in tandem.

As much as possible, we try to work within the projects like GNOME for KDE that benefit us, rather than doing the patches in our own little area and then maybe submitting them back, or letting them come get them, or whatever.

Obviously, for the desktop, we add our own artwork and some polish in those areas. We also make some configuration decisions that may be aren’t handled by default GNOME or KDE, but in a large way, we really try to work with the upstream projects to avoid duplication of effort and that sort of thing.

Scott: How do you handle it when you identify upstream work that would be valuable to your customers when that work is produced after the last stable release?

Joe: For example, we knew the Mozilla folks would be shipping Firefox 3 some time after we shipped openSUSE 11. We had to make a decision about whether we should stick with Firefox 2, which would be pretty old midway through openSUSE 11.0, or whether we should go ahead and ship a pre-release version of Firefox 3.

Ultimately, we decided to go ahead and ship Firefox 3 beta 5, and when the feature freeze came into effect, we stuck with that version. Since then, we’ve gone ahead and shipped the final, via the package updates.

For community distribution, it’s easier to do that sort of thing, because we don’t have the five-year enterprise contracts to worry about, which gives us a little more flexibility.

Scott: What about the smaller projects, like Pulse Audio, an IM client, or Pigeon, when there’s maybe not an official release, but you want to back-port a really important patch to the version you’ve shipped?

Joe: Generally speaking, the policy would be that if there is a security fix or something, we would back-port it. We usually don’t do full-version updates or anything like that, with some exceptions for major apps like Firefox. As for new features, though, we have a rapid enough release cycle that it makes more sense to just put the new features into the next release.

Sean: Let me ask you a question in a different area. Generally speaking, in the open source community, there’s a somewhat mixed reaction to Novell having agreements with Microsoft. Do you feel that any controversy in that area has an effect on uptake of openSUSE?

Joe: When I took the job, that was one of the first things that I expected a lot of questions on, and in fact I did at one time, because everyone thinks that it’s the elephant in the room. But when I go to open source conferences and actually talking to users and whatnot, I generally haven’t found it to be as big of an issue as is generally suggested in the press. This is not to say that it’s not an issue at all, but it’s hardly the only issue, and after nearly two years, a lot of people have realized it’s not the catastrophe that some painted it as when it was announced.

I think there are some folks that are very active online trying to complain about this particular issue, and they’re welcome to that viewpoint. I would suggest that maybe if you are deeply committed to open source, perhaps your time would be better spent in doing something positive.

Sean: [laughs]

Joe: I often wonder, in fact, whether people making complaints like that would go on to something more productive if they got exactly what they wanted, or whether they would just wander off and find something else to complain about.

But, that’s a tangent.

Sean: To follow up on that, there are of course cases where people make big changes, like when Red Hat walked up to the conference stage and said, “Nope, it’s not Xen anymore folks; it will be KVM.” And Novell at large has spent a lot of time and money partnering up with Xen which then gets bought by Citrix, which is another for-profit parent, right?

I know you do support KVM and Red Hat supports Xen, but what are the thoughts broadly in the openSUSE community about Red Hat making the push toward KVM?

Did that changed your perception about what you want to do with support for virtualization moving forward or put increasing emphasis behind KVM? And how does that map to Novell’s goals in that area?

Joe: I don’t think that there’s a decision that’s been made. I’m not actually on the Enterprise side of the house, so I would have to defer that to somebody else for that, but in the community, I haven’t seen a strong reaction either way.

I think that within the kernel community, there’s an obvious preference toward technologies that are in the mainstream kernel. But beyond that, I haven’t seen a huge movement either way due to Red Hat’s announcement.

Joe: I’d like to go back and finish addressing the Novell/Microsoft thing, though. I think that probably there is a small percentage of people, now that the deal is nearing two years old, who still have an objection.

But most people have realized, I think, that the deal has not really been detrimental to open source or Linux, and they have moved on. And we have a lot of folks who really enjoy openSUSE and who continue to use it, and we continue to get new users every day.

Sean: Tell me a little bit about the roadmap for you guys in terms of what you think you’re adding in the future that targets more the home office worker or business productivity user. What do you think are the top things that you need to add to continue to grab more of that audience?

Joe: We’re still very early on in the 11.1 cycle, so right now it would be hard to say exactly what’s going to wind up included, although we do know that the next version, for example, will feature OpenOffice 3.0. We are very active in OpenOffice.

Most Linux distributions actually ship the Novell version of OpenOffice, with some of the patches that we include as opposed to the actual Sun version. A lot of things in OpenOffice 3.0 are going to address the needs of people who use their computers to do office productivity type stuff, and they’re going to address them pretty well.

Sean: Apart from specific things included in the next release, can you speak a bit to the broader vision? If you want to continue to work with that audience, what do you think they’re asking for in a Linux desktop distro?

Whether it shows up in the next release or two releases from now, or three, what are the things that you feel openSUSE needs to do to continue to pull in that audience?

Joe: From that perspective, I think that what people are asking for is continual improvements in ease of use and familiarity. If you set somebody in front of a computer for the first time, I think Linux is actually more or less as intuitive as any other operating system.

But we have to keep finding ways to address the people who are coming from Windows or Mac OS 10 and making the computer not identical, but still familiar and easy to use.

That’s one of the reasons we spent so much time in the last version reworking the installer to make it much easier for people to actually get Linux on their computer in the first place.

Sean: Let me add one quick follow up. Wubu is a very interesting thing for Ubuntu, and I almost look at what they’re doing as kind of like what could happen with Apple when they reach a critical mass of boot camp, plus switching to Intel, plus virtualization.

We talked to a person on a train trip who said, “Oh, yeah, somebody put this thing on the computer for me that lets me run Windows in a window on my Mac.”

And we realized that this person didn’t know or care that it was virtualization; it was just something that let them run both OSs. And all of a sudden, these things kind of come together.

I’m curious if you guys are thinking of similar approaches. I know they’re doing images on USB, and it seems like the Ubuntu guys are trying really hard to make that initial five minutes to get into Linux happen really, really fast. What are you thinking about in terms of installation and things like that?

Joe: I don’t know how many people actually run off of the USB key or even off of a live CD for any amount of time. And I think that we do offer the email, just as Ubuntu does, and we do offer live CDs with KDE and with GNOME, so you have the same options there.

I have not seen something like Wubu on the openSUSE roadmap so far, although that’s something that very easily could be added by a community member. And so if it’s something that people feel strongly about, I hope we will have work in that area.

Scott: Some open source projects like the Linux kernel or Apache are designed to be used by a very technical audience. But if you get further out into things like OpenOffice, and as you get further out into the distro itself, you’re potentially targeting non-technical people.

On some projects, the sentiment is that, “If you want a feature, code it up. Show up on the mailing list, submit your patch, and we’ll talk it over.”

But as you get more out into consumer-facing stuff, the people who have need for features, and the features they need, are very different from the needs of the people who are actually working on the project.

How do you guys balance that out? How do you, first of all, know what your users are going to really want and find beneficial? And since those people don’t necessarily have the ability to actually work on those features, how do they actually get implemented?

Joe: As you may know, a couple of years ago we did a desktop usability project, and one of the ways we implement the higher-level features is by handing them off to the desktop teams.

We are in the process, for example, of opening up our feature and tracking stuff, so that a user can come in and interact with the development community and have some input into features that are being worked on, without having to put their developer hat on and actually implement it themselves. In a way, it’s similar to the way that features are implemented in commercial projects.

I think it’s a little bit of a fallacy that open source projects are less receptive to their users than proprietary ones. For anyone who’s fairly interested in seeing a feature implemented, we actually have mechanisms where users can interact directly with our developers and lobby for a feature, whereas if you want a new feature implemented in Adobe InDesign, good luck getting direct contact with an engineer who works on the project.

Scott: It also seems also like there’s a lot of benefit in the pairing that’s happened in open source between commercial entities and the community. The community is really good at enabling technical people who are coders to scratch their own itch, which you can’t do with proprietary software.

And at the same time, the corporate entities are doing some of the stuff that people in the community wouldn’t find fun or interesting. They’re also doing things like usability studies that you’re just not going to have a community go and organize.

The open source community can add a lot of ideas, spark, and innovation, but the commercial entities are able to whiteboard out a feature, take input from users, do usability studies, work on documentation, do some of that heavy lifting that isn’t necessarily as good a fit with the community.

Joe: I would agree mostly with that, although I would say that the kernel development that occurred up until companies got involved with it surely constitutes heavy lifting as well. Still, I do think that it works best when you have both represented more or less equally, when you have a company that can hire people, for example, to tend to a Bugzilla and do bug fixes and product support, and you can have people in the community who are doing more innovative work.

You’ll find very few people who come along saying, “You know, I really know all I really want to do is squash bugs.” You don’t find a lot of people who only want to do product testing or things of that nature.

So, I think it works best when you have a company like Novell working with the community like the openSUSE community, or you have companies like Novell and Sun working with OpenOffice, and still making it possible to have the community help guide the project.

I think that companies alone don’t necessarily do as good a job at anticipating what users want and need as communities do when they have some input directly.

Scott: Right–it seems that there’s a balance, and if it’s done right you get the best of both worlds. It’s not people off in an ivory tower saying, “Here–we built what we think you’ll want,” because you’ve got so much community input. But at the same time, it can be more directly addressed when good documentation needs to get written or the bugs are piling up and need to get fixed.

Joe: Another good example is what the Mozilla Foundation has managed to achieve with Firefox, which has spawned most of the innovation that has come out in web browsers. They have married the two sides by having a corporate entity that actually funds all of the development and hires people to do specific things, and they also have a tremendous amount of external and community developers.

Scott: The bigger projects like the kernel or Apache have a lot of structure in place, but some of the smaller projects have less of that. Do the Linux distros act as a sort of clearing house for some of the smaller projects? Do they give another point for people to interact if they’re having a problem with something like a compatibility issue or bug?

In other words, do people sometimes go to the distro they’re using to resolve issues with upstream projects?

Joe: I think that happens to some degree. For example, if I’m new to Linux and I’m using openSUSE, my focal point for any complaints with a program is typically going to be the distribution, not the upstream project. So we can serve to distill those problems and figure out where the problem really lies. We also do give feedback.

And a lot of the time you also find people who work on those upstream projects work within the distros too. For instance, we have Vincent Untz on our GNOME team, and he’s also a GNOME Foundation board member and participant in GNOME aside from his work with Novell.

Scott: Like you said with GNOME, you guys are actually doing a lot of development on GNOME, because you don’t want openSUSE to have a fork that’s significantly different. There isn’t as much value in that as just working with the upstream project to move it forward.

In the early days it seems like the methodology in a lot of open source projects was to post code to a mailing list that got examined by a lot of people, and it got rolled in or it didn’t get rolled in. There was sort of a “many eyes” approach.

More recently, people at Mozilla and others have developed tools to automate code inspection, although there’s also obviously a lot of manual inspection. I know Coverity has been involved in doing scans of code and making those available to different projects.

What are some of the things that are coming online that are further raising the quality of open source? How are you seeing that evolution happening across open source projects?

Joe: Slowly. Things like Coverity are being developed, although that’s not an open source tool. They simply like to use open source because it gives them a good way to show off. They can actually talk about the results as opposed to if they do work for commercial customers.

Scott: Right, exactly. But at the same time, some open source projects find the information they get to be extremely valuable, even if others may just find that they get a ton of false positives.

Joe: I would like to see the community come up with more tools of that nature, but I am not aware of any. You mentioned the Mozilla one, but I’m not aware of any major efforts in that direction.

Scott: Do companies like Novell take a little bit of that on? As projects move from openSUSE to SLES and SLED, is there a certain level of hardening that happens, or is it not really needed? Is it more the case that the code is of a sufficient quality to be enterprise ready as it is, and those things might be nice, but they’re not really essential because the code is good enough where it is?

Joe: There’s additional quality control and hardening at every step. There’s the initial project, with all the alphas and betas that they do, and then there’s the stage when a project like openSUSE decides to go ahead and adopt a particular version.

We do additional testing, and then we go through our alphas and betas and so forth. Then, there is the additional testing that goes into putting together the enterprise release. So, there is a little bit of hardening and improvement at every step.

Scott: It sounds as if it might be fairly typical as you do your testing to maybe find some bugs that didn’t get found in the alphas and betas, so you work with the upstream project. The quality goes up another notch there and then as SLES and SLED are put together, and as Red Hat, Sun, and a lot of other distros do their work, more things are found, pushed upstream, and fixed.

There’s this incremental hardening and improvement of the code, and by the time it gets to any vendor supporting the enterprise release, it has gone through a lot of layers, each of which made it better. Is that a fair characterization?

Joe: Yeah. And the additional layer, of course, is once it’s released to the enterprise version, then there’s additional testing. It would be nice to say that there are absolutely no bugs shipped once it gets to that level, but of course you do find things, and you do find areas where you can improve it. And then the process starts all over again.

Scott: I do have one other question. It seems like for a lot of things that are released by companies like Novell or Red Hat, the architecture must have been sketched out around a whiteboard, and they had to staff developers on it.

But then, the output of that is code that’s put out for community inspection. Are some of the features designed in a traditional way like that, in a typical conference room and white-boarding scenario? And then, are the architectural design documents, and specs, and so forth released as well?

Joe: There are occasionally similar things with open source projects without necessarily any corporate influence except to say the various participants are employed by the different companies.

For example, I’ve gone to things like FOSS camps and developer summits where you get a bunch of developers around a table working on a vendor-neutral program. They often do essentially follow the same (or some of the same) software development process that you would see if you were to go into a Novell office where they are working on something. Let’s not forget that a huge percentage of participants in FOSS projects work for companies that have an interest in FOSS, and they have day jobs that involve standard software development methodologies and those methodologies are brought into FOSS projects (and vice-versa, of course).

And so, it may be less prevalent, but I think that you do still see some of that, depending on how big a feature is, or whether you are talking about iterative change versus the difference between KDE 3.5 and 4.0. Then, I think developers still get asked by the other developers, “Hey, write something up and show it to us”, because you can’t code something like that from scratch and show it to people.

Scott: Well, it has to be comprehensible, right? There’s got to be a higher level of abstraction that people can look at, because it might be 30, 000 lines of code.

Well, I want to be sensitive to the time. Are there any closing thoughts you have or something you want to circle back to?

Joe: I guess I’ll take the opportunity to reiterate that we’ve put out (finally) the 1.0 version of our build service, which allows people to contribute more directly to openSUSE. So I’m looking forward to seeing more direct involvement in the direction of the distro and in contributions.

Sean: OK, that seems like a good place to wrap up. Thanks for taking the time to talk with us.

Joe: Thank you.

Bookmark this:

Interviewee: Mark Kortekass

In this interview we talk with Mark Kortekaas from CBS. In specific, we talk about:

• Choosing and implementing technologies for the production environment
• Relationships with open source communities
• Tailoring the organization for maximum benefit from open source
• Building infrastructure for capacity and agility
• Future-looking, cloud-aware data centers

Sean Campbell: Mark, would you tell us a bit about your professional background?

Mark Kortekaas: I am the Chief Technology Officer for CBS Interactive, which among other things runs cbs.com, cbsnews.com, and cbssports.com. My group runs core technologies for CBS Interactive.

Sean: What’s the technology that underlies those sites, broadly speaking?

Mark: We’re primarily open source underneath, with everything running on top of Linux variants of some sort. We use combinations of PHP, CGI with Perl, and some C-based stuff. We’ve got some instances running Tomcat, we have Lucerne running for search, and we’ve got some MySQL around.

We also have some commercial search applications, both internal and external, and we’ve got some commercial database applications running, such as Oracle. So we don’t limit ourselves to a purist view that it’s open source or nothing, but from a base design perspective, we run on top of open source. And that sort of lends itself to a lot of custom built solutions, on top of that.

Scott Swigart: Tell us a bit about your decision-making criteria, when you’re evaluating databases, operating systems, or something else, and you have open source and closed source options to choose from. You mentioned a preference for open source but some inclusion of commercial software as well. How do you make the decision on a case-by-case basis about whether you’re to go with a particular open source option that’s available, or whether you need a proprietary piece of software?

Mark: We make the decision about what technology to use based on the problem we’re trying to solve, so there’s no one size fits all answer. There are times when a given project’s resources will be aligned with a particular technology, so they’ll decide to use technology “X.”

It’s a sort of a hammer problem–if you’ve got a hammer, everything looks like a nail, and if you’ve got a screwdriver in your hand, everything looks like a screw. We’ve got some development groups that are very Perl-focused and prefer to do things in Perl. We’ve got other development groups that are very PHP-based and therefore lend themselves to PHP. We have other development groups that are very Java-based and lend themselves to doing everything in Java.

Still, the biggest consideration is the business problem we’re trying to solve, so if we’re trying to solve a problem that lends itself to commercial solutions, we’ll use commercial solutions. There are certain things that call out for commercial solutions, like when there are control issues and things like that. How are you doing control systems for things that are in scope for Sarbanes Oxley, for instance?

Other times, when there aren’t outside pressures, we look at the problem more from a resource perspective.

Scott: I saw you interviewed a bunch of different times about the March Madness implementation you guys rolled out. I know that MLB made a big investment on the Silverlight side, and to be frank, there were some challenges with it–especially this roll out at the beginning of the month.

I’d be curious about what decisions you made around that, because that’s obviously a challenging platform to work with. You’ve got millions of people that could log on, many of whom aren’t all that technical, and they’re probably not really able to deal with failures the same way that maybe somebody with a little more savvy would.

What was the decision making process for that, because that’s obviously a pretty big signature piece of technology you are pushing out.

Mark: We didn’t utilize Silverlight, but not because we felt that Silverlight was a fundamentally good or bad technology. We just knew that using just straight Windows media, which is what we delivered March Madness in this year, in an embedded window on a regular HTML-based page would work.

Silverlight is fairly new to the market, and people are just getting the clients and all that. We were concerned mostly with traditional things like penetration rates, the ability for people to download the client, how long that might take, and so on.

This is a case where we did look at other technologies, but at the end of the day, we went with something that we’d used in the past and knew we could make work again.

We looked at the decision not so much from a technology perspective, but more as history of success. I think you always face the choice between rolling out new technologies versus staying consistent.

There’s never a good answer for that. There are certain times you need to push the envelope, there are other times you need to go with a more conservative approach, and there are times we take those different approaches at different times, depending on the property.

Sean: Closed source products all have armies of sales people with data sheets, marketing vehicles, and demos, and all that sort of stuff. How do you go about analyzing a given open source effort, whether it’s a new product that you want to transition to or a new version of an existing one? How do you guys analyze the effort to determine whether it’s reached the right stability point and so on?

Mark: You look at a couple of things there. You start with the problem you’re trying to solve and the support criteria you’re trying to deal with.

I also think you try to do things in test modes and trials and things of that sort. Apache’s a good example–you look at the last version of Apache 1 and you compare it to 2, and you see that 2 has a better threading model, and over time, you want to get the better performance that will give you.

Does that mean that when we rolled out 2 that we rushed it out immediately? No—we waited a fairly long time to roll from Apache 1 to Apache 2, and we did it in a fairly controlled environment. We rolled it out in a smaller scope first to see how well it would perform and to iron out the issues.

And then over time, as you get the hang of doing rollouts and things of that sort with it, you eventually upgrade your entire farm. In our case, I’m pretty sure we’ve upgraded the whole thing at this point, but for all I know we’ve still got some Apache 1 farms around.

We wouldn’t roll out a new version of Apache the day before a major event, unless there was a sizeable security issue that we were trying to address. So we have to take a risk-benefit approach, and we try to mitigate risk whenever possible.

Sean: With a project like Apache, you’ve got a really clear roadmap of features, and there’s a real active community. What about bringing in a little bit more of a dark horse, like a project that you’ve had your eye on, and now you want to bring it in?

Mark: We would try to bring it in on small scale, in a project that is off to the side a bit, rather than risking disruption to the main line. There are always new things going on, and so we’ll bring things off to the side, set up a new farm, and do some trials. That’s how things like Tomcat and Lucerne got implemented into our environment over time.

After you try it in a small environment, if it works, you start to use more and more of it.

Scott: What are some of the advantages that you see in the open source model, in terms of what it has enabled for your organization?

Mark: Anyone who says that cost doesn’t factor into their thought processes is probably being a little disingenuous. Clearly, cost factors in over time, and in the end, that does weigh in to any soft process we do.

One thing that open source helps with is when you have a new project you want to try. You don’t know what’s going to happen with it. It may turn out to be a big business, it may turn out to be a small business, and it may not even exist anymore in three months. You just want to try it out and see if it works.

Being able to do that with open source means you don’t have to worry about the financial implications so much–it’s possible to do trials without signing data agreements and things of that sort that may require you to make a purchase three, six months down the line. That lets us more fully understand the business implications for what we’re doing before we commit significant resources.

This is easier to do when you’re dealing with a known business, but if we’re trying to set up something new that we’ve never done before, it’s hard to know what the business problems are going to be. The ability with open source not to have to think about financial implications so much in the early stages is a big deal and has to be thought of in terms of flexibility.

I don’t know if the access to source code is so relevant, to be honest. It’s rare that we take advantage of that, although there are times that my engineers will go in and find issues inside the kernel or issues in Apache and will submit changes back to the main project. That does happen on occasion, but by and large, we don’t really have the staff time to go in and deep-dive code unless something is really wrong.

Most of the time, we’re doing things through a known API, so I could just as easily work on those with a commercial web server as well as we do with Apache.

Scott: Given that point, what’s been your level of engagement in terms of contributions to open source projects? That contribution could be actual code, or just communities that you feel like your team’s more actively involved in, either helping other users or just solving problems. Is there anything you could point to in that particular area?

Mark: I know we have done some contributions on miscellaneous bug fixes and things like that, where we’ve submitted things back. By and large, it’s not one of the things we’re necessarily out there trying to do.

We have done it when we found certain things, but my developers are not kernel developers. They’re not core Apache developers or things of that sort, so it’s not something that we’re necessarily striving to do. But as we do stuff that does have relevance outside, we have made things available. I think most of those have probably been done under employees’ names directly and not sponsored by the company at large.

Scott: You mentioned that you were largely a CentOS shop and that, obviously, with open source, like you mentioned, you can try things out without incurring a lot of licensing expenses. Are there places where you have opted to pay for support agreements with Red Hat or MySQL, or things like that, or do you largely just self-support?

Mark: Certain companies have a model that’s not true open source; it’s sort of a hybrid, where for non-commercial use you can use it free, but for commercial use, you have to pay a licensing fee or a support agreement.

We have done those sorts of things, on occasion, but we largely support our own kernels and do our own off-shoot kernel builds.

There hasn’t been a lot of benefit for us in support agreements, although we have had discussions with them. I think those agreements work really well for a lot of companies, but our environment is fairly fluid in terms of what we’ve got done, and we have some very high level engineers that keep things running.

Sean: Regardless of whether you’re using proprietary software or open source software, you could define three ways to keep things up and running. One is that you can enter into the support agreements with the Red Hats and the MySQLs or go with a support aggregator, like an OpenLogic, who supports a wide variety of open source stuff.

Second, you can be an organization that sort of pulls in consultants when you need hard stuff done and you need hard problems solved. Finally, you can have some extremely technical staff, who basically are expected to solve any problem.

It sounds like for your organization, you’ve gone with that third option in terms of having some extremely technical people who are sort of go-to people if others in the organization are stuck. What do you see as the merits of going that route?

Mark: I think the question sort of assumes that we went down that path as a conscious decision. I don’t know whether that’s the case, since many of the systems that we’ve got predate me and the company.

There is no doubt that we’ve got some very technically savvy people. I can speculate that very technical people with a sort of “build it yourself” mentality tend to gravitate toward open source, because that’s their personality fit. And therefore, if you’re hiring for open source type positions, that’s what you tend to get.

So I think it’s sort of a self-selecting technical group of people that are out there, in terms of skill set. Candidates that we see traditionally come in knowing a lot of that and having the feeling that they want to go off and build and tinker and things of that sort.

I think one downside to open source is that you do have to have a fairly technical organization around you, and you have to have a slightly higher risk portfolio or a risk perspective than you might if you were in a different organization. I think there is an aspect of risk acceptance that we take on and that pervades the types of things we do.

We are not a financial institution. We take calculated risks with what we do, and we try to move very fast. Therefore, we feel open source helps us get there, but other people may not have the same wherewithal. Documentation on some open source products is frankly pretty terrible. It requires you to go in and spend a fair amount of time just figuring out how to make it work.

In order to be successful with a project that doesn’t have good documentation, you have got to have good individuals around that want to make it work and will spend the time to do it and not get frustrated at the lack of documentation.

I do think staff selection is a major consideration, in terms of people’s abilities to make open source work. Staff acceptance of it is another one. If your shop works in a risk-averse environment, where tinkering and mucking around with the innards of things isn’t promoted, I don’t think open source is probably the right fit for you as an organization.

It should very well be noted that there are certain aspects of CBS where that risk portfolio would not be acceptable either.

Sean: We have talked to some different CIOs, and to paraphrase, I have had people say, “We just need our servers to work, our databases to run, for everybody to get email, and for it to be easy to maintain, so proprietary software makes sense for us.”

So for you team, which is doing innovative customer-facing web software, you need to be as unconstrained as possible. You need to have control and freedom, to change or not change, to develop or not develop–whatever you want, and open source gives you that. It gives you that at a higher risk, but that risk makes sense for the kind of stuff that you guys are trying to do.

Maybe, for other parts of CBS, like people who are office workers or information workers, they have got a different infrastructure that makes sense. Do I have that right?

Mark: That is a fair way of saying it.

Sean: Let me ask you a question in a different area entirely. What do you think people would find interesting about the CBS infrastructure or the user base you have to support, either internal or external? What do you think is unique about it, and what in those unique features ties you to some open source efforts?

Mark: One thing that makes what we do challenging is that peak-to-average server-load ratios for us are completely different than those in most businesses

We build for a nominal load of 10%, because we know we’ll need that capacity when we get a peak. For our sports business, for instance, football Sundays and the first couple days of the NCAA tournament are going to very busy days for us.

We build an infrastructure around massive scale for very finite events. The challenge that we face is how you deliver when you have a massive audience, yet never know what the peak is going to be year to year. We have estimates, but we never know how good they really are until the event comes. For example, we didn’t know exactly how large March Madness was going to be this year, and so we hoped we got it right, and we were successful. At the same time, you don’t want to overspend either.

I don’t know that open source helps me here. I think open source gives me a lot of benefit in the development realm, but I’m not sure it gives me anything in the capacity one.

Scott: That is an interesting transition, because I want to ask you a more development-oriented question. I think we have all seen web users who fall into the camp of guys like us that probably can thread through a piece of data on the web page very quickly, look and find what is relevant and what is not.

And then, even today, you have a mass of users who may be confused by the density of information, by the way the site composes itself, and the way it folds out information.

Do you feel that the open source development frameworks are moving more rapidly to give you more ability or the things you like in the development frameworks that are being pitched by closed source companies?

It’s an interesting kind of interplay happening now, because ASP.NET is a pretty cool technology by all accounts, but PHP is getting a lot of momentum. There’s a lot of community building around it; Microsoft has even embraced it and is running an IS, so there is an interesting battleground there.

Is there an area where the ability to build a good usable infrastructure and have strong development tools plays well with your user base and comes into the decision making?

Mark: My feeling is that what we use on the back end has no bearing at all to what my consumers feel. I feel that I’ve got a product issue in terms of what products I offer to my consumers and I’ve got a back end technology problem on how to deliver that product.

I honestly believe that if my developers were 100% Microsoft .NET developers, I would be able to deliver our product on .NET. It happens to be that my developers are not .NET developers. I don’t feel that, at the end of the day, open source enables a new business. I mean what Microsoft has done has been very successful. Oracle has been very successful.

You know, there are pros and cons to using either open source or proprietary technologies. You can solve your problem in any number of ways, and I think your skill sets and how you’ll choose to go about developing those solutions will change, based on the infrastructures you use.

I don’t think the problems you can solve are necessarily any different with proprietary versus open-source software. The economics around the solution might be different, but that’s a different question.

Sean: That’s interesting, because people sometimes have pretty rigid opinions about development technologies, rightly or wrongly.

Some people take the line, “I shall only code in C#,” or “I shall only code in PHP.” The fact that Google’s sort of “data storage in the cloud” solution that they released today has a Python interface is creating a bunch of consternation and happiness, depending of what camp you’re in. Of course, you’re always going to have that.

Mark: On the developer workstation side of things, I’m a firm believer in emacs and make my SAs install emacs on all the machines, just so I can personally edit files.

We have programmers that still code in a shell window and use VI or emacs, depending on their predilection. We have others that are coding on the exact same systems that have chosen to use an IDE on their Windows desktops and use more traditional IDE-based and graphical-based front ends that just happen to save it off to a subversion repository. And then we pull it off on the SVN on the other side to actually run it in production.

I think there’s a lot of overlap in a lot of ways, and it’s not an either/or scenario. I think there are things about some commercial software that works really well for developers that doesn’t necessarily change my approach.

Scott: You mentioned fairly unusual capacity-planning challenges. One of the things that you hear around this type of problem is the concept of fabric computing and grid computing, where your data center is just a pool of CPU, disk, and memory capacity, and you can allocate things fairly dynamically.

I don’t get the impression that a lot of that is really happening in practice, though. Today, do you mainly model loads to estimate capacity for things like Sunday football, with a fairly static allocation of resources?

Do you see virtualization or other technology coming online that would let these kinds of systems be more dynamic and, for example, spin up 50 more virtual web servers or provision 50 more systems automatically? Do you see that kind of stuff becoming a reality? Is it already somewhat of a reality?

What can you tell us about the dynamic landscape of the data center versus the reality of today?

Mark: It’s a great space, although there are a lot of issues that are going to have to be faced, in terms of the way applications are written, provisioning times, and things of that sort. If I’m going out to a vendor and I want a certain number of CPU hours or whatever, there are a whole bunch of models of how you sell this stuff.

If I’m going to go and do a reservation for six months for a particular sport season, I still have to plan ahead of time in order to ensure that I’ve got enough capacity. Today, it’s largely a way of solving capacity with different financial sort of parameters around it. That said, a lot of applications don’t lend themselves to not being run in the same data center, and there are issues with not having access to the same raw storage, for instance.

As you look at this, I think over time what you are going to find is people are going to start to write applications that are more “cloud-aware,” and I don’t think we’re quite there yet. There has to be a fair amount of time people spend understanding how to write applications that can do things like scale quickly without breaking the application. I have a feeling that a fair number of applications out there, both our own as well as others, will be fairly particular about data latencies and things of that sort.

The other observation I’ll make is that I don’t have many hours to pull up additional servers. So, if I am looking at a sports season launch, that only happens once a year. And I have to hit that out of the park on that day. Therefore, I have to have an amount of capacity already in the racks, spinning, tested and everything else. The fact that I could spin up another 50 additional servers in eight hours doesn’t really helped me if I have already blown my first day.

There’s no miracle bullet out there that says, you can just do it. That said, I think there are some very interesting things out there, in terms of capacity, but even if you look at the largest of the farms that are out there, they are still being built on some economic reality. And there are still some peak periods that they are trying to hit. They are using the excess capacity to have around the edges to basically put these clouds out there.

So you still have a pretty traditional problem of capacity planning. It’s just changed how you pay for it. If I need 50 servers, I still need 50 servers and I have to plan for 50 servers. Someone has to pay for 50 servers, if I really want them. That’s my high-level observation. At the same time, I think anyone who is not looking at these things as part of their long-term strategy, and taking a close hard look at how these things work, isn’t really doing their job.

Scott: Well, thanks for meeting with us today. We’ve enjoyed talking with you.

Mark: Thank you.

Bookmark this:

Interviewee: Kevin Kluge

In this interview we talk with Kevin Kluge from Zimbra. In specific, we talk about:

• Zimbra’s history and finding a niche
• Profitability, ownership, and managing commercial products
• Interacting with the community
• Source code availability and its impact on the public
• Managing customer needs and usability

Sean Campbell: Kevin, would you tell us about your background and your role at Zimbra?

Kevin Kluge: At Zimbra, I’ve had a variety of engineering management positions since I started in September of 2004. The company was about nine months old then, and we hadn’t shipped any products at that time, and we hadn’t decided to go open source. I’ve managed various parts of development, QA, and support.

Prior to Zimbra, I worked in the messaging space at a variety of companies, including Onebox.com, Phone.com, and Openwave, as well as Corvigo, which was an anti-spam company, also doing mostly engineering management..

Sean: Tell me a little bit about the history of Zimbra. What’s the big picture history, going way back past the Yahoo acquisition?

Kevin: The original concept for Zimbra was for an email archiving appliance. This was prior to when I joined. When they guys started talking to potential customers, most of them were satisfied with their archiving solution, but they were dissatisfied with their mail system. So the guys had implemented some early Ajax technology around archiving, and they were able to repurpose it into a mail system.

And so the original concept morphed, and we were going after larger customers who needed a small set of functionality, but they weren’t really ISPs–it was more companies like airlines, banks, and so on, that had a large number of boundary workers that they wanted to put on the mail system.

We shipped the first product release in about April or so of 2005, and we started getting real customers in late 2005.

Sean: Was some of the drive to have Zimbra constructed the way it is an effort to make the back-end mail server integrate with a lot of front-end points, all the way to MAPI integration with Outlook, high-end version?

It’s a fairly full-featured suite, and I find that technology stacks generally have a little bit of a blind spot. I had a Mac for the last year, and there were certain things in that platform that I loved, like the add-in chat client was probably the best chat client I’ve ever seen on any platform. But at the same time, Exchange back-end integration if you really wanted a shared calendar was lacking.

The open source communities always had a little bit of a gap there, and I wondered if that was an intentional thing Zimbra was trying to shoot for–to reach out at a multi-platform level via a web interface with a really robust shared calendaring solution.

Kevin: There were two things where we thought there were real opportunities in the marketplace. Certainly calendaring was one. Even the commercial calendaring products weren’t that great, and the open source ones were even more lacking. That was certainly a large opportunity that we saw and, after doing the basic mail application and address book, that was the first thing we went after.

We’ve also done a couple of other strategic things, because we noticed that a lot of the open source groupware was very difficult to install. We decided from day one that we would have a strategy of using existing open source components where possible and we would bundle them into our download, assuming that the licensing allows us to do so.

We wrote an installation mechanism that wrapped all the different components and then we also wrote an admin interface and management tools. These allow administrators to change one thing and have it propagate to two, three, four different open source components that need to know about it. We think that this enabled people to download and use Zimbra who wouldn’t have been able to if they had to download it and compile it or download it and 10 other packages, each of which had to be a particular version.

You’re also right that we did go after a cross-platform environment. We wanted to be the best desktop neutral solution for collaboration. We didn’t want to be a Linux desktop solution or a Windows desktop solution or a Mac desktop solution or a mobile solution. We wanted to allow people to choose whatever client they wanted, from the beginning. It took us a while to implement some of the features, but that was definitely the strategy, and I think it has been very helpful in getting both community adoption and commercial customers.

Sean: It sounds a bit like what Ubuntu’s doing with desktop distributions–they keep trying to make the process more and more elegant for installs, even with the integration of Ruby and so on. They want to make it increasingly, progressively easier to get the distro down on metal.

It sounds like you want to give someone a collaboration messaging solution that’s fairly easy to just get down under the metal and have a decent set of features.

Kevin: Exactly. Our target market is the IT administrator more than the developer, which is a different skill set.

Sean: On a different topic, since you were acquired by a “for-profit” company, what’s it been like to live inside that bubble? Depending on who’s talking about the MySQL acquisition, for instance, there’s either a massive amount of consternation, fud, or benefit.

You guys have lived in that situation since September, so what’s that been like?

Kevin: Certainly for us, it has been positive. Yahoo is a great company with a lot of resources and a lot of technology in-house that we now have access to, as well as resources in things like the hosted offerings we’re now working on. We’re able to do it with a speed and a scale that we would not have been able to do if we were a stand-alone company.

In terms of the commercial side of it, that’s been helpful as well, because Yahoo has more mature sales operations facilities than what Zimbra had, so we’re working to take advantage of those. But in terms of the product strategy and how we do support for our customers and how we interact with the community, it really hasn’t brought much change.

Mostly, Yahoo’s instructions to us have been to keep doing what we’re doing. The one thing that we are doing aggressively right now that we weren’t doing before is to build out a hosted offering. You can imagine, given how Yahoo runs its data center, that would be a priority. We’re working to put up a hosted ZCS offering. And we’re going to go after a wide variety of target markets with hosted Zimbra.

Scott Swigart: Let me ask a question in a little bit of a different area, in terms of some of the prominent old-school open source projects; like Linux and Apache.

The Apache Foundation has a way of doing things, where the project has to have at least three different committers who are not part of the same company, and things like that. But a lot of the kind of vibrant open source that you hear about–SQL, Alfresco, Zimbra, Zend did their work around PHP, XenSource and their work around the Xen hypervisor. There seems to be an open source project that has a primarily–if not exclusively–for-profit wrapped around it.

How do you view the open source landscape, and what do you view as advantages and disadvantages of some of the possible models? I’m thinking of the range between the one that’s basically “community” or at least no central corporate sponsor, one where there’s a primary sponsor, and then all the way over to proprietary software where there’s a primary sponsor and the code isn’t even available.

Kevin: I can tell you a little bit about our experiences, and where we’ve arrived in trying to navigate those three different options. When we were initially discussing whether or not we wanted to go open source, there was quite a bit of concern that we’d give away all this value, and then how would we make money?

The answer that we kept coming back to that had been proven in the market is that you can charge for support and people will pay for it. That lets you give something back to the community that’s a great piece of software, but as long as you can keep your development costs under control, you can still have a very successful business model. That’s where I think a lot of the older, more established companies had ended up.

We recognized that there was an opportunity to create a great product that gives a lot of value to the community. At the same time, we saw that we could also create a little bit more profit for ourselves by adding a few additional features that we believe that enterprises or large ISPs would want, and that they would be willing to pay money for.

When we started talking about that, and looking at what we wanted to get built in the next few years, we started to identify features that we could leave out of the community edition and still have a spectacular community product, and then potentially get a little bit more profit for the company by putting it in the commercial version.

Those discussions were very lively, and very animated. We really did debate almost every single feature, and try to decide which bucket to put it in. We have been able to do that successfully, so we do have maybe five or 10 features that are available only in the commercial edition. So even though we don’t give away everything, we still do give away enough so that people can be very successful running a collaboration system.

The community is extremely helpful on two separate fronts–one is the product management front, and the other is the quality front. In terms of product management, we have a Bugzilla system that’s open to the world, and then we also maintain what we call the “PM Portal.” Basically, it’s a website where people can go and view what features we’ve had, and what releases, and what we’re thinking about for the next major release.

When we make product management decisions, we look quite a bit at the data in Bugzilla. We look at what people are voting on, and we look at the comments they’re making about the features, and the use cases that they have for those features. That plays very significantly in our decisions about what features we’re going to go do in what releases. On the quality front, we get feedback primarily from Bugzilla, and then also of course the forums. The community has really been invaluable in helping us get new releases up to a stable point quickly.

We’ve done quite a bit of learning there–initially we did beta releases, both of the Community version and the Network version, for the 3.0 and 4.0 releases. For version 4.5.0, we did not do a major beta release, and the quality of the 4.5.0 release was below where we wanted it to be. I think that was directly because we didn’t do a good beta cycle. Then for the 5.0 release, we did a beta with the community, but we only did the commercial beta toward the very end of the cycle.

We think this was the best compromise between the different needs. It allowed the early adopters in the community to get the software, and install it, and give us feedback as early as they wanted to. At the same time, it didn’t put software out there for our commercial customers to install and then potentially have trouble with it.

The community’s involvement both in helping to define the product, and then helping to refine it has been tremendous. Without the community, if you’re in a proprietary commercial situation, you have to run these large-scale beta programs. I don’t think you get feedback as quickly, and I think it’s more expensive for you as a company to get that feedback.

Scott: Does the community actually submit patches, or is their role just posting bugs that they find, or posting issues that they find, and then you task your engineers with making fixes?

Kevin: That’s an area where we really weren’t sure what we were going to get when we made the decision to go open source. In total, I think we’ve had far fewer community code contributions than we expected, and my suspicion is that’s because Zimbra is primarily a product for the IT administrator and not for the developer.

We do have a process set up to accept code contributions from the community. The few code contributions we’ve gotten have been great, but it really hasn’t had a major impact on the product, aside from the fact that we’re bundling all these other open source components. I’ve been a little bit surprised by that.

The community has contributed a number of localizations for us that have helped spread ZCS to users that probably wouldn’t have run it otherwise. That was really helpful, since we were a start-up with no internal localization team.

Scott: What feedback do you get from customers about the value of making the source code available? Could you talk a little bit about the value that you think customers get, and the value that Zimbra gets back?

Kevin: The number of people who attempt to make use of the source code is fairly small–certainly with our commercial customers, it’s a very small percentage.

But for those who do, and who have the skill set to do that, they can make custom changes that they can reapply on releases, or potentially some of them will actually go in and make bug fixes for their own personal versions, because they can’t or don’t want to wait for the next maintenance release where we would address the bug. So, it definitely creates some value there.

That’s also helped some customers to be less concerned about the Yahoo acquisition, because they know that no matter what happens with any acquisition of the Zimbra technology, the source code has been released under a licensing agreement that can’t be changed.

So it’s definitely been helpful to some extent in the sales cycle there, as well as giving customers a little bit more of a piece of mind. We wanted to see the browser continue to evolve as a platform for applications, and we felt that we were pushing the limits of what could be done with browser-based technologies in our Ajax application. By putting the source code out there, we hoped that other people would take a look at what we had done and learn from it, and go do things as good or better than we had done with other Ajax applications.

The last part that we have been working on creating is a community around Zimlets–our mechanism for extending the functionality of the web app. We do have to define Zimlet APIs, and people could write code to the Zimlet API without looking at the underlying source code. But having the source code available certainly is helpful, when you’re trying to figure out why your Zimlet doesn’t work or when you want to implement some particular feature in it.

Scott: When Sean and I were at OSCON last year, people kept starting off their presentations with almost an apology for being a for-profit company.

How do you balance between your commitment to the community and the need to be profitable? As you work through decisions about what to offer as open and free versus what to put only in the enterprise version, how do you communicate with the community about those issues?

Kevin: We’ve given a fair amount of attention to that. Others at Zimbra might answer this differently, but from my perspective, I’ve always remained focused on what it was that we were giving to the community, and to creating a fantastic open source Collaboration Suite. We think we have done that with the constraints that we’ve put on in it, in terms of not handing out five or 10 additional features.

A key consideration for me has been that, while we may choose to limit the distribution of some features, doing that should not prohibit someone from being successful with the open source version.

Sean: Since your customer base tends more toward IT admins and less toward developers, that must inhibit the standard open source model of people contributing the features they want directly.

So what’s the filtration mechanism in terms of your average IT guy saying, “I don’t like the way you built this interface, ” or, “My users are complaining about the way the web email looks, ” and that kind of thing? How do you filter those requests to get them into the product; I find that for some open source projects, that’s a little bit of a struggle.

Kevin: I’ll make a couple of comments on that, and the first is that, even without the contributions of the community, there are a lot of people in Zimbra who are passionate about usability and the interface. They’ve done a lot of thinking and work and really had some tremendous ideas in order to enable us to create the application that we have. That’s definitely been a part of our success there.

The other part where the community has been very helpful has been interacting with us via Bugzilla. And we have built forums that we spend a fair amount of time on, but ultimately for us, for something to get done, there needs to be an entry in Bugzilla.

We’ve tried to steer the community folks into putting their ideas into Bugzilla, and we also have a couple of guys on staff whose primary job is to work with the community. Part of that is going through the forums and trying to figure out what people are complaining about, or else where someone has a great idea. We make sure we get that into Bugzilla so we can evaluate it.

The other part that has helped us is that pretty much everybody knows how to use email, so we’re not developing an esoteric product. We’re developing something that all of us use every single day. Of course, we run Zimbra internally, and have for several years now, and that has helped us work on usability as well.

Scott: Would you like to close off with something?

Kevin: To me, the most educational part of this process has been learning how to work effectively with the community, and how to integrate their input into the product management process. Then how to also work with them to give them something that, in terms of the beta cycle, gives the community something that’s good enough to be useful, yet immature enough to still give them an opportunity to really impact the quality and the feature set before it got released.

We’ve spent a lot of time working with folks in the forums, and we’ve built tools around Bugzilla to let us view the data that gets in there from the community. We had to set up some processes for when the community does file a bug, what we do with it. How do we make sure that all the information we need is in there? How do we make sure that we can reproduce it, and that we know exactly what the problem statement is before it goes to development?

Certainly, the community adds a lot of value. But you don’t want your development team to turn into a support staff for the community, at least not if you’re going to scale to a very large community base, which we certainly want to see happen with Zimbra. To me, that’s been an engineering management challenge, and it’s been a lot of fun too; the end reward is really great. I’m sure we wouldn’t be where we are today with the feature set and the quality that we have, if we didn’t have the community helping us.

We’re also really excited about the future. We continue to add big features to the product, the most recent being Zimbra Desktop and our support for BlackBerry devices with Zimbra Connector for BES. We’ll keep bringing the Zimbra experience and data to new platforms. And we’re going to launch a hosted Zimbra offering shortly that will make it even easier for admins to provide their users with the Zimbra experience. The community keeps growing, too, and we’re excited to see that continue in 2008.

Scott: It’s really been great talking to you.

Kevin: Thank you.

Bookmark this:

Interviewee: Ross Gardler

In this interview we talk with Ross Gardler – Service Manager for OSS Watch. OSS Watch is an organization that helps institutions in the UK make decisions in and around open source topics. In specific, we talk about:

• Different cultures in open source projects
• Notions of value and control in the software business
• Commercial involvement in open source and giving back to the community
• Comparing open source in the UK and the US

Scott Swigart: Thanks for taking the time to chat. If you don’t mind, could you give us a brief introduction?

Ross Gardler: Sure. I’m a computer scientist by background, but I’m in a management role now in an organization called OSS Watch, which is a publicly funded organization in the UK that provides an advisory service to the UK higher and further education institutions. We talk about things like “How would you engage with open source in procurement?” and “How do you develop open source in research projects?” Basically, any questions to do with open source come through to us.

We’re not an advocacy service–we don’t go out and say that open source is the best or only way. We try to enable people to make a fully informed choice.

My most active involvement with open source is within the Apache Software Foundation, where I’ve been a member since, I think, 2003.

I’ve done some dabbling in other things on the outside, but in my earlier life as a contractor, I found myself getting drawn more and more into the Apache way of doing things, and I’m fairly ingrained. These days I’m lucky enough to just do it for fun. I get paid for doing the consultancy and advisory stuff now, which is great. I can hack purely for fun again.

Scott: You’re probably in a good position, then, to talk about one of the first really eye-opening things we found in doing this project–that open source isn’t just one thing. Apache HTTD does things one way, the Linux kernel has its own culture, and MySQL is different still. What are some of the different common models you see for how the software is actually built or the philosophies that exist in different projects within the open source cosmos?

Ross: I think from the outside that they often look massively different. Take the two extremes, of say the benevolent dictator of the Linux kernel, where Linus theoretically has all the control over what happens in there, and compare that with something like the Apache Software Foundation, where it’s what they term a meritocracy and there are potentially a large number of people involved. I think there are currently 60 people involved in the Project Management Committee within the HTTPD project.

But once you get involved, they’re pretty much the same thing. Linus is there because the community wants him there, and they want him there because they respect him, he listens to their opinions, he works with them on patches that need to go in, etc.

At the end of the day, because the kernel could be forked at any point, if Linus didn’t represent the community properly, the kernel would be forked and the community would go off somewhere else. It might not be a meritocracy, but the community feels that they’re being listened to. So there are more similarities than perhaps there might first seem.

Sean Campbell: When we talked to the Apache Foundation, they talked about how they went for a commercially friendly license. But My SQL is known for unlocking the GPL and letting you embed My SQL with a commercial product. There is a lot of nuance around open-source licensing.

Ross: The real difference, at least for our target audience, is the type of community that builds up around the license. If you decide that you’re going to go with a GPL because that will enable you to dual license–if that’s going to be your exploitation route–it has a significant impact on the types of people that are going to contribute to your project, because it means you have to have full control over all copyrights. So any contributor has to sign over their copyright.

Some people simply won’t do that, especially for those of us in the academic community. Institutions like to hold onto their copyrights. The tech transfer groups within universities, who are traditionally responsible for exploiting advances from within their research groups, are used to dealing with patents.

That’s particularly true in the engineering sector and the sciences. When you go to them and you say that if they want to engage with an open-source project, they’re going to have to give away their copyright, that’s a no-go in many cases, straightaway. And that can have considerable impacts on collaboration between universities partners.

Sean: I saw that you have put up a couple of blog posts about the idea that just because someone puts a GPL bumper sticker on a project does not necessarily mean the project will just be embraced by the world.

Where do you think open source communities go right or wrong when they start to build a project?

Ross: I’m a great believer that the strength of open source is not in the license. The license really has come about to protect the passion and development methodology that has sprung up around the free and open source development methodology.

If you look at the history of free and open source software, you’ll find that we did collaborative development before the free and open source licenses existed. The licenses emerged to protect that development model as it became threatened by closed source approaches.

I think we have begun to lose sight of that over recent years. I shouldn’t really single MySQL out, because they are not the only ones doing dual licensing, but of course they are very much in the news at the moment.

They have used a GPL license and made the software available, and they have done an absolutely fantastic job of producing a really fantastic product and company. Nobody can question that.

But what they haven’t done is create an open development community. Their business model has been very much about closed development. It is true that people could get at the source. It’s not totally closed, and I don’t mean to imply that it is. But it is certainly less open than in something like Apache.

Again, I wouldn’t like to really single Sun out. I mean Sun is doing some fantastic stuff in the open source arena, but they also do some questionable stuff in some areas. Sun seem to have a certain need to control the software. Again, this is not necessarily wrong, if they have a business model that requires they control the software, I can understand that.

But let’s not pretend that’s an open development model. Open development is about saying, “I know this software can do something really useful for me. And it can almost certainly do something useful for other people as well. Let’s free it up and see where it goes.” Often you will end up with something that’s far better than you could possibly have come up with if you had retained full control.

However, this doesn’t mean that you have to give up all control. You know, you only need to give away partial control to people you have grown to respect and understand and see that they actually have good, genuine ideas that are complementary to your own.

Some companies seem to think that they are the only people who can be technology leads. In some projects, they are the only people who can innovate around the software. They slap an open-source license on it and they make the source code available, but if you try to engage with the development process of some projects, it’s next to impossible. In my view that is not a true open source project. Legally it is–there is no question about that. But the development model is different from what I have grown to love about Free and Open Source software.

Scott: On the Apache projects, you have to have maintainers who are from three different companies, so that no one company can ultimately have control over the project.

Sun, on the other hand, has taken heat around OpenLDAP, and some other projects, where their governance is written so that Sun employees have to be at the top.

And so if people leave Sun, they lose that position on the project, whereas in a lot of other projects it doesn’t really matter what company you happen to be working for. It doesn’t really affect your stature on the open source project and in the community.

It seems to me that community driven open source is really difficult for companies that produce proprietary software, because the mindset is so completely opposite whet they’re used to. Companies like Adobe are experimenting with taking certain products and moving them to an open-source license, and Microsoft is dabbling with putting some things under an open-source license, but it seems like the really the hard thing for a company to get past is this notion of control.

A lot of times it’s also hard for companies to recognize that if they spend money and are good stewards of the project, they don’t really have to worry a lot about somebody forking the code and taking it off in a completely different direction.

That’s really only going to be an issue if they go off the rails like Netscape did, and end up with Firefox emerging out of the ashes. But that only happened because Netscape was messing it up so badly, and it was really obvious to some developers what would make a much better browser.

Anybody could fork My SQL, but who’s going to build a company of 200 people to do the level of engineering that MySQL AB – now Sub – is are doing around it?

It seems like the sticking point for a lot of companies as they play around with open source is this fear of losing control.

Ross: I think you hit the nail on the head. It’s interesting that you brought up Firefox emerging out of Netscape (although I wouldn’t say it emerged out of the ashes, it was a business decision to try and turn the tide in the “browser wars”). I recently re-read the first essay in Open Source 2.0 (a book published by O’Reilly), which talks about the transition from Netscape through to Firefox. It was written at the time all that was happening, and the essay ends at the point that Firefox 1.0 is released.

The article talks about the difficulty that the Netscape management had in releasing control to the newly formed “.org”, it is an enlightening read. I recommend that anybody who is trying to go through any of these processes should read that chapter.

I think you’re also right that well managed projects don’t really need to worry about forking. There is a huge barrier to making a strong fork. I mean, you can fork the code, and it is definitely possible that a competitor could come along with deep pockets and throw money at it. But they don’t have the knowledge and expertise of the staff that you have working on a project. That’s where the real value lies, in the project team.

This is one thing that worries me about many Sun projects. There was an open letter to the OpenDS community and Sun Microsystems in Nov 2007 from Neil Wilson that described how Sun had essentially kicked all the developers off of the OpenDS project as a result of corporate restructuring (http://directorymanager.wordpress.com/2007/11/28/an-open-letter-to-the-opends-community-and-to-sun-microsystems/). This got me thinking, “Well how can they do that? This is supposed to be an open project, it should be the community who decide when it is necessary to change the project leadership. Sure, Sun have every right to decide where their employees spend their paid time, but should they also have the right to dictate what they do in their own time on voluntary open source projects [Neil reports that he and the other developers continued to work on the project as volunteers]? Furthermore, the people being kicked off the team are the ones that have got the true knowledge about that software and, more importantly, the community around the software.”

Again, I should stress that Sun are not unique in this kind of behavior and perhaps more importantly Sun have some really fantastic community led projects out there. Like all large companies it must be difficult for Sun to have a consistent approach to everything you do. They have room to improve, in my opinion, but then other companies have much more room for improvement than Sun.

The important lesson, I believe, is that people are the valuable thing. It’s not the software–it’s the people who create the software that are valuable. That’s why you see people buying open-source companies. They’re not buying the software–it’s available through an open source license. They are buying the expertise.

Scott: Right–Sun didn’t buy the MySQL code base, because I could run off and start doing my own thing with that code base tomorrow, and if they aren’t good stewards of the project, they may have ended up spending $1 billion on nothing. But they would have to mess it up for a very long time before someone would sink the resources into a strong fork, a fork that the user base would have confidence in and follow. The barrier to doing a fork like that is so high.

I think you’re right about what company acquires when they buy an open source company. When you see JBOSS get acquired. You see XenSource get acquired. People will say Citrix bought Xen. No. They bought the company that had a business model and was able to staff people on the Xen code base, and were able to move it forward. They bought the mindshare.

Ross: That’s it, exactly.

Scott: It seems like if a Fortune 500 company is deciding what it wants to acquire, Linux and Apache aren’t really for sale. They don’t have one main corporate sponsor that you can buy.

On the other hand, it seems like it’s really hard to get to the point where you’ve got a project as big as, say, the Apache Web Server or the Linux Kernel. And so the half step, which is really getting to be the norm, is these open source projects that have a primary corporate sponsor.

The Fortune 500 company is then making the buying decision on pretty much the same factors as they would if they were acquiring a closed source software company. They want to know the annual revenue, profitability, growth, market share, etc. of the software company. The fact that the company being acquired is wrapped around an open source code base is almost incidental.

If you look at something like Canonical and Ubuntu, it’s a fantastic distribution. They have done a lot of things to take Debian and make it more usable. And they have done a lot of great work.

But I don’t think it’s a profitable company. In a 2007 interview with Forbes, Mark Shuttleworth basically said, “Look, I put $15 million into it. I don’t know if it will ever turn a profit. I’m just doing it because it’s the right thing.”

And I picture a CIO for a Fortune 500 looking at that and saying, “Well, do I want to bet a piece of my infrastructure on some guy’s philanthropy?” So, what are your thoughts around that type of issue?

Ross: Again, I think your observations are absolutely right. One of the things we do is to advise institutions who procure software about how they can actually evaluate open-source software. And the kinds of issues that you have raised there are things that get raised within our community.

Really, the way we advise people on that is to ask questions like, “How much am I going to invest in this? How much is it worth to my organization?” And then say, “OK. So how much am I going to invest in the ongoing development? How much am I going to contribute to the ongoing sustainability of that project? And am I willing to give that money to the project?”

Normally they don’t contribute cash. They tend to contribute through involvement by their IT departments. But it may be cash. It might be paying contractors to work on it. It doesn’t matter how you get that resource to the project. There are loads of different ways of doing it.

The key thing is you have a hand in making sure that that software becomes sustainable. And the question then becomes, “Is the balance sheet right at the end of the day or do I trust a closed source company more?”

Scott: That’s a great point, because I think that, in the same way that a closed source company has a lot of trouble getting over the hump of saying, “We can’t give up control of this,” the same thing happens with a lot of corporations that adopt open-source. They have trouble getting over the hurdle of, “This is something that I really have to participate in the care and feeding of.”

I think a lot of companies take the attitude toward open source that it’s free, and they will just take what’s there. And they hope they will be able to continue to take the new versions as they come out, without supporting the overall effort in any way.

Do you find that perception in the corporate world of, “Well we can sort of take what we want from open source, but we don’t really have an obligation to give back,” or are you finding more companies becoming aware of the need to participate in it they want to be able to count on it?

Ross: I think people are beginning to understand it more, but there is still a huge number of people who think that open source software means it’s free as in beer and that they approach it in order to save money.

It can often cost you less to engage with open-source software, but we always try to encourage people to look at their budget and decide how they’re going to spend it. If after getting your open source software and your training, ongoing support and maintenance and all that kind of thing, you’ve got a few thousand pounds or a few tens of thousands or a few hundreds of thousands pounds left in your budget, then invest it in that sustainability. The sustainability of the product you are consuming is also part of your own sustainability. With open source you can choose to take the responsibility (with staff effort or cash), or you can sit back and hope for the best.

I think that message is being heard more and more now. In the academic sector, it’s slightly different, because budgets are very tight, and they are always tight. There is no sort of thing like increasing your profits. You just want to pay less money, and the government is giving you less money, and so on.

So there is more of a tendency within the academic community to think, “Well, we can save money here.”

Sean: For full disclosure here, I taught at Purdue for a couple of years–a pretty university in the Midwest–and I know you have to save a buck everywhere you can.

I imagine it has got to be an interesting dynamic to make the discussion point resonate when you say, “I know it’s free, but for this you really should go for the full-on support package or you should partner with Red Hat. Is that a pretty big uphill climb?

Ross: It is hard. I mean people have got to save money where they can, and if open source is going to save them money, that’s great.

But institutions in the UK, at least, have been rolling out open-source quite a lot recently. Over the last four to five years, our national survey has been showing an increase in open source adoption.

What we are finding now is that some people have been running open-source for a reasonable amount of time. Because they have got IT departments, they have been customizing that open source in-house. So they have actually been giving back to the software in some way, they’ve been developing it. It’s just that, in many cases, they have not been giving it back to the community; they’ve kept it in house.

We then go to them and say, “Since you have expended resources on customizing software to your needs, why don’t you just spend a little bit extra effort and resources to improve the project? So you are giving back within the confines of the budget that you have available. And the payback for you is that your upgrade path is easier and therefore cheaper.”

I think that message is beginning to become accepted a bit more now as people struggle with upgrading. We’ve got over 60% deployments of Noodle, the virtual learning environment, here in the UK. And people are finally beginning to realize that if they are going to customize locally and they want to upgrade, they have to engage with the community.

A lovely convincing argument is that staff–particularly techie staff–feel that they are doing something rewarding. “I’m getting peer review, I’m getting recognized for my contribution, et cetera.” They are happier staff, and we all know happier staff stay around longer (so there’s another saving).

There are other ways that you can contribute back. It doesn’t have to be something that appears in your budget line. It could just be a side effect of the way you engage with the software. For example, simply making people aware of a successful roll out helps the project remain sustainable as it attracts more users and potential contributors.

Sean: Broadly speaking, what do you think is more unique about the open source software environment in the UK as opposed to somewhere else? We are in Oregon, which is a hotbed of open-source. The state and local government has a really passionate interest in it compared to some other areas in the country.

Ross: I get the impression that companies engaging with open source in the UK do engage with the software because it’s part of their core product and they recognize the fact that they have to engage with the software and make it sustainable. So that’s great.

But what’s not happening here, that you can see happening in other places, particularly in organizations like the Open Solutions Alliance in the States, is they are not cooperating across company borders. So they are not working together to make the software solutions that they provide to their customers play happily with solutions that are being provided by other companies. They failing to form consortia to provide end to end solutions that the big players can do in-house.

We do have, here in the UK, the Open Source Consortium, which has over 80 members, and if they read this and they hear me say that companies aren’t collaborating, they’ll probably argue this point next time I see them. But, that’s the way it looks to me. It might not be true–it might be that they are collaborating behind the scenes, but they’re not letting people know about it. They’re not doing it in an open and collaborative way like the Open Solutions Alliance in the States do.

It would be great to see that kind of thing happening more, and that’s perhaps a little bit of the British reserve. I think there is a cultural thing that might have some impact on the way we engage with one another commercially.

Having said that, there are similarities in the types of people who get involved with open source, and I think they transcend the national boundaries a bit. You’ve got to be a bit of an egotist. You’ve got to have a thick skin and all these other things.

So I think to a certain extent, we are kind of a nation unto ourselves, those who work with open source.

Sean: That’s a great observation. Is there anything you feel like adding in closing?

Ross: Well, I’d like to reiterate the point about open source not being just a license. Just slapping a license on code will not give you the full benefits of open source. And if you are going to engage with it, then you have really got to understand the open development model, and understand that open source governance requires you to let go of ownership (but not necessarily full control). Otherwise, you are just doing a slightly different version of closed source development.

I’d love to see something like the open source initiative for open development. I’d like to see it somehow define what open development is. Of course, it’s a lot more difficult than defining what a legal thing is, because it’s much more fluid, but that’s something we need to keep an eye on.

We’ve got to stop open source being diluted with all these other sorts of things that are just loosely grounded in the open source ethos. We need to really focus on the benefits of open development, particularly in the academic community, where we don’t have the same exploitation routes in mind.

Sean: Ross, thanks for chatting.

Ross: Thank you, it’s been great.

Bookmark this:

Interviewee: David Campbell

In this interview with David Campbell we talked to him about:

• How the development methodology of SQL Server has changed.
• How the changes affect the ability of customers to give feedback.
• How virtualization makes it easier for users to test-drive SQL Server.
• As products and open-source projects grow, the developers have to realize that they are not the target user.
• David’s belief that the best product doesn’t necessarily have the most knobs.
• Building a community around a closed-source product like SQL Server.

Scott Swigart: David, thanks for taking the time to chat with us again. One of the things you mentioned in the first interview was the common engineering criteria. I’ve seen that mentioned in a number of different places, but I have never seen it explained. And I’m guessing that the common engineering criteria impact a product like SQL Server. It might be good to start off just by explaining what they are.

David Campbell: It started in the Windows Server system. We’ve been telling customers that one of the advantages of our stack is that it works better together. It does, but we started really challenging ourselves by asking, “OK, what are the real things we’re doing to actually make it be better together?” The way I think about it is, “What are the things that would add value for customers who require coordination or consistency across groups that otherwise wouldn’t happen organically?”

We started this a number of years back, when Paul Flessner was running the Windows Server group. We have a set of requirements, which are updated every year, and all the enterprise products within the Server and Tools Business (STB) have to adhere to those requirements. These are things like having a best practice analyzer, having a Management Pack for System Center before you RTM, having consistent user education, continuous publishing, a community engagement program, that kind of stuff.

Scott: Previously when we talked, you also mentioned that SQL Server went through a pretty big change in its development methodology between 2005 and 2008. Talk about that a little bit.

David: This is something that a small group started, and it ultimately took great effort across the division to pull off. It will never be “done,” but it has been an amazing, fascinating transformation. I’ve learned more in the last few years about culture and change management in a large organization than probably anything else.

I’ve been working on the product for a long time. Back in the good old days, we had something like 20 people who knew the whole server, end to end. Back then, as you worked on things, if you had an issue you just walked across the hall, sketched something out on somebody’s whiteboard, and went back and coded it up. Because things were so intimate, you could keep things moving at a rapid pace.

To somebody who understands the standard Microsoft development methodology, one of the things that’s very important is the daily build. We continuously build and test the product, and one major vital sign is the health of the daily build. Back when the product was small and there were 50 to 100 people working on it, you could do the daily build live, with everyone just checking stuff in and keeping it together.

But as the team grew and the size of the code base grew by several orders of magnitude, you just couldn’t have as many hands in the soup at the same time and get anything that tasted good.

In SQL 2005, we had sort of a standard large-team, waterfall model. If we were doing a feature that spanned several component groups, we would have one group get together and they’d do everything by the book. We had a process and everyone followed it. They’d write the spec. They’d develop a test plan. They’d review the test plan. They’d write the code. They’d test it. They’d check it in.

Then the next component team in the sequence, maybe it was the client data access team, would pick it up and they’d start working with it. And they’d go, “Hey, wait a minute. This interface isn’t quite right. There is something wrong here.” They’d go back to the first team and ask them to change it. Of course, the first team had moved on to something else, so if they had to go back and do major surgery, it was a hassle.

We had one feature in particular that, to be honest, we built two or three times during the course of SQL 2005 and we still didn’t have it right. So I said to the program manager at one point, “Look, either we take it out of the product or you get everyone, end to end, involved, get them in a room and just sit till you work it out and get the thing going.”

This involved the storage engine, the language parser, the query processor, the client data access libraries, the management tools. They had done it serially several times. They just never had it work correctly, end to end, when integrated.

They all got in a room. Of course, you know how that works. They were able to figure it all out and get it all done. We said, “OK, what can we learn from this?” We completely threw the development process up in the air and changed it around. How do we go “back to the future” and try to capture the intimacy of what we had in the “good ol’ days” now that the team and product are many times larger and more complex.

When we had a small team rebuilding the database engine for SQL Server 7.0, a number of people had the whole system in their heads. Furthermore, when we did SQL Server 7.0, we really didn’t need to think about the market that much. We were building a new relational database engine and introducing an OLAP system. The playbooks were already written for these technologies and the markets were already established. Now, we did it in a way that was different; we tried to make the product much easier to use so we could actually grow the market and bring the technology to a much larger base. For SQL Server 2000 and, to a lesser extent, SQL Server 2005, we were building out on the architectural base we had established with SQL Server 7.0.

The original model worked great for several releases, but when we got to SQL Server 2005, the product was much bigger, the team was much bigger, and the market expectations were much higher. Things just weren’t fitting together in terms of consistency across the product. So the other piece that we had to add for the SQL Server 2008 process was, instead of defining the release from the bottom up in terms of what we were going to do, we had to create a definition process that looked both at the market, top-down, and at what needed to happen to the technology base of the product and what people wanted to do, bottom-up, and bring them together.

The way we framed it to the team was, “Look. We want to have the PowerPoint presentation that we’re going to use in the keynote for the launch before we write the first line of code.” So we developed a set of themes for the release. With each theme we had this notion of scenarios that were end-to-end value propositions for customers. Underneath those scenarios, we had a number of what we called improvements, instead of features.

An improvement is what we would consider an engineering transaction. The idea is that we bring virtual teams together to work on an improvement. An improvement team is cross-discipline—development, testing, and program management—and it’s cross-component. For example, the database engine, data access, and management tools teams can come together to work on a single improvement. So if a particular improvement needs focus from four different components, they’ll all come together as a virtual team.

We assign an improvement team lead, who’s got a set of requirements they need to meet to integrate the result into the source base, but we’re not very prescriptive about how they run the team. If they want to use Scrum and have stand-up meetings and do a series of sprints, that’s fine. If they want to use a test-driven development methodology, etc., that’s up to them. We’ve got tools and a playbook that we encourage them to use, but we’re pretty liberal in terms of how they go about it. There’s a very strict contract in terms of what needs to be done before it is integrated into the main branch of the product source code, however.

These improvement teams work in their own separate branches in the code management system. Once they believe that the improvement is ready to ship to customers, then and only then will we integrate it into the product. What’s interesting, if you think about it from outside SQL Server, from the customer’s perspective the release looks much, much different than it used to look.

It used to be the case that—let’s say we were going to do 300 features. Everyone gets started on them. We put a bunch of stuff together and put it out for Beta 1. Everything would work about 30 to 50 percent. Then we’d keep going for Beta 2. We realized, “We’re not going to be able to finish 20 percent of these things, let’s cut them.” With Beta 2, things are working about 70 to 80 percent correctly. Then you have Beta 3, a bit better. And in the end, you’re still trimming stuff out, cutting it, and refining it to get it ship ready for customers.

We were constantly in the mode of having stuff that kind of worked in Beta and trying to get feedback from customers. We flipped it around with the new process. What customers see now in a particular drop is very high quality. They used to see the 120 percent of the ultimate features in Beta 1 in some measure. Now they only see 20 or 30 percent of the features early, but they all work. We also changed the name from Beta to Community Tech Preview, or CTP.

As we go through each successive CTP, more and more of these improvements show up. They’re baked, end to end, and they accumulate in number. What a customer is seeing is that the quality is actually very good the whole way through, so it’s an interesting change.

Scott: It seems as if the SQL Server team went through the same process the Visual Studio team did: Instead of putting code out there that is 30 percent functional and people can’t really get it to work end to end, they decided it’s better to wait until a given feature is 100 percent even if that means customers see it much later.

But there has to be a trade-off, because your window for getting customer feedback is smaller, and you have potentially invested a good deal more time in the original code before you get any feedback. Talk about that balance.

David: That’s a great question. One aspect of the previous releases is that we were frankly playing catch-up in an established market. We knew what we wanted to build and we knew how we wanted to differentiate in the market so we really didn’t need a lot of customer feedback on the feature set for SQL 7.0 because in many ways we were rearchitecting and reimplementing the existing product. This approach worked for SQL Server 7.0 and 2000, but one consequence is that we hadn’t built up muscle around how to engage customers earlier in the design process for completely new features. In some sense, we built stuff and threw it against the wall to see if it would stick. This worked well for awhile, but it made much less sense starting in SQL Server 2005.

Basically, you could say that the old process was quite inefficient, in the sense that we built a bunch of stuff, actually committed code, before we even knew what would stick in the market. Now we’re trying to do that through more concept testing, more up-front work, spending more time with customers to look at their real pain points. Are you going to build a set of features that’s going to be compelling? That’s one piece of the puzzle. Are you building something all up that will be valuable to your customers to you customers end to end considering they way they really work? That’s another pieces of the puzzle.

But then the question becomes, have you built each feature in a way that makes sense to the customer and is useful? So it’s less about the value proposition of having a particular capability and more about have you designed and built each thing correctly, on an improvement-by-improvement basis.

One fallacy around the Beta process, in my opinion, is that we would throw a Beta out there, and tens or hundreds of thousands of people would download it. A small fraction would really play with it, a smaller fraction would actually think about how they’d use a new feature, and a fraction of those people would actually give us any real feedback. It wasn’t very structured. Our TAP program tried to target specific features and scenarios but it mostly kicked in after we had written the code.

I believe it’s a bit different in the Developer division, because you’re close to developers. One of the challenges we have with SQL Server is that, for our real validation, you need to get closer to a production environment. And it’s very difficult to get the feedback from the production team.

Scott: Right. With SQL Server, a single dev banging away isn’t a real test. You could get somebody banging away at the feature, but that doesn’t really tell you whether it will scale, whether it will be reliable, whether it will be secure.

David: Right, exactly. So what happened with this release was, we had some teams that said, “Oh, crap. What are we going to do to get feedback for this really complex improvement?” For the database engine there’s an improvement called a Resource Governor. It allows you to constrain queries that consume specific resources so you can limit a particular connection to 10 percent of the CPU, and that kind of stuff.

The question is the mechanics of getting the architecture and user interface right. We think very hard architecturally about separating mechanism and policy. In this case, the mechanism would be things like how do we keep track of the various resources and who do we attribute the resource usage to. The policy would be how the user describes her intent in a way that allows the mechanism to enforce it. In the case of the Resource Governor, there’s a complex interaction between the policy and the mechanism, – not the least of which is that we needed to build new mechanisms to hold and enforce various policies. This is a great example, because what we would have done before was stare at the whiteboard, do a design, and throw something together. We’d ship it out in a Beta and hear, “No, that’s not what we need.” So it turns into a rock fetch. “No, this thing stinks. … This thing stinks. … You’re getting warmer …,” etc. It’s a pretty inefficient process all up.

In the case of the Resource Governor, we got together with some of our MVPs and key customers who had a need for the improvement and did more of an outside-in design process starting from use cases. What’s interesting is that this outside-in approach is completely natural for many people building applications, but for a development team that’s been creating low-level system software for 10-plus years it’s quite different.

The bottom line is that we definitely needed to figure out how to engage customers differently with our new development process, and there are opportunities for design councils, concept testing, early prototypes, and early builds. In fact, things like virtual machine technology allow an improvement team to create a build of their early thinking and test it with a small group in a focused way without having to integrate everything else that’s in various states of completion, like we used to do with the standard Beta process. They just create a VHD with their bits and have some customers try it. With this model they can iterate faster than was possible with the old Beta process. Ultimately, I think we’ll get better improvement feedback by finding the right 10 to 20 customers and working closely with them rather than putting it together in Beta and throwing it out to 100,000 people hoping they’ll give us valid feedback. I’m overstating things a little bit to make the point, but the general concept still holds, although not everyone here agrees with me on this one.

Sean Campbell: I’m curious about the point you just brought up, because Scott and I have been knee-deep in virtualization for a long time. The point you just made about letting people evaluate it on a VM, how has that impacted the overall development process for the SQL Server team? Has it given you the ability to basically give, let’s say, broken bits out earlier and get more people working with it earlier because they don’t have to install it, they don’t have to configure it? I’m sure in the past people would try to install it on top of whatever else they already had. You could tell them a hundred times, right? And they’re going to try to install it as a multiple instance on top of what they’ve already got.

David: It has, in exactly the ways you said. These applications are so complex, and the state that they put on the machine is so complex, and we work so hard to make sure uninstall really uninstalls all the stuff that it put there in the first place.

The other piece is that it’s hard for us to build and test the setup and the installation. With VHDs, we can basically handcraft them and then clone them, and they don’t pollute any other state on an existing machine. So it’s much easier to get together. The bulk of the savings is in the setup, for us to construct both the setup and the uninstall, to make sure we don’t pollute the machine state.

So I think from that perspective, it’s very, very helpful. I think we saw that, but we haven’t really perfected it as a process yet. I think we’ll probably go a little bit deeper on it in the next release.

Scott: In talking to a lot of people in the open source world, especially community-driven open source, like the Linux kernel and the Apache Web Server, I hear there is a pretty strong correlation between your ability to request a feature and your ability to actually code it. You will get on a mailing list and say, “Hey, it’d be great if the Web server did this.” And the response is basically, “Yeah, that would be cool. When will you have the code ready?”

What you have talked about is very different from that, in terms of interviewing customers, interviewing influencers, and putting together a team. Talk a little bit about how a specific feature happened from end to end, how the feature was initially conceptualized, how you validated it, the portions of the product it touches, etc.

David: Well, I’ll talk about it in the meta form, and then I’ll talk about a particular feature. It goes back to the thing I was saying about technology evolution. And it’s actually one of the challenges.

One of the challenges the open source community faces, and this will be a sweeping statement that I get flamed for, is that they don’t yet generally appreciate the difference between the consumer and the producer. By that I mean, as a technology matures, the gap between who’s using and who’s producing it becomes greater. It’s a challenge we face at Microsoft; we are no longer simply building products for people like us, the engineers. In one of the talks I give to the product development community at Microsoft, I stress, “Look. We’re no longer just building products by engineers, for engineers. The PC is moving toward a consumer electronics device for many users and we must build a product that interacts with them on their terms, not ours.” I then show screen shots of Task Tray bubbles talking about “Virtual Memory” and “Pagefiles,” etc. My friends and family shouldn’t have to know what virtual memory is.

We’ve seen the same thing in the database space. Twenty years ago, every DBA had to be a wizard and needed to know how to fiddle with hundreds of knobs, figure out how to manage raw disks, and learn all sorts of low-level details just to create and maintain a database. When we were building SQL Server 7.0, I used to troll the database newsgroups an awful lot. You’d see people asking perfectly reasonable questions and you’d see responses like “RTFM” or “You don’t belong here if you don’t know that.” So we started this campaign to fix what I called the “you dummy” questions. Basically, if an expert responded to a question with a “you dummy” tone, we saw it as an opportunity to address the issue by engineering—teach the product to do it rather than teaching hundreds of thousands of DBAs. We turned the “you dummy” on ourselves and did something about it.

I don’t have to be an expert on my automobile to keep the thing running, and that’s fine. If Toyota wanted to come to me, I wouldn’t have to talk to them about ignition advance, dwell time, and those things anymore. I’d talk more about my experience and what I wanted the car to do for me. I think that’s just an evolution, and you can certainly see it in the database space.

To get back to the specific feature, I think the Resource Governor is a good one, where there was a complex set of mechanisms inside our system that we needed to go off and engineer, but how we constructed those and how we presented them to users was the challenge, right? What is the policy? How does someone describe a policy around constraining resources in a particular query or session? And then how does someone bind different policies to different classes of users or applications or queries or times? What degrees or what dimensions do they need?

Another way to think about this, from the perspective of design, is that the software development community has not made the leap yet from expressing the product control surface as knobs on the underlying mechanism to capturing the user’s intent or objective and then acting to achieve that. A simple example is: Rather than having an administrator configure how many dirty database pages will trigger a database checkpoint, we can capture the user’s intent in terms of the trade-off between fast recovery in the event of a failure, which can be achieved by increased checkpoint frequency, or better run time throughput, which can be achieved by fewer checkpoints. Having a control that captures how long the administrator is willing to wait for recovery to complete in the event of a restart looks directly at the business intent rather than fiddling at the level of the mechanism. I mean, how many dirty log blocks can I have on a particular database and still have it recover in 60 seconds, anyway? Instead of publishing some equation in the documentation, capture the user’s intent and treat that as a constraint during run time.

So to go back to the Resource Governor—we go to the customer and ask, “OK. What sorts of things do you want to constrain? What dimensions are most important? Are they applications? Are they time of day? Are they this or that?” And then gather all that, design something that’s consistent from the user intent perspective, and then figure out how we build a mechanism underneath to marry up with it.

Certainly, there’s a lot of success in the open source community, using open source technology in consumer electronics. But I don’t think that they’ve gone end to end in terms of doing a great job of user-centered design.

Sean: Well, that’s one of the things I’ve talked to people about, too, and that I personally feel pretty passionate about, innovation in the usability area.

I am curious, from a closed source company perspective, about what processes you feel should be put in place in order for a development team to excel in terms of usability. You go out and talk to people about potential features to help make sure that a feature doesn’t turn into engineers building it just for engineers, and that you actually end up with something—like the Ribbon in Office—that honestly has been a success, if only because, after release, nobody complained about it.

It’s the proverbial tree that fell in the forest and nobody heard it, but everybody was freaked out about it. And I keep telling people, “Well, think about this. Microsoft changed the UI on the one application people use the most, on a daily basis, and yet nobody really complained once Office launched.” There must have been something that really went into the thinking, in terms of usability.

For you guys, especially with a product like SQL Server, where I’m sure you can lose yourself in B-tree discussions until the end of time, how do you ensure usability in the product?

David: It’s a great question and a great challenge. It’s funny, but one of the things I’ve been doing for awhile that’s had a good effect is to hold the mirror up to the product development community at Microsoft, to get them to see that we’re no longer building products for people like us.

I have a set of slides that I’ve been using for six years or so, where I have collected some crazy error messages and I have real-life scenarios. I’ve got a screen shot where, in Windows XP, if it extends to your page file, you get this balloon coming out of the system tray that says, “Your virtual memory is low. We’re extending the page file. Blah blah blah.” I came home one night and my wife hit this and she said, “Hey, I think my computer’s broken. It says something here.” So I started describing to her virtual memory and page files, and she said, “Shut up! Just fix it.”

It dawned on me that perhaps 99 percent of the people using PCs these days are not engineers, yet we’re still throwing these exceptions and doing these sorts of things as if they were. And we ridicule the users instead of ourselves. I go help my neighbors fix their machines, and they all start with, “Oh, I feel like such an idiot around computers.” And I just want to say, “No, you’re not the idiot. We are, because we’re not building computers for you yet.”

And so I have a set of rules, or things that I’ve captured, that I talk about, such as the “principle of least astonishment,” where a reasonable user action should do something reasonable with least astonishment. The example I use is another story from my wife. I don’t know how many years ago, when I first got her a PC, someone emailed her some pictures. She said, “How do I look at these pictures?” I said, “Just double-click on the filename right here.”

So, of course, up comes the picture in the picture viewer. And there are little buttons down on the bottom of the viewer: “Next picture,” “Previous picture.” What do you think happens if you click on “Next picture”? You don’t get the next picture in the set of attachments. You get the next picture of whatever the next file was in your IE cache or whatever it happens to open.

And she’s going, “What the hell is this?” She gets some random icon from the temp directory or something like that. So I started talking to her about temporary directories, the IE cache, and blah blah blah, and again she’s like, “Shut up! Just tell me why it’s broken.”

Sean: She’s saying, “I’m looking at photos. I want the next photo.”

David: Exactly. And you see that even in the database space. We’ve taken the market from tens of thousands of servers out to millions or tens of millions of servers. You can’t have every one of them require a highly skilled DBA. We just couldn’t do it economically. And so you have to get out of the mind-frame of an engineer and go adopt a perspective of what the customer needs and how the customer wants to express their intent.

That’s the other sort of piece I talk about: In terms of the degrees of control and the dimensions of control, get rid of all of those dimensions of control that don’t capture any user intent and figure out which dimensions of control are orthogonal with respect to the customer’s intent, and then figure out how you build your system, and express things that way. That’s been the most effective thing I’ve found to motivate the change.

The bottom line is that the software development community doesn’t teach “design,” certainly not with a capital D. Most developers believe that if they think for 10 minutes before coding up some function, then that’s design. It’s not. This isn’t a development methodology thing, but rather a cultural change that we need to go through across the entire industry.

Sean: That’s great. The chief creator of a project called Quicksilver for the Mac was giving a talk at Google, and I just saw the video of it the other day. His theme for the project is “act without doing,” which has an interesting similarity to what you are talking about. If you saw the app, you would probably see some interesting things in it.

To follow along that line, I have a question about Oracle. Oracle, through the years, treated the product as though the more complex the cockpit, the better it was. If you walked in and it looked like an old 727 cockpit with 50,000 switches, that was nirvana. And you were paid to know every position of every switch, right?

David: Yep.

Sean: The open source community has, to some degree, a similar bent, right? Not in everything, but in a fair amount of places. But even in a product like Ubuntu, which is seen as the ultimate of end-user experience—and it has made some great strides—there’s a massive list of steps just to join that machine to a Windows domain, for example, if that is something you wanted to do.

David: Yep, yep.

Sean: So, do you see any similarities with this line of thought in the way Microsoft helps you to administer a database and the way open source helps you with administration in projects like MySQL?

David: That’s another good question. Yes, Oracle—I think I mentioned it earlier, they marketed our ease of use against us, frankly. They said, “How can this thing be a real enterprise database product? It doesn’t have nearly as many knobs as ours does.” And when we left Digital—this is an interesting little story—we had a product, RDB, that had roughly as many knobs as Oracle did at that point.

And like engineers, we were saying, “Oh, shoot! Hardly anyone knows how to turn the knobs on this thing.” So we wrote more software, something we called RDB Expert, which was a physical database design and some other things, to recommend and actually turn some of the knobs for you.

I think the first step that you go through in this evolution is to put a facade on it, whether it’s a system that recommends turning the knobs or a GUI to cover up knobs in config files. But really, at that point, you haven’t got a good end-to-end design, because you haven’t, in my opinion, eliminated those points of interaction or control. You haven’t captured user intent.

It’s like having a TV that still has horizontal and vertical control, but you’ve put something on the TV that can watch it and turn the knobs for you. So I think the next step in the evolution is to have the system software itself become more adaptive.

Scott: That’s true. TVs used to have horizontal and vertical knobs on the back. They don’t have those knobs anymore, even though TVs are much more complex, and no one misses them.

David: And what you wind up with—here’s the key point, and this is something I didn’t understand a priori, but after the fact it really made a lot of sense. In SQL Server 7, when we did some of the memory work to have the system automatically adapt to the environment and adjust the amount of memory it consumed, we changed the system so that you could adjust the amount of memory that it was using dynamically. And the mechanism was dynamic as well, so we had an automated policy that was a closed loop, but for those people who wanted to get under the covers, they could change it themselves.

But the neat thing about the way we reimplemented it top to bottom, and not just put a facade on it, was that the underlying mechanism itself was dynamic. By that I mean you could change the amount of memory that SQL Server used while it was running and not have to shut it down and restart it to get it to adopt a new value.

It’s back to the separation of mechanism and policy. We made the mechanism dynamic, we made the policy automated, and, where it made sense to try to capture user intent, we allowed user intent to be captured and actually influence the policy. And that takes architecture, top to bottom. I think the evolution is: You recognize it’s an issue, you put a facade on it, whether it’s a GUI or whether it’s some other code, and then you have to step back and architect the thing, top to bottom, to do it correctly.

Scott: One of the attitudes I ran into back in my C programming days was, “Look, programming was hard for me. It’s hard for a reason. It should be hard. If it isn’t hard, anybody could do it, and that’s the last thing in the world you’d want.” I run into that attitude whenever I look at systems administration and other technical issues, where people say, “Look, this is complicated. It should be hard. You should have to be a little bit of a rocket scientist, otherwise you have no business doing it.”

So one of the complaints is, “Well, yeah, Microsoft’s made it so that any idiot can set up a Web server, therefore every idiot does, and they don’t keep it patched, and it’s vulnerable.” Same thing with the database: “Sure, people could install the database and set it up, but they won’t create their indexes correctly, and they won’t keep it patched.”

So there are certain people who would say you should stop more people at the front door and not let them in. Because if you do let them in, they get farther down the path, thinking that they know what they’re doing, and then they run into a disaster. What would be your response to that kind of attitude?

David: I think it all comes down to, or goes back to, design in the sense that if you have the means within the technology to bring it out to a broader market and let people do the job, then I think it’s incumbent on you to try to figure out a way to do that. It’s a great example and a great point around physical design of the database—how do I design the index set? It’s very difficult.

One of the things we did here is, we designed the Database Tuning Advisor. This was work with Microsoft research where we’ll look at a workload and actually propose indexes based upon the workload, and do physical design that way.

And the way we test that is kind of interesting. We bring databases in, we measure their response against a real user workload. Then we strip all the indexes and we use the Tuning Advisor to rebuild the index set. More often than not, the Tuning Advisor does a better job than the guy who gets paid a lot of money to do it by hand. Of course we can’t see the reporting jobs that are running at the end of the month if they haven’t been part of the workload, but it does a pretty darn good job. So I think it’s a matter of how far can you take the technology.

Another sort of thought experiment here is if you go back to the TV front. Go back to 20 or 30 years ago. Every small town had two or three TV repairmen. The technology is more complex now than it was, but it’s perhaps 50 or 100 times more reliable. So you don’t see the TV repairman in every small town anymore. The TVs just go and go and go until you decide you want a slick new plasma one. I think that’s the way software is going to move as well.

Sean: Obviously, one of the strengths of open source is the community. I think that is one contest they tend to win hands-down, if for no other reason than that open-source projects were driving community long before closed source companies were. But at the same time, I know Microsoft has been making some significant efforts in this area too.

You mentioned in the previous conversation about CodePlex. I knew about CodePlex, but I have not really looked at it with an SQL Server focus before. A brief glance at it showed me some of the activity that is SQL Server related. Talk to me a little bit about what is happening there, how that activity is affecting the product team, etc.

David: I agree with you. I think I mentioned earlier that tapping into the energy of the community is super valuable. I think of a lot of these things in terms of closed loops and how quickly can you run the loop, how fast can you get the feedback, and how fast can you incorporate it. Tapping into the community is a super way to do that. There are lots of examples of closing a loop, like Watson, to have the machine send back error information and actually close the loop and automate things to enable continuous improvement.

As I think about community, absolutely I give the open source guys credit for tapping into that phenomenon first, but I don’t think it requires open source in terms of the development or distribution model to engage the community. I think about community engagement in terms of layers. With the product itself there’s a way of tapping into the community by harnessing their feedback to improve the product over time. I think with the documentation content and general knowledge around how to use the product, there’s a deeper way of tapping into the community. As we publish content for the product, can we allow people to modify it, link to it?

Some of our content could provide a spine for community activity. For example, we could build a collaborative site around the product error messages and allow the community to link their responses into each error message to help one another out when they hit a particular issue. Today much of this happens in forums and newsgroups, but it’s not done in a way that closes the loop effectively. If we did it in the way I just mentioned, it would be much easier for the product development team to review and mine the activity to improve the product going forward.

I think the next level beyond documentation would be extensions and tools. Things that you could build around the product where the core of the product is still in a closed source model but you could open up and distribute add-ons, add-ins, and other sorts of tools and actually build the community around that.

Sean: This has been a great conversation, David. Thanks for taking the time to chat.

David: Thank you.

Bookmark this:

Interviewee: Justin Erenkrantz

In this second part of a two part interview with Justin Erenkrantz we talked to him about:

• How the Apache project ensures good collaboration.
• The Apache Foundation’s philosphy of having no single person as the leader.
• Apache’s security committee.
• The process of removing someone from a position of responsibility within the Apache Foundation.
• What would make someone want to be part of the Apache Foundation’s group of projects.

Scott Swigart:  In this part of the interview, we wanted to dig into some of the tenets, if you would call it that, of the Apache way. And the first of those of course is collaborative software development.

So talk a little bit, if you would, about how Apache does collaborative software development. I’m sure some things are very traditional and similar to the way that other open source projects might do it, and there are probably things that also might just be a little bit unique to Apache. So how do you try to insure good collaboration?

Justin Erenkrantz:  So the main center point of all of the collaborative software development that we do in Apache is the mailing list. That’s where pretty much everything happens. As one of the guys mentioned before, the maxim has been: "If it didn’t happen on the mailing list, it didn’t happen." And that generally tends to be true.

Basically, if you follow the mailing list for a particular project, then we’re expecting that you should know what’s going on within the project. Within that, those are pretty much all public lists. Everybody can subscribe, even people who are committers, people who are just users, people who just work on another project that may consume the Web server or maybe PHP modules or something like that.

It’s pretty much an open forum. Anybody can voice their ideas. Generally though, just the way things work, most of the traffic tends to be from the core developers who are active at that time. The traffic patterns of the list change, and you get an idea of how much discussion. You generally see peaks and valleys for the mailing list discussion. Things get really heated or things are just chugging along and there’s not much traffic on the list.

So that’s where all the discussion should happen. And then of course there is the source code repository. And in Apache we always had the thought of having shared repository. A lot of projects now are starting to ‑‑ Git and Mercurial and all of these distributed version control systems to be centralized version control systems.

And that’s something that, within Apache, that goes against what our thoughts are, because we want to all be agreeing on, "This is the Apache version."

Scott:  Sure, no problem.

Justin:  So there’s no leader within the ASF. There’s no person, if you look at say, Linux and you say, "This is Linux’ tree," or this is Andrew’s tree or this is Alan’s tree. Instead there is just the Apache tree. So there’s really no concept of, "This is Justin’s tree," or someone else.

Scott:  Is the feeling then that essentially there should just be intensive discussion before something gets checked in? So when you’ve reached a point of consensus, it should get checked in rather than the way other projects work where different things get checked into different people’s trees. And then when it’s time to build a release, you have to pull stuff from these different sources to figure out what’s going to be in the release and what isn’t.

Justin:  Right. So, generally, what tends to happen is there are two states a tree can be in within Apache. One of them is "commit then review." And then there is "review then commit." And you’ll see some projects differ on particular trees.

For example for the HTTP Server, the trunk ‑‑ which is the main development ‑‑ is usually always under the "commit then review," which basically means that anybody who has commit access can feel free to go and make changes and they basically have the benefit of the doubt that the change is going to be good. And there’s generally an implied threshold: that if you’re going to make a really big change, go discuss it on the list. But if it’s a minor change or adding a little feature, that’s probably not going to be controversial, go ahead and commit that to trunk.

But for our stable releases, generally things that have already been released and we’re doing maintenance on those, are under the "review then commit" model. So that’s going to be RTC, and that means that any change to those trees has to be pre‑approved. That means you need to get three binding votes from other committers to say, "Yes, this is good change and no one has vetoed it." Technically you would use a file called STATUS, just a plain text file that some projects will use, that basically tracks all of the things that are under discussion to be back‑ported or added into this tree.

Scott:  Got you. Like any open source project, and this one is democratic, there is a vote to commit. Well, there is a vote if it is a release product. If it’s not a release product and people later decide it wasn’t a good thing to do, it can be reverted out.

And you mentioned before that part of your governance is that out of these 60‑something maintainers, one of them can essentially veto a change if they want to. So talk a little bit about how conflict resolution usually works. For example, when you have people who are—and I understand it doesn’t happen often—adamant one way and another person who’s adamant a different way. How do you see it play out that those eventually get resolved and things move forward?

Justin:  Generally what happens is, like before, the veto just tends to be a last resort. So let’s just say that someone makes a change to the trunk and I don’t like it. I might just say, "You know, we should talk about this change," or, "Here’s the problem with this." Generally, the person who has committed that says, "Oh, yeah. Here’s why I did it this way," and comes up with an explanation, and then through a process on the mailing list, figures out and resolves those conflicts.

That’s what you tend to see happen. And it’s all tends to be, for the most part, "I’m sorry, I forgot about that particular corner case or that." And everything tends to get resolved very naturally.

The veto tends to be when someone says, "No. I have to do it this way," and someone else says, "No, that’s wrong." That’s basically when the community is at risk of breaking down. But generally it doesn’t get to that point. Everybody is, "We’re all going to go in this direction; this is the right direction for us to go in. We want to add this feature. Let’s work through whatever issue you may have about this particular commit."

Scott:  One last thing before we move on to the next point. Other than the fact that everybody has their own private tree, is there anything else about the collaborative nature that you think is somewhat unique to Apache?

Justin:  We tend to do a roll call before the release. So at this point there’s been a review of everything. But then let’s just say that I want to release the next Apache 2.4 or whatever. Then basically what I will do as a release manager is say, "OK, I’m marking this as 2.4," and I produce all the artifacts. I produce the tarballs, [inaudible]generated files, whatever. And then I send it to the list and say, "Hey, is everybody happy with that?" And then that goes to a voting process.

There’s review at all stages, but there’s also a review at the point where you do a release, and you have to get, at least, three people to approve a release. One thing that’s different is that it’s not possible to veto a release. It’s strictly majority rule on the release.

Scott:  So in other words, there is veto capability on the individual check‑ins. But, as you said, when it comes down to doing a release it is a majority rule vote.

Justin:  Yes. You’ll tend to see someone voting ‘no’ on a release if it doesn’t work on Linux or something, and generally you’ll see it get stalled and it then gets fixed. But there have been a couple of cases when we did the release even though we knew that it didn’t work on a particular platform, and so we made a release note. But the veto does not apply to releases.

Scott:  Interesting. So moving on to licensing, one of the key things with Apache is the commercial‑friendly standard license. Talk a little bit about what that means.

Justin:  Basically within Apache, we like to have a big tent where everybody can come in and play with us. We think that the community that we developed within Apache is going to be the motivation for you to stay involved. For example, within the HTTP Server community, you have all these experts and Web servers, and if you’re part of this community, you get the benefits from them. And so there’s an incentive to play within the community, so that I don’t have to hire five guys and do a whole team; I can leverage the other people within the community.

But in turn, all those people who are part of the community say, "Whatever you want to do with the code is fine. We’re not going to get hung up if you make it a commercial product or an open source project. We created it and it served our needs, and if it serves your needs, that’s great."

Scott:  What springs to mind are things like GPL. Am I seeing it right in that Apache is more commercial‑friendly than GPL, V2 or V3?

Justin:  There are companies built around GPL licensed software. But what we tend to see are two classes of GPL products: In one, there is a real community, maybe within Linux, and they’re all happy to make all their changes available to everybody, and that’s a very good community. The other community you tend to see has a single stakeholder that has a prevailing interest in the GPL product, and they basically have an unfair share.

You can see this with some of the GPL projects that require copyright assignment. In order to participate, you license your changes in the GPL and you have to give a copyright assignment to the principal stakeholder. Now they are then free to release the commercial closed source based on your work because they have the copyright or whatever legal mechanism. There is an imbalance there when you look at those two.

Generally when you think about the GPL, you’re divided, with broad strokes, into those two groups. This is a real community but the other groups are aware and want to be clear about which one has a dominant role, and that’s one thing within Apache we don’t like to see. As our projects go through incubation and get added to the Foundation, one of the things we do is make sure that the community is diverse. In fact, there is not a single dominant stakeholder that can direct the project in any untoward way.

Scott:  Right. I can make changes to it, distribute it as part of a commercial product, and I would not be required to contribute those changes back to Apache. But under a GPL license, any modifications made require you to make the source code available. You cannot have closed source proprietary extensions or modifications of it. Any modifications you make, you have to open source and it has to be under the same license. So that’s the key differentiator?

Justin:  Yeah, our philosophy is that the community is what’s going to bring you and keep you there, and that’s why you’re going to stick around. If you released a commercial product around one of our projects it’s going to be to your benefit, to basically keep your commercial project as close to whatever we’re releasing.

Scott:  Right.

Justin:  You can pick up all the bug fixes and whatever improvements; you get those as a free rider. But in a sense, you are contributing whatever changes you’re making voluntarily back into the greater community.

Scott:  Yeah, that makes sense. Do you have examples of companies that have used different Apache projects because of the commercial‑friendly licensing, where they probably wouldn’t have it if the license weren’t so commercial‑friendly? Is that a topic that comes up?

Justin:  Absolutely. You see companies like IBM that release their versions of the Apache Web server or Geronimo under different names, but in the core, they are Apache projects. We’ll see that even with smaller companies such as Covalent that does commercial support. Basically, they added in a couple of extra things that provide support to their users.

One thing that the Apache community really does not focus on providing is 7/24 support. Covalent goes in with their business model and provides the support and training around these particular Apache projects. You will see businesses like JBoss using Tomcat. So you see all of these commercial companies using things that are Apache projects under the covers.

Scott:  Right, so that freedom has led people to be a lot more creative about how they structure their business. They have a lot more options in how they participate with the different Apache projects, how they contribute back and how they structure their own products. What is the relationship between the Apache Software Foundation and the Free Software Foundation? Is there any or are those fairly separate endeavors?

Justin:  There is no formal relationship. I’ve never had a conversation with Richard Stallman, but I’ve had conversations with Bradley Kuhn who used to be, at that time, the Executive Director of the Free Software Foundation. So there’s an informal get‑together of foundations to compare notes, and that’s generally a very good thing. How do we keep our ears open to what Mozilla’s doing? If they’re doing this new technique, then we can give them a call and ask, "What are you doing? We’d like to follow on it."

One thing we’ve been doing with the Eclipse is a joint conference. There’s going to be a conference in Asia that’s now scheduled for 2008. So it’s a way for us to get the communities talking to each other.

Scott:  Sure. So basically, if I can summarize, you guys get together around joint events and joint things where it makes sense. You share information because you’re all part of the open source community. Philosophically you may agree to disagree in terms of the details of licensing, commercial friendly, and that kind of stuff.

Justin:  Well and you ought to be using more…projects have different circumstances. Apache‑‑we have a very vocal membership and we have this and you compare that to let’s say the Mozilla which has a completely different governance structure. But if you look at Brian Behlendorf, he’s been on the Mozilla board for a very long time and he was one of the founders of Apache.

Scott:  Gotcha.

Justin:  So you have this intermingling of the communities. So someone like Brian Behlendorf who was brought in through the Mozilla and says here’s how we did things within Apache, and here’s his expertise and his experience that he got, he can share that with the other people within Mozilla.

Scott:  Gotcha, gotcha.

Sean:  Let me refer to one of the other tenants of the Apache way. I’m curious about this just because I was thinking about the conversation we were having. To state the obvious, you’re focused on producing software. I notice that you have a security committee that—if I’m reading it right and for lack of a better phrase—provides a service to all of the projects that are part of the foundation. And it looks like those projects can turn to the security committee and ask security related questions or possibly look for guidance from them, regardless of whether they’re Tomcat or some other piece of the foundation? Is that accurate or is that not accurate?

Justin:  It’s somewhat so. We have a security team, which I believe is currently a Board committee. But basically what they’re responsible for doing is ensuring our security at Apache.org mailing address gets responded to. And these are generally people who are very security savvy.

But there tends to be some people from Tomcat, from the HTTP Web server, from a higher profile project on this internal mailing list. So let’s say that, to give you an example, let’s say there was a security vulnerability in Derby and they could parse to those reps and say, "Hey, we have a security vulnerability. What do we do?" And so there’s expertise and, "OK. Here’s what you do. Here is your administrative contact. Make sure your mailing list… Go talk to…" Kind of a shared resource. But we’re not getting the focus on producing the fixes for the project but it’ll be "OK, here’s the responsible disclosure policy and an attribution policy." So that’s generally what their role is.

Sean:   Are they providing fairly prescriptive guidance but just not down to the ‘I’m going to change you’re code’ level because they don’t know the individual projects at that level? Would that group essentially be the center for discussions around a security development lifecycle for the Apache Software Foundation? And an attempt to pull those best practices together?

Justin:  Yeah. I think basically our project concern…we have something. What’s the process? What do we do? And that’s as an advisory role. OK, here’s the process and the procedures to follow.

Sean:  But it’s purely advisory, right? I mean one of the projects where they feel that their code is "secure enough", or they’ve looked at it long enough or they feel that they’ve handled it.  Then the advisory committee comes back and says, "Well, we really think you could take a look at this again." That’s where the communication would stop and it would be up to the individual project whether they want to take that under advisement or not.

Justin:  Yeah. I think so. I think record security, can you maybe at that point write back to your original reporter and say, "We looked at it and we don’t feel there’s a security vulnerability here." That may be…that has happened where we look at things and we say, "No. This is not an issue." But generally, really the security team is more of a reactive. So they’re not proactively performing security analysis on our code or anything like that.

Scott:  I just want to clarify. It sounds like they have a little bit of an all‑up policy for somebody sending an email to that address; somebody reports what they perceive as vulnerability or reports some kind of issue. They do a little air traffic control. They route it to the project.

Justin:  Exactly, exactly.

Scott:  There’s a general process that the different projects would follow. Basically that sort of happens and that’s one of the things that the security group advises the other projects on. Well this is generally a ‘way we do it’ sort of thing.

Sean:  Let’s go to a different piece of the Apache way, the emphasis on a technical‑based interaction. One of the things that I find fascinating about open‑source projects is the way that they exorcise community members that maybe aren’t following those rules.

[laughter]

Sean:  Because we got some interesting responses when we talked to people about it. It’s like, fine. We understand that everybody is an adult. We understand that everybody will try to handle themselves in an appropriate manner. But if anybody’s worked on a software project they know that not everybody does, right? So…

[laughter]

Sean:  Considering that you can learn a lot from a story…if you’ve got a story or two about either fully pulling the ejection handle on somebody that would be interesting to hear? Or a scenario where it just took serious counseling to get somebody pointed in the right direction.

I would be curious to see how you guys handle that. You obviously have procedures in place. But at times you have to go beyond those with some amount of intervention and I’m just curious how that played out.

Justin:  Yeah. So there’s one case that comes into mind but I’m trying to reserve the right to figure out how much of this has been disclosed.

Sean:  Yeah, Sure

Justin:  So I’ll tell the story and then leave people’s names out of it but I have to go back and see how much of this has been told. So recently within one of our larger, well‑known projects, there was a bout, to use the word, between two committers. And they basically ended up vetoing each other on everything. It’s like no, no, no and tempers got flared. And it got into a very unhealthy situation. They’re two very strong‑willed individuals.

And basically what happened is the PMC, so it was the PMC responsible for this particular activity, basically had to step in and say, "OK; we need to come up with some policy or come up with some new rules to get everybody back to ground zero. So it wasn’t a matter of ejecting anybody. It was never really an option that…basically what happened was they said, "Here are the ground rules. Here is…if you’re going to go do this you have to follow this set of rules. If you’re going to go do that, you’ve got to go follow this set of rules." Basically the community agreed to say we’re going to go and we’re going to voluntarily adopt these rules. But as a settlement process; lots of flames and a lot of innocent people getting…

Sean:  Right.

Justin:  …accused of things and that process. The other one that’s in our history was a project called Avalon. And this is one that’s definitely well known so this won’t be any issue about this one. Avalon was a container framework. And what happened was two individuals just did not get along. And they were ending up in what we would call a commit war, where they would basically be reverting each other’s changes as soon as they came in. And it is just this whole really poisonous environment and basically in that case, the community wasn’t able to deal with it. And so basically what happened there was they fractured.

And so that one of the individuals went off and he took his code and you know, we wished him luck and said have a nice life and he went off and then it was kind of some other people came along and they did a project called Excalibur, which was basically the remnants of this whole Avalon project. You will just see if you look at the mailing list traffic, you will just see this giant peak and then this sudden nothingness because the project got shut down because no one could play well with each other.

Sean:  Right.

Justin:  By the way what it is interesting is some of the veterans of Avalon, they really got involved in the greater Apache community, one of our new directors this year. He was in the middle of all of this, but during that whole experience, he was one of the people trying to keep things level and stuck around and this year he got elected to the board of directors.

Sean:  Since Apache is a large foundation, I’m curious about a different point. If you were a closed source company and you feel you are short on testers or short of security experts, you simply go to HR and put out a requisition and hopefully some good folks come back.

So has the foundation had to answer requests from projects where they say, "OK, look, we think we are geniuses on nine out of 10 of the things we need to do, but this one thing we really need people to help." How does the foundation help with staffing up a project in this type of case?

Justin:  It is more bottom‑up than that I think in the sense of the culture that we have. You will see some overlap between the HTTP Server committers and the Tomcat committers because you will see that ‑‑ sometimes the Tomcat committers came over and they say, "Hey, we need some help with HTTP." Well lucky us, we have some of the world’s foremost experts in HTTP server, and that basically got them within the communities.

So I think our community’s diverse enough to, "Hey I am looking for a person who knows SQL." OK, I am going to go on the Derby mailing list and say, "Hey, I need some help with SQL" or for something for build systems, I’ll go to Ant community. And there will be some of the people who are the foremost experts in that. That’s actually one thing of having such a large diverse community is that you can pretty much find someone who understands something about something somewhere within the Foundation.

Sean:   I figure that’s one of the advantages. You’ve got a massive talent base but at the same time it is segmented into the project, so the Foundation can help orient a little bit of that knowledge of where the talent base is.

Justin:  Yeah and as I said if you look out the social graph and it is a weird mix of people who are committers on Cocoon, maybe committers on Mina, then maybe on Gump and so you will see that developers themselves, the committers aren’t necessarily staying in their silo, there are some who do, but there are also just as many who will go to other communities and work on other projects.

Sean:   Well I have couple more questions but Scott can go ahead. I want to give you an opportunity to jump back in.

Scott:  So talk a little bit about standards because one of the other tenets or pieces of philosophy is faithful implementation of standards. Talk a little bit about what that means for Apache?

Justin:  Right. So this was initially when we started off there was HTTP and there was the IETF standards and that was when you have editor of the HTTP standard is one of the people behind the code base, there is the knowledge is going both ways in a sense is that we are that able to influence the standards process. But at the same point we also have some of us involved with the standard process and feeding those changes back and to the development and supporting those standards.

But since then, the initial, you see our participation within some of the key web server specifications, then probably most importantly our participation in the Java Community Process and that is, as you know, we have so many Java projects and that so many of our projects are implementing some JSR specification and our involvement within the JCP has been to ensure that we can implement the specifications and we have projects have representation on these expert groups that device these standards.

Scott:  Right. So it isn’t just like the standard shows up and then you figure out how to implement it. There is this two‑way street where you are shaping the standard because you guys have such a big, real world implementation. And meanwhile, the standard is telling what you guys do because they have their own stakeholders, but they are considering your recommendations and you want to conform to what they eventually approve.

Justin:  Correct.

Scott:  Yeah, I don’t know. I don’t have anything else specific. Sean, do you?

Sean:  No, not right now. I think this led us into some new stuff and from our end, we really enjoyed chatting about it. Justin, do you have anything you’d want to add or things you think we should address overall based on the theme of where we were going?

Justin:  No, I mean you did a good job of asking me the questions.

Scott:  I guess there is one final question. When somebody is starting or has an open source project, they can pick the license they want  they can do what they want for their community, things like that. What makes people want to be, in your mind, an Apache project? What is the draw I guess?

Justin:  Well, I think from my perspective, the draw is we handle a lot of the mundane governing structures and all this and the infrastructure and the licenses. And that is all essentially managed and I think you see a lot of open source projects like, "Oh we need to go get a foundation."

And that’s a lot of overhead, there is a lot of overhead to create a corporation, handle donations and handle essential infrastructure and that is what our goal at the Foundation at the broadest level is to provide support. So that these people who are working on all these different projects all they have to worry about is doing code. They don’t have to worry about, "Oh I need to go and buy a new server, how we are doing to deal with this donation or this tax policy." We try to deal with all of that. So I think there is a critical mass that works in our favor.

Scott:  Right, right and let them focus on the piece of it that they really enjoy, which is whatever this project is that they have come up with. They have a passion, like you said, for not having to worry about all the housekeeping stuff.

Bookmark this:

Interviewee: Britten Martin

In this interview with Britten Martin we talked to him about:

• The role of support inside Microsoft.
• A walkthrough of how a support request is handled.
• The ways in which the outside community can interact with the internal support processes.
• How a product is supported differently in beta than when it’s finally released.
• How a bug eventually become a fix in the product.

Scott:  Britten if you could just introduce yourself for the record. That is usually a good place to start, and then I’ve got an initial question after that.

Britten:  Cool. So, my name is Britten Martin, I am a group manager within CSS which is Customer Service and Support. I’ve been with Microsoft since 1999 where I came in as a support engineer supporting Windows 2000. So I have been a support engineer, I have done some work with our outsource partners in a partner technical lead role, then I moved into frontline management and did that for about three years. In my current role I am a manager of managers, I have an organization of about 75 people and I am the global sponsor for System Center Support, which is your SMS and MOM products.

Scott:  So talk a little bit about what falls under the scope of support inside of Microsoft. I mean, engineering rather has its domain, marketing has its domain: what is support defined as and what are kind of the responsibilities roll up under that inside Microsoft?

Britten:  That is a good question. It really depends on which segment of support. It is a great clarifying question, Scott, because I am within Commercial Support which is the enterprise-based support, your back‑end, your servers, many of your big customers that buy dedicated support contracts, the government, many of the major companies around the world. We also have certainly have a consumer end which is your home user running Vista, Office, also Xbox, Zune. Any product that you can think of that is released by Microsoft has support. So really what it is, the thing that separates, I think, Microsoft from other companies in that, is that no matter what’s out there, you can pick up the phone, call 1‑800‑MICROSOFT, tell them what your product is, and you’ll get through to someone to support your issue. Although it certainly depends on the customer segment that you are in, who you are routed to, where you go and how much it costs.

Scott:  Sure.

Britten:  In Commerical Support, we support customers who have technical how-to questions about the products and certainly those who are getting error messages.† Also, if you need help setting it up, if you need more consultative type services, we do that as well.

Scott:  OK. Then one other thing I was somewhat curious about too is ‑‑ one thing I have always been curious about I guess – is how a problem works its way through the cycle.

Britten:  OK.

:  So, you start out when the phone’s ringing and you have one customer, two customers, and three customers: they seem to be having a similar issue. How does it work its way through the process from the technician helping them on the phone, to maybe later on deciding it needs a KB on it, to maybe a hot fix, to maybe being rolled up in a Service Pack?

Britten:  That is a good question too. OK. I will speak in today’s terms because major things have shifted over the last couple of years. There used to be a lot of process behind just what you were saying, which is, you know: "When do we do a KB? How many issues before we make a hotfix?" Nowadays with the advent of blogs, which is kind of, what you are doing, you know, what this interview is about, in support we have become a lot more proactive with getting immediate information out to our customers. I will give you an example from my group, within System Center: last week we had an issue with SCCM, the new version of SMS. Just like you described: one call, two calls, three calls. All of a sudden we are like, "Wait a minute, there’s something going on here!"

So we got some information together, dumped it out into our blog so it was out there. On the back‑end our support engineers, who are your front‑line resources, are working with the escalation team, our escalation engineers who do the debugging, who look at source code, who work with the product group. They are debugging the issue, and they get it down to, "OK, this is clearly a bug." At that point, that escalation engineer has a hotline per se to the product group, they pass along all of their research, they are reproducing the steps, reproduce the issue, send it on to them, all the documentation.

Then it is kind of in the product group’s hands to decide, "Is this something that we need to fix now. Do we need a hotfix, is it just something that we can put documentation out." You know kind of that form of decision making process. In this case it was a major issue, it was a deployment blocker, huge customer pain, so they released a hotfix. Then once again, depending on the urgency, is it a publically available hotfix or is it something that you have to call support to get a hold of? In this case, we knew this was a major issue so we made it available out on Microsoft.com, put a KB article out there, and linked the hotfix to it. Therefore, customers could immediately just go and download that hotfix.

Down the road certainly that’s going to be in the Service Pack 1 or similar, so you can pretty much figure for all of our major products, any bugs that we fix, or any hotfixes we release up until, there’s a cut‑off date, will make it into the Service Pack.

Scott:  And then talk to me a little bit about the timeframe, right, between the time where maybe the first few calls came in, until the time that hotfix and that KB article are out there.

Britten:  Well, once again it is very, very dependent upon the criticality and the widespread nature of it.

In this case from first call to KB/hotfix was about five days. In addition, some of those delays ‑‑ they had identified the code, they had written a hotfix within about two days, but there were some delays, you know, going through legal on official KB review, and getting everything up there and replicated. So from beginning to end, about five days.

Scott:  Yeah, that is fast. I do not think most people would ever think of Microsoft turning something around that quickly.

Britten:  Some times, I think Microsoft is seen as very slow to respond, but in reality, itís often not the case.† †If it is truly a deployment blocking issue or something that causes widespread customer pain you will see a quick turnaround on it and you will definitely see the documentation out there in a timely manner. I have been here eight years, I have seen issues where it was tough to figure out what the fix was, what the resolution was. Where it was something that would hide itself into multiple components so it was a complicated fix to write, it takes a little longer. However, if we can identify it, if we do a good job on the support side with identifying it, providing reproduction steps to the product group, you know, they can turn it around quickly.

Scott:  So one question then too, would be, what does the triage process look like if after reproducing the bug you quickly realize that it has security implications versus something that’s a deployment blocker, which might be the next layer and then further down which are just kind of usability or general problems in using the software?

Britten:  Yeah, I think, once again when you are running into security type issues what you get with that is even more resources thrown at the problem. We have security aliases internally, they have product group members, and they have certainly a high-level support presence as well, that if something comes out we can fire it up and quickly. There †is a whole process that is enabled. Let us say there is a new Sasser or Blaster or any of those. We have processes in place that will quickly call together members of support, product group. We have open lines 24 hours a day. It is kind of the war room type mentality.

It has product group members on it. It has support members. So people on the field can call in and say, "Look, I have got an issue with this bug or with this security hole and just want to make sure I have the up to date information". So they can call into this grid, they can get the information.

On the back‑end, it is hard to say how long it is going to take, depending on the fix but you are going to have even more resources available. You will see a fix as quickly as we can get it out there. After the Sasser/Blaster/Nimda pain, we really turned the corner not only on product development but also on the support side to ensure we could have a quick turnaround.

:  What do you think the level of community involvement is in the support process because obviously that is a huge piece of the open source model. Right down from many eyes looking forsecurity bugs to a very well populated mailing list where there is a high discussion level, traffic, and things like that. However, at the same time with the closed source support process while the team is staffed potentially adequately with full time employees, there are general concerns out there with a closed source model in terms of how much the community can get involved.

So what are the avenues the community can actually affect the support
process at Microsoft and/or see what is happening under the covers of the support process?

Britten:  Yeah, I will tell you that it has really been something we have been working on for the last few years. The product groups and the support teams have been trying to find ways to tap into that community model because there are many folks out there that do our support day in and day out with customers that have a lot of knowledge that could be helping the broader community.

So, one of the things that we have is the MVP program, the Microsoft Most Valuable Professional. One of the things that these folks do is to actively take part in the public newsgroups for our products. I can point to a couple of products ‑ Small Business Server is the poster child for that. It has a very strong community. Customers can go up to the Small Business Server forums and post problems and the MVPs will quickly get involved to help resolve the issue.

There is a lot of energy around building a stronger MVP community and empowering the MVPs to help solve customer issues. Many of the product groups have CPE, Customer Partner Experience, teams, that are really focused on tapping into that partner community.

Another example from my world System Center Essentials, or SCE. We actually launched support for that on a complete forum based model, where we had engineers from CSS monitoring the forum, and we worked to build strong partner participation. If the partner could not answer the question, or did not answer the question within 24 hours, someone from support would jump in. We are trying to grow more community presence around support.

Globally, there is more partner presence out there as well. A good example is SoftGrid or Softriciy, a recent acquisition. In Europe, before Microsoft took over, there was not really a ton of dedicated support. It was all done through the community. Therefore, we have been looking and trying to learn from them on how they involve the partners, and how we can keep that type of interaction going. I know there is a log of energy around this and it is a
huge initiative.

Sean:  Well, one question which is really back to the second part of the original question which, is that a lot of the product groups in Microsoft are being affected by the open source model.† For example recently we talked to Shawn Burke who has been instrumental in getting the source code† for the framework out the door. How much of push to learn from the open source community is affecting the support team in general? For example could you see a day where the status on bug fixes is more open and less a black box until the actual fix arrives out on the web?

Britten:  So thinking more about ‑‑ I am partner, I am out there. I find a bug or a hole, I put it out there, and really being more transparent if that is going to be fixed. Is that kind of…?

Sean:  Yeah, exactly. I realize it’s not going to be a plate glass window ‑ it never is. But at the same time I think a lot of people will look at support right now and say, once I send in my support request I don’t really have a lot visibility of how they are stacking it, ranking it, prioritizing it, and where it goes, until maybe I get an email that says you have been deferred. Right?

So I am just curious, I know a lot of the product groups specifically are trying to expose a fair amount of their development processes, so I was curious if maybe that was making its way over to the support side.

Britten:  You know it is a fair point. Let’s say you are running a beta version of the product and you are involved in the one of the official beta programs, such as TAP ñ our Technology Adoption Program.† †The beta version is fully supported and you can actually go and directly file a bug. In addition, I know that they try to make a lot of that process transparent, so you know if the bug is a duplicate. Is it being fixed? Is it being deferred to the Service Pack 1…whatever. So actually, I know on some levels we do that.

On the support side, I think we make good use of our blog. There is a huge blogging effort, because there is a lot of knowledge within our support engineers and within our internal databases. For one reason or another that has never been made public. So I think we try to do some of that as well. But I can see the point. I mean maybe someday and maybe that some day is not too far away where you can log in and say, here is a bug that I submitted. Alternatively, even better for Windows, here is the bin of things that are going to be fixed in the next two weeks, months, whatever.

Sean:  Exactly

Scott:  Well, and so you know not every support request anymore comes in through a phone call, right? There are things like the Connect site. There are things like Windows Error Reporting services. I guess if you would not mind talk for a minute about some of the different channels that, a problem can kind of come in through and… Go ahead.

Britten:  No, no, it is fine. So today, and if I use Americas Commerical Support as a kind of a measuring stick, somewhere around 85% of our support issues come in via the phone ‑ very traditional means of operation. We are really trying to push more of our online type offerings. One is online submission where you can go up and create your issue; you can attach data, submit it to a support engineer. That support engineer reviews that data and calls you back, hopefully with an answer.

The Connect site is a lot more beta focused at this point. I am not an expert there, but that’s where you get into many of the cool things where you can actually submit bugs and all that. We also have the managed forums, which we talked a little bit about. The managed forums are a completely different avenue. Most of those are staffed by Microsoft support engineers as well as product group representation and the partner community.

So you have an issue with Windows, you can go up to a specific Windows forum, post a question. You might get an answer fromsomeone in the product group, a partner, or a support engineer. Therefore, you have those as well.

On the consumer side, they do really cool things with automated responses. Therefore, you submit an issue and it checks your data, checks your error log to see if it is a known issue. Before you would hear back from the support engineer, you might get an automated response with, you know, try these two things. If this does not work respond back and the engineer will contact you, then as you said, you had the whole Watson or error reporting. The error reporting stuff is very cool, because regardless of bugs filed or anything else that goes on, all the product groups take all of their Watson or their error reporting data and they make sure that as they look to roll out service packs, with new versions of products especially, they want to handle those top 15, top 20, top 50 Watson issues. They want to make sure that they are knocking those off.

Therefore, it is an interesting mix of many different avenues for customers to submit data, and in some cases, a Watson log, the error reporting, customers might not always think they are really contributing to the next version of the product, when in essence, they truly are.

:   I realize, too, that you are right, that is largely a mechanism for when a product is in Beta. I guess, I have two questions, but one is a follow up on this thread initially. Talk a little bit, I guess; about how a product is supported differently while it is in beta, versus once, it has been released?

Britten:  That is a good question. We have a few different types of beta opportunities for customers.† There are more formal relationships, such as TAP (Technology Adoption Program) and RDP (Rapid Deployment Program).† With these formal programs, customers agree to deploy this product into some type of production environment. In return, they get access to submit bugs, they get access to product group members, and they get
access to dedicated support engineers.

From within support, let us say we have a new version of Windows, we might have three or four support engineers in the US dedicated to this beta support. These engineers get a hold of the product early, they get access to product specs, they visit the product group, and they may even get to go out on site to customers who are deploying this product so they get some real world hands on experience.

Now, that is for the structured beta program. If you’re not a customer in one of these programs, you likely can still download beta copies of the software. Even in these situations, there is support out there, through the forums that we talked about before. Once again, you probably get answers from those same beta engineers and some of the product group folks. You just do not have that one-to-one interaction in a lot more, you know.

Does that make sense?

Scott:  Yeah. Yes, very much so. One of the other things that ‑ I will kind of circle back to something you said earlier – If it needs code to solve the problem, you know, if it is not just configuration file that was messed up or any one of the myriad of things that could cause a problem. You know, some registry key got corrupted or whatever. However, it actually needs the coding hot fix, you could hand it over to the engineering, send it over to the product team and they triage it, compared to other stuff.

I mean, have you ever run into situations it seems like this would inevitably happen where they’ve got their engineers dedicated to building the next version of the product, and meanwhile, there’s an endless list of bugs in any complex software that they could be tasking engineering resources with. In addition, it seems that they kind of have to make that call about pulling people off engineering new features to work on a hotfix or something like that.

Britten:  Yeah.

Scott:  And how does that process of triage work? Because support is kind of on the front line with the customers, it seems like sometimes you guys might have to go back to them and say, "Um, you know the time frame that you’re looking at fixing this in probably isn’t really going to work, so think about it again."

Britten:  Right. In addition, I will tell you, it is really dependent upon the product group. You look at something like Windows. Windows has a whole team, and you hear it from time to time called Win SE. ,Windows Sustained Engineering. They have a whole team dedicated to current version, and whatever supported versions are out there, on the hotfix front.

With some of the smaller products, you may run into situations where they have to pull developer resources off the next version to fix current version or legacy version problems. However, I tell you, in my time here, I have never seen that as being a blocker in getting something fixed. If there is truly an issue that needs to be resolved in the current version of the product, especially if its security related, you can be assured that they will find someone to handle it.

However, as a general rule of thumb, all big product groups have sustained engineering, folks that are dedicated to current version problems, or legacy version, so it is not necessarily that huge resource problem that you would think.

Scott:  OK. Therefore, many times, they are not pulling people off features. They have people who are dedicated just to the maintenance and hot fixes.

Britten:  Right. That is your sustained engineering folks, for the most part. As I said, I think you would run into some smaller product that might not be in that same boat, but the major ones have sustained engineering.

Scott:  It seems, too, like support, it is a little bit of a difficult situation, because, I guarantee, if I call with any bug, it is a critical show‑stopper. The world will end if it does not get fixed. So how do you kind of manage customers, who are generally going to feel the importance of their bug is high, but Microsoft does not necessarily determine it so in the grand scheme of things?

Britten:  Yeah. I tell you, our people are experts at conflict management and really setting customer expectations, and certainly the empathy piece as well. I mean, you have to be empathetic with the situation, but you also need to be very transparent and upfront.. It is saying, "Look, here’s where the bug is at this point. It is something that may or may not get fixed. Here’s the timeline."

With many of our customers in the Commercial space, these customers also have account managers that are Microsoft employees that really serve as the liaison between support and the customer. Therefore, we do a lot of work with the account managers to help us on the customer maintenance side of things, when maybe the messaging isn’t as good as we’d like it to be.

However, frankly, all major problems end up getting fixed. They might not get fixed in the period the customer would like to see it, but, between a hotfix, the next service pack, and the next version of the product, all issues do end up getting resolved.

Once again, it is really just dependent upon how many customers are having the same problem. Is it something that other customers could run into? In addition, really prioritizing, like you said before. I mean, the product group, while they have a sustained engineering team, has a limited number of resources, and they need to prioritize what are the big-ticket items that need to be worked on.

However, that skill of conflict management is one of the key pieces for successful support engineers, because, every day, when we answer the phone, when we respond to an email, that is generally that networking administratorís worst day of the year. The servers are down. They are catching heat from their management.† Something is going on, they are being yelled at. Our engineers deal with this on a day in and day out basis and as a result, they get very good at handling the heat.

:  So one question I am curious about is, how do you deal with it when the support engineer knows that it is really a usability issue and not a bug, sure there is a sequence of phone calls and bug reports, but it is pretty clear that this isn’t a bug, it is more of an issue of usability. And it might be in the land of the undiscovered feature right? We were told, if I remember, some person in the office team told me that something like 90 percent of the feature requests for office, are already in the product, but nobody knows the feature is in there. So how do you people educate the product teams and what does that flow look like? How do you finally classify that, "OK this isn’t really a bug, it is an issue of usability?"

Britten:  That is a good question.

Sean:  Because obviously the product team, like everybody else is blind to what they build, right? Of course, it is useable. Because I built it, I mean every developer has fallen to that problem and we all have seen some pretty horrible UIs over the years. So how does the support team educate the product teams on this point?

Britten:  Yeah. In the past, if you back up five years, 10 years, support was as you said an island. We were out here, taking all the heat and there was limited interaction with the product group. About five years ago, things drastically changed.† we started doing what is called supportability in our world. We review what are the top issues, how long does it take us to solve those issues and how much do they cost the company for us to support these issues?

Therefore, we came up with a list of the top 10 issues. We then present that back to the product group. Now the interesting thing about that is the product group actually funds us to do support for their product. So, when they see that a problem in their coding is †costing them $800,000. they to take some notice , and say wow OK, what does right look like, what are we missing here?

Sean:  Well, just a quick follow up to that. What are some examples of things where you feel like you touched the product, and you made a real delta that the customers ended up bettering from, but at the same time you folks probably would never personally got credit for it

Britten:  That is right. I donít have a list in front of me but DNS is one that consistently scored high for Windows 2000 and even some into Windows Server 2003.In addition, there was so much push and so much interaction that there were some major changes that happened later in 2003 and into 2008 that have drastically changed the way DNS works.

Another one I can give you that clearly had all the supportability people driving this was terminal server licensing. Terminal server licensing, when you go back to 2000 and once again, kind of into 2003 with a complete and utter nightmare for something that should be very simple. Consistently that silly process ended up top two, top three on our supportability list for Windows Server. And some very small tweaks based on advice provided by support completely changed this, dropped out of the top 10, out of the top 20 in support issues and saved the company a lot of money.

Those are two examples of technologies I know we beat on. However, on all the products out there, SQL, Exchange, Windows, SMS, MOM, they had the supportability groups that review the top 10 issues. In fact, I was just in Redmond last week, looking for the SMS side and they do cool things with it. I mean they look at top 10 issues for previous version of the product, look at the top 10 issues for current version, and review whether or not they fixed the top 10 call generateors? If not, what do we need to do to make sure it is in the next version that we do? Support is a huge data mine that we are just starting to realize the power of all the data that we have.† We are a great voice of the customer.

Sean:  One quick follow up to that and then Iíll turn it back to Scott. I am curious because you mentioned that five years ago there was some type of a change, right? In addition, all of a sudden, you folks were a lot more integrated. What drove that? In addition, I realize with all changes ongoing, there is probably stuff you can say and stuff you cannot, but I am sure somebody reading the interview is going to be like, "Wait, wait, and ask him that." So what drove it at a high level?

Britten:  I think that is fair. I wish I had some great and gory backroom drama to share, but if it happened, I just donít know the details.† In my opinion, you know, part of it was just the fact that, wow, especially on the Enterprise side, Windows Server 2000 started to really turn the corner and become a huge enterprise related product. People were out there buying it and implementing it and someone on the CSS side as well as, I can guarantee you, the Windows product group was like "We need to tap into this.î

Around this time, I was a support engineer and I suddenly realized that the product group was really listening.† In all businesses you started hearing about some monthly conference calls where we were there to provide feedback on top issues.

It was a time when we were moving from being just a consumer, home user company to being a legitimate player in the enterprise. In addition, there was a lot of pressure at that point to make sure these products were just better, and I think someone realized "Wow, CSS has a lot of knowledge out there; we need to tap into it."

Scott:  Well, how much now do you guys get involved in product development from the standpoint of, you guys are looking at the product, while it is in development and saying, "You know what, if that error happens, there is no way we will able to trace it back and figure that out. We need instrumentation here, we are going to need these things, spit it out to a log file, we are going to need these kind of support features built into the product to be able to kind of quickly
and officially troubleshoot things."

Britten:  That is a huge piece of our beta involvement. I know we talked about it before. For each beta, we dedicate engineers from our support teams to the pre-released product. Well, one of the way they make an impact is by providing that ongoing feedback to the product group. They are not only working with beta customers, they are not only answering newsgroups, but they are also messing with the product itself.

Therefore, they filed their own bugs. They have supportability war room meetings where they say, look there needs to be some logging here, or this error message does not make sense, or if this component is blowing up, I need something to trace it back. That is generally our involvement.

We do spec review. So many of our engineers have been on betas before, they are hooked in with the product group contacts. As the next version of the product is being planned, our engineers get involved to review specs, and just generally throughout the beta process from the beginning of our involvement through RTM, we are providing that ongoing supportability feedback.

So not only are we always supporting our customers, but we also look to help the product group on the supportability side. It is like you said, I mean, they are focused on here is a new component, here is a new thing that customers can use, maybe not always focused on how would I support that if it breaks?† Letís be honest Ö if we donít provide our feedback to the product group on what we need, it just causes more pain for us once the product is released.

Scott:  One other one question is, so a new product is coming out and how do you get support folks up to speed on it? In the OpenSource model the development team in some cases is the support team and there is a tight integration between that team and the open source community so there isn’t really a need to educate the "support team" in this case. So how do the support engineers come up to speed on a given product?

Britten:  This is another benefit of our involvement on the beta side.† †We get access to the source code. We get engineers playing with the product. We get that constant interaction with new builds of the product as it comes out. As a result, months, in some cases years, before the product actually ships, we have experts on this product.

Now, our experts certainly might be great technical writers, but we have a group called Global Technical Readiness. These folks are really program project managers who will say, "All right, we need Vista training for our support engineers." Therefore, they get their list of top issues. They get a list of new components, features, all of that. In addition, they start mining for data using these beta resources to come up with, "Here’s what engineers need to know. Here are the troubleshooting pieces; here are the top issues we have seen with our beta customers. Make sure engineers know this and that."

Therefore, over the course of this beta cycle, we are also in the background writing our own training. So these engineers write the training and as RTM approaches, depending on the training strategy, they might train all their engineers or a subset of the engineers. However, the knowledge is based off spec; it is based off beta engineer experiences, and beta engineer customer interaction.

In addition, the product group, in recent years, has really stepped up to help with training. They will review our training. They will provide insights. They will attend beta deliveries of the training and help the trainer get comfortable with the material and the product.

In some cases, the product group will also do a session of live meetings on, "Hey, here’s how we intended you to support the product. Here are the supportability features that we thought would be helpful." Therefore, you get a lot of interaction with the product group as well. From the beginning of the product to the end, not only are we helping customers on the beta side, but also our selfish win is to get the training that we need.

Scott:  So, let me ask the flip side of that, right? So, you mentioned Configuration Manager. You’ve got System Center Configuration Manager 2007 coming out. Everybody wants to learn it…

Britten:  Sure.

Scott:  …because everybody is motivated. Have do you rather incent people to keep wanting to support the previous version when there is something new, shiny, and interesting.

Britten:  At least on the Commercial stage, I do not know that I have ever really run into that problem.

Scott:  OK.

Britten:  Just simply because, in the enterprise customer segment, what you generally see is not the consumer model where a product releases, and everybody goes and gets the new version. However, it is a gradual uptake, usually. So, say Configuration Manager. Right now, and I can give you the specifics on this one, it is 10% of our total SMS volume. Therefore, engineers get a couple of SCCM cases, but they get the majority of SMS 2003.

In our minds, that is beautiful. Because what happens is, slowly over time they can build their SCCM knowledge while still providing that long held knowledge on SMS. They can help those customers resolve their issues. Therefore, it is a slow mix that generally happens. I do not know, it is ‑‑ they are tech junkies. They love the new stuff, but I guess maybe we all know that we have to pay our dues with the old stuff as well. [laughing] I do not know.

Scott:  OK. That makes sense.

Britten:  That’s a good question but it never really happens quite that way on the Enterprise side of support.

Sean:  What it sounds like for you folks, is that from an enterprise standpoint, you can point to pretty clear statistics on customers who are significant accounts. Alternatively, you could point to customers who, for lack of a better phrase, probably spend a fair amount of revenue, right. In addition, it is probably easy to also look at the segment of the market and say, "We’re not going to give up 15% of SMS just because SMS, the latest version is out" ‑‑or in this case, obviously the rebranded version.

Britten:  Right.

Sean:  Because you folks can point to real metrics in terms of your percentage of the number of people who are putting in support requests and things like that.

Britten:  Right.

Sean: Thanks for taking the time to talk with us Britten.

Britten: No problem.

Bookmark this:

Interviewee: Robert Bray

In this interview with Bob, Architect for Geospatial Products at Autodesk, we asked him about:

• Introduction of open source within Autodesk
• Role of project steering committees
• Decision to create an open source product
• Open source impact on closed source product
• Developer and engineer adjustments to open source process
• Differences between feature design in open and closed source
• Test and documentation for Autodesk open source
• Community contribution to open source projects
• Business reasons for creating open source products

Sean: Bob, why don’t you give us some background on your role with Autodesk.

Robert: I’ve been in software development working for independent software vendors for about 17 years now. My current title is Architect for Geospatial Products. So, I’ve gone through the various ranks, from being a programmer in the trenches, in the cubes, to managing a product development team, to my current role as architect.

So, as I said, I’ve been through various things, mainly in closed source. Until roughly 18 months ago when we open sourced MapGuide. I think it was about six months before that that we really started heavily investigating that idea and started to put the ball in motion to actually make that happen. Basically, I’ve been involved in an open source project very actively for the last 18 months, and I won’t say that I was the key driver that made the decision that we were going to open source this, but I basically took it from that decision point and said, “OK, these are all the things we need to do…” and made that happen.

The product itself is MapGuide.The open source version is called MapGuide Open Source. The closed version is Autodesk MapGuide Enterprise. The codebases are the same. There’s one open source code stream for the whole product. There are a couple of minor functional differences between Enterprise and open source, but they are extremely minor, and we are trying to close as many of those as we can, actually. What we really have with the Enterprise version is basically a commercially tested and supported version. It’s much like the Red Hat model or the MySQL model, in that respect.

Sean: So, just out of curiosity then, if there are some functional differences, what led to those? I mean, they’re minor, like you said, but was some of that just by the nature of having to switch from closed source to open source? Is there some functionality that just couldn’t make it over that transform right away?

Robert: Basically, that is exactly it. We have a commercial desktop mapping product called AutoCAD Map 3D. We used some technology from that in the Enterprise version of MapGuide, and we do that for compatibility between the two products. So, those particular pieces, our coordinate system library, in particular, and our raster image handling library; those are things that we just could not open source at the time.

Sean:So, when Autodesk made the switch, did you model against an existing open source project in terms of how you wanted to handle QA or how you wanted to establish a community feel? You look at different products, like Subversion – I’ve heard they had one vote in the entire history of the project, right?

Robert: [laughs]

Sean: Literally. I watched their presentation at OSCON, and at least that’s the public face of things – that they had one vote. Or, for example, we talked to PostgresSQL yesterday, and they said, “It’s a very communal democracy” in terms of their decision process. They don’t really have a hierarchy. I know it varies across the projects, so did you look at the open source community and say, “Well, now that we’re going to open source this, we’d like to mimic this particular project” at least in some aspect?

Robert: Absolutely. So, when we made this decision, one of the first things I did was take a broad look across all of the projects. I was mainly focused on looking at the geospatial projects, because, if you will, our kin in the community is all of the open source projects – things like MapServer, GDAL, PostGIS… All those open source projects are very close in nature to us, so I certainly did look at those.

I also read a couple of books. The one I liked best was Karl Fogel’s book, “Producing Free Open Source Software.” Of course, he led the Subversion effort, but if you read that book you wouldn’t know there was only one vote in Subversion.

Sean: [laughs]

Robert: [laughs]

Sean: I was a little surprised, too. But, these were two guys from Google, so technically they’re reinforcing each other in their viewpoint, at least. It’s not just one guy saying it. They’re committers on the project…

Robert: I’m shocked.

Sean: Yeah, yeah. I was a little bit, too. But that’s part of the fun part of this investigation is some of the stuff we hear, and then we get to cycle back and say, “Really? So tell us a little more about that.”

Robert: So, we definitely took the approach of forming a community, and the project, and I say not from day one, but within six months, became a community driven effort. MapGuide has always been a product that, it’s almost like there’s a religious following of users out there. Even when it was a commercial product, it had some real hardcore, dedicated users.

So a handful of those users said, “Wow! This is going to be great!” They knew nothing about open source at the time either, but they kind of all jumped on board right away, started testing the open source version, and started getting active in that.

So, when we formed the community, basically, because we were modeling after MapServer and some of the other open source projects on the geospatial side and I really felt that was important we basically created a project steering committee for the product out of that group of people. So that project steering committee, right now, consists of three people from Autodesk and four people from outside in the community.

Sean: Are those people then the chief committers on the project?

Robert: Actually, that’s an interesting thing, because I would say there are a couple of people there that are active committers; the rest of them are users.

Sean: Got you. Because the interesting thing to me about this is that, obviously, a lot of these projects have grown up over time. PostgresSQL has a 20 year history, and to become a committer on their project, they told us, “Well, we wait two to three years to figure out if we can make you a committer, because we’ve been around for 20 years.”

But, if you birth a project out of the ether how do you determine who’s going to be a committer? Because, obviously, there’s probably a stampede of people at the start that say, “I’d like to be a committer, I want to be involved at the highest levels,” that kind of thing. How do you go through that winnowing out process?

Robert: There’s a URL I can send you, (http://mapguide.osgeo.org/psc.html) but basically what I did was draft a set of rules for the project steering committee to follow that include who can be a committer and how they become a committer.

We basically go through a similar process to, I think, Postgres – although we probably wouldn’t be two years in the road doing it. Basically, people start submitting some code from the open source community. They submit it to a developer, we’ll have that developer review it, and then do the actual commit. It’s almost in a patch style, right? Then, once we’re comfortable with them, we’ll vote to make them a committer – it’s a project steering committee vote.

Sean: Got you.

Robert: So, it’s kind of interesting, because, again, the project steering committee is not the core developers; again, it’s a core set of mainly users.

Sean: Once you transitioned over and said, “I want to make this an open source product, ” since you came mainly from a closed source background, what was the thing that most pleased you about being able to transition into an open source development model? What was the thing that you were, I guess, either most afraid of or most reluctant to see transition over? Because, obviously, the models are pretty different.

Robert: I think I was scared to death for quite a while…

Sean: [laughs]

Robert: …because it was completely unfamiliar territory to me. I’ll be honest, right? I’d never worked with an open source project before. I’ve used a few, but I’ve never really participated in an open source project. So, from my perspective, open source projects were these big free for all things, and I’m like, “Oh, my God. What are we getting into here?”

But, when I stepped back and actually looked at it, what I realized is that open source projects…yeah, they run a little differently than commercial projects, but they follow a fairly well-defined process; or at least the large projects do, that are successful. You can see a well established process there. It’s not a free for all, anybody can commit code, and a thing like that. So, that was my biggest fear going in; it was the step into the unknown, right?

Sean: What do you think was the biggest lesson learned from that? Now that you’ve got a couple of years at it, what would be the thing you might’ve done a little bit differently, given an opportunity? It’s apparent you wouldn’t have changed any decision to go open source, and that rings through clear. But, is there anything you would’ve changed during the process of going in that direction?

Robert: If anything, I would’ve been more open with the community right from the get go. I think I had a lot of reservations going into it, and so I didn’t really communicate as much as I should’ve right up front about what we were doing, how we were going about it, and all of that. So, I think I would’ve been much more open around communication right from the get go.

Sean: OK…

Robert: That’s the biggest lesson I think I learned.

Sean: …then what was kind of the biggest positive that you got? I didn’t want to leave that off the table either.

Robert: The biggest positive is that from a community perspective we are getting a lot more up front review; a lot more users using features before they go into, say, the commercial version or whatever, right? We’re getting a lot more early feedback on things that we’re thinking about doing or the ways we’re thinking about implementing new features, lots of early feedback that has a very, I would say, positive influence on the end result of the code that we actually write.

Sean:Well, I’m curious, too. That’s a functional change in the product, but have there been any change to the engineering process on the closed source side of the product? So, you guys open source it, that obviously rattles around and after a couple of years; I’m wondering if that has had any kind of change in the way you guys have developed internally, just having an open source mirror of the project, so to speak.

Robert: Well, it’s not an open source mirror. We only have one copy of the code, and that’s the public one. So, any change that we make to the code, whether it’s for the commercial release or for the open source release, typically gets vetted through the same process. We’ll go to request for comment. That’ll go out to the open source community, because it’s going into the open source version, one way or the other. So, we get early feedback on pretty much everything that goes in.

Of course some features that we may plan are for the enterprise version only, and those of course do not go through public review. Also we reserve the right to decide which features go into which enterprise release and we make no announcements about that because we are a public company right?

Sean: Right, right.

Robert: We don’t comment on that. We don’t do anything like that in the RFC process. But at the same time, the open source community certainly gets to vet all this stuff early. So, any code changes, and basically the whole product’s development process, are morphed into more of an open source development process for both the commercial and open source releases, because, again, we only have one copy of the code and we have one development process.The entire engineering team has basically transitioned to an open process, which is quite interesting.

Sean: Taking it down one level, as we talked to people sometimes as they’ve gone through this change, and the engineering team has a variety of differing reactions to it…various resistance points, or to put it maybe slightly more diplomatically, just things they have to learn as they go through the process. So, anything that would be worth talking about there in terms of lessons learned by the engineering team as they walked through this?

Robert: Yeah, I think there were a lot of lessons learned. Immediately it comes back around to being open about the kind of change we want to make and how you actually communicate with an open source community, because communication within the internal engineering team is quite different in my experience than communication with an open source community.

Sean: Now all of a sudden it’s a much more communal feel and development managers all of the sudden, I assume, had to make the switch from assuming everybody was on the payroll; there had to be a little bit more negotiation that affected the process of getting something developed, vetted, built, etc.

Robert: That’s definitely part of it. But also for our internal developers, I mean, they’re used to this kind of hierarchy of being able to walk down the hall and talk to somebody else. They’re used to being able to call a meeting and go in and start scribbling some stuff on a whiteboard.

With an open source development community that doesn’t really work anymore. You need to use the mailing lists a lot more. You need to use things like IRC for communications. So, it was really a complete change in the way you communicate.

Scott: So, that’s one of the things that I wanted to drill into because you’re obviously steeped in closed source development methodology. You’ve been in open source long enough to where you’re now immersed in that.
What do you see as the differences, kind of taking them side-by-side when you think about the aspects of the SDL, right? …how’re ideas envisioned, how they’re kind of tasked for somebody to code them, how they’re put through QA, how a product is released… If you don’t mind, I wouldn’t mind just kind of walking through how each of those is sort of fundamentally different in a closed source versus open source project.

Robert: Sure.

Scott: So, I guess starting with analysis and design and that sort of thing, how do you see analysis and design and requirements differing in open and closed source?

Robert: Sure. The requirements from a closed source perspective, I mean, we have project managers who basically go out and talk to our users. They’ll do focus groups. They’ll do surveys; all kinds of things like that just to basically pin down to a set of requirements for release of commercial software.

For open source software, you have a community of users who propose some ideas or a developer who proposes an idea. It gets kicked around on a mailing list for awhile. Eventually somebody either just picks it up and implements it, or the other option is they just walk in with something they’ve already written and say, “Hey, how about this? Isn’t this great?” So, the difference is really to me that it’s a little bit more organic, if you will, in the open source community. Whereas analysis and requirements and product definition is a much more rigid process in the closed source world, in the open source world it kind of happens organically.

Scott: So, where do you see those potentially? I’m guessing that both sides have their advantages. What do you see as the advantages of focus groups, meeting with customers, and project planning and what do you see as being the advantages of kind of more of the organic method of sort of showing up with a feature and saying, “What about this?”

Robert: If you show up with a feature and it’s already got a prototype or an implementation done – it’s a lot easier to talk about because you can actually see it and you can play with it. Whereas in the closed source side, you’re talking more in the abstract, if you will. So, the advantage in the open source side is really that you typically get a lot more users providing some feedback on it, at least if they’re really interested. Eventually you get to experience it before it actually goes in, which are both useful things.

On the closed source side, I mean, the advantage is that the release potentially comes out with a set of features that are much more…I won’t say well-defined because I think that’s the wrong term, but a release will typically come out that is much more cohesive as a whole. Instead of having a bunch of new things sprinkled about like a typical open source project has that’re mostly unrelated, a typical commercial product release will have the sort of features that are more or less a cohesive unit, if you know what I mean. Does that make sense?

Scott: Yeah, yeah. When I was a developer one of the problems with me kind of deciding what should be in the product was that I could have a project manager come along and say, “The product should really do X.” I would try to convince them that it shouldn’t, not because I had a better knowledge of the customer than them, but because I knew what a mess that would be to try to code given the existing architecture.

Do you feel that that kind of thing happens on an open source project because a lot of the people working on it kind of know what it would take to code it? Like, they might shy towards or away from certain things?

Robert: Yeah, you don’t see that in the open source world in my experience.

Scott: OK. So, ideas get proposed on the mailing list, things like that. You guys have internal developers who are still working on it. Is it that everybody sort of volunteers to work on different things or do your own internal developers get tasked with certain features? Obviously in the community it’s sort of all volunteers, so somebody external has to decide to pick something up and run with it. I guess, talk a little about how that works in terms of actually getting the work done.

Robert: So, it doesn’t work all that differently for us because we do have an enterprise version of the product and we do have a product manager who still does all those typical closed source things. What typically happens is that the features that we foresee as important to get into the product will basically go through a similar process as a typical closed source world, but then we transition them over to the open source world.

So, we’ll basically write up, you know… if we have an idea we’ll either code it first, or more likely we’ll write what we call a “request for comment” document that basically describes the change we want to make then actually put it out there to the open source community to vet. Again, because we’re sponsoring it, the things that have somebody behind them who’s actually going to write the code will typically get approved by the PSC.

But, the benefit that I’ve seen in that is that we get a lot of really good feedback, positive influence, and changes to those RFCs before they actually get coded because we have a pretty broad user community out there in open source basically providing us with some really solid feedback.

Scott: Then in terms of things like QA and documentation, how much of that kind of gets done by the community? How much of that do you guys still have to take on?

Robert: For the open source project we do no QA. The open source community is responsible for all QA. So, between the software development team itself and the open source community, that’s where all testing happens for the open source releases. We do not have any type of internal QA or formalized QA process on the open source project.
We’ll decide that we want to cut a release, and the project steering committee will release a beta. We’ll get some users testing it, providing us with bugs. We’ll fix those bugs.

Autodesk has traditionally done betas, but they’re fairly late in the game. I would like to see us actually release our betas a lot earlier than we currently do to get more of that community testing done earlier in the cycle. Then, we have more of a chance to fix bugs, and make changes before we do our final release. We’ll go through basically one or two betas and a couple of release candidates. We’ve done up to two. We may do a third on 1.2, and then we’ll release the code.

So, with open source there’s no formal QA process other than the unit test has to run successfully on every build. But basically, it’s users hammering it, whereas for the closed source project we have our rigid, internal QA process. We have our internal QA team that does a fairly formal quality assurance testing that you would expect from commercial software.

Scott: Do you feel that one way or the other is more effective at kind of wringing the bugs out of the product?

Robert: What my gut tells me right now is that a blend of the two is actually best. The nice thing about the rigid QA testing is that when you release the product, you know exactly which code paths have been tested and which ones have not. What you don’t get out of that typically is the kind of real world testing that a user will put it through. So, my current thinking is that a blend of the two approaches would be better.

Sean: Just a piece of context – how late is late for you guys? If the project has a 36-month development cycle, or 24 months, or whatever, from the initial release definition or spec process down to when you actually have people buying the product, how far along on that trajectory are you guys typically releasing betas?

Robert: In commercial software typically our first beta release is about four months before release, and that for us is late. Part of that, though, is because Autodesk has adopted an annual release cycle, so that has dictated shortened cycles for everything. But, if we could find a way to release those new features earlier for beta testers to test, I think that would be a huge improvement and a way to blend the best of the open source world with the closed source commercial QA testing.

Scott: One of the challenges people sometimes run into when they open source an existing project is that on day one when you open source it, all the jobs on the project are already filled. You already have a team of 10 or 100 or 500 or whatever number of people who are engineers employed working on the projects, and they’re already sort of doing everything. So, there isn’t necessarily a feature that’s just waiting for somebody in the community to write.

So, from this perspective that kind of caused a lot of consternation internally about what was that going to mean for the people whose job was to work on the project. It kind of created issues with the community on day one about OK, well great, what do you really want us to do? I mean, what’s really open for us to work on that you’re not already working on?
Did you guys run into anything like that on this? I’m interested in kind of comparing the experiences. You obviously have a different product, and I’m interested if you ran into similar things or not really.

Robert: I would say we ran into the exact same thing. In fact, community contribution I think has been slow. It was really slow in the first year for two reasons: one, it’s a fairly large and complex codebase. It’s not the easiest thing to jump into. Two, for that exact reason. We had developers that were doing everything, so why do I need to do anything?

That has changed, however, I think in the last…I think this last six months or so we’re starting to see much more significant community contribution. Again, MapGuide is more of an application development platform for web mapping than anything else. It’s not really a product you install and start using. You install it. You build an application. You display it to a bunch of users. You refine it. A typical web product, right? So, what we’ve seen is that a number of people have seen limitations with respect to what they can do particularly around the client end of the product. They want more flexibility in the AJAX client. So, they started with some simple submissions – really just adding some little functionality.

But, of late we’ve actually had a solutions organization that got heavily interested in MapGuide early; DM Solutions in Ottawa. Basically, what they did is they actually wrote a new client front end; a new AJAX front end for MapGuide. They are actually now in the process of donating that back to the project. So, that’s a very significant piece of code that grew out of a need. Once they started developing some applications, they thought, “Hey, I’ve got some limitations here. We could probably do something about this with our experience.” So, they started writing some code. Sooner or later they had deployed it a couple of times for a couple of their customers. Eventually they came back to the project and said, “We think this will be useful for everyone, so we’d like to donate this.”

So, I think again it’s a matter of time on those commercial projects that choose to open source. I think eventually the time will come when developers will come onboard. But it takes a little bit of time for that exact reason. Initially, what do we need to do? Don’t know.

Scott: Well, it sounds like you guys are doing it the right way. I mean, the other reason sometimes why companies open source something is just because they want to sunset it to some degree. They’re done with it. They’re tired of it. They don’t want to put people on maintaining it, so they just say, “Here, we open sourced it. Community – go take it from here.” Those ones usually just kind of die pretty much on the spot.

Robert: Well, for us there’s some interesting history here you guys should probably know that I maybe didn’t introduce early enough for you. MapGuide was first released as an Autodesk product in 1996-1997. About three years ago we took on a complete ground-up rewrite. We were about halfway through that effort that we decided to open source.

So, we basically have a legacy product there in MapGuide. We started a complete rewrite project. We actually completed that complete rewrite project. That’s what we open sourced. But, halfway through it we decided the right thing for the future of this product was to actually open source it and we intend to continue maintaining and being a huge part of the development community on that project, but at the same time getting some open source involvement.

Scott: Yeah, that seems to be key. It seems to be key when the company that’s open sourcing it is going to continue to invest in it – that really seems to be a key to success. You know, companies seem to open source for a lot of different reasons. Sometimes it’s because they feel like maybe they’re not in the best position to do a certain piece of work – like if it requires porting to a different platform or architecture – or they feel like the community will have great ideas that they won’t generate internally.

Sometimes companies do it as a market disruptor. They see that the market is saturated with closed source, proprietary solutions, and by being the first ones to open source they can radically change the market. For you guys, what was your motivation in going open source? Why did that make sense for this particular product?

Robert: Well, I’d be lying if I said market disruption wasn’t a huge factor, because it certainly was. What we recognized halfway through this project… I mean, again, it’s basically a piece of web mapping technology. It’s basically a web based GIS platform. So, what we realized about halfway through… We lifted our heads up and looked around. We saw that web-based GIS was growing in popularity by leaps and bounds and ESRI is pretty entrenched. They are a commercial competitor. All of our other competitors have products in this space as well.

Then you’ve got Microsoft. You’ve got Google and Google Maps. You have Yahoo! Maps. There’s a bit of a commoditization around web mapping that we discovered. What we really realized is that the revenue opportunities for us are not in the platform per se, but more in the solutions that we can deliver on the platform. Does that make sense?

Scott: Yeah. So, by open sourcing the platform and being still heavily invested in it, you’re in a great position to provide solutions on top of that platform that might be hard for your competitors to match.

Robert: Right, because we can influence the platform’s direction as well as direction of our solution providers. Again, we’re a company founded in establishing developer partnerships. It’s really partners that develop solutions on our product.

All of our partners now can influence the direction of the open source project by either: a) actively becoming involved in development on the platform; or, b) hiring companies that specialize in open source development to go enhance the platform in some ways that they never could when it was a closed source project.

Scott: So, how does it work? Has open sourcing it disrupted the market and provided a lot of the benefits that you guys were initially hoping it would?

Robert: It’s definitely been disruptive and that’s beneficial. The larger benefit for me, though, is the fact that our solution partners now have an opportunity to influence the new platform in ways they never have before. They get to see things coming early. Typically we release the open source version two or three times a year, so they are getting features faster and they can start to work with them.

On top of that, they can influence things. The idea of this whole AJAX front end is a great example of that. It’s where a solutions development partner for us basically came in and developed a whole new client framework that’s going back into the platform.

Scott: Have you guys at this point had to contend with anybody who has wanted to do a really strong fork of the source? How have you prevented that or dealt with that?

Robert: I don’t see anybody wanting to do a fork. It’s LGPL licensed. That helps a lot in that respect.
My crystal opinion on that, by the way, is that if you have a project that has a good community behind it, that is a cohesive community, I don’t think you’ll see a fork.

Scott: Yeah, it seems like there’s definitely a high bar. Forking is the nuclear option. For somebody to fork and have it get traction, they have to be making at least the same amount of investment in it as you guys are. That would be a pretty high risk thing to take on.

Robert: Right.

Scott: It seems like as long as the company is really meeting the needs of the community and supporting the community in the right way, it doesn’t really happen. But Netscape is an example of where it did and the outcome of that was Firefox. People really felt like the main sponsor was really not taking it in the right direction and they had a better idea.

Robert: Exactly. Let’s be honest – that’s one of the reasons that the project steering committee, who really do have control, is actually four community people and only three employees. We can always veto something because it only takes one vote to veto. But, in my experience so far, working with this committee in the last 18 months, I don’t see us ever actually using a veto.

Scott: Great.

Robert: In communities you can suddenly get a rift with the company that wants to maintain control, or sometimes you get a rift in the development team itself. Drupal is a good example of that. They had a rift in the middle of the development team, and that resulted in the fork of Drupal. I don’t think that’s a common thing.

Scott: Right.

Robert: If you have a good, well-oiled community, I don’t think that’s likely to happen in a company that’s open.

Scott: Great.

Scott: This was fantastic. This was really informative from our perspective. It’s nice to be able to talk to somebody who’s got so much experience in both realms and can provide some objective insight about their experiences on both sides. So, this was really great for me.

Sean: Yes, same here. And Bob, we’ve extended this courtesy to everybody. We’re just curious if there’s anything else you want to get on the record or another question you’d like to propose, or things along those lines before we wrap up?

Robert: No, I think the only thing we really didn’t get to was one of your questions earlier about QA and documentation; I didn’t touch on the documentation piece. I would say that’s one of our biggest weaknesses with open source right now was that we really don’t have a solution around documentation in the community right now.
That’s something that I think will still take time to formulate. Everybody is busy and nobody has time to write documentation. That’s actually my experience as one of the issues with open source projects.

Scott: Unless you’ve got a sponsor who staffs that, it’s just not the most fun stuff to work on.

Robert: Right. So for now, most of the open source documentation has been staffed by Autodesk. But that’s getting harder and harder for us to continue. So, that’s one of the places that I am, if anything, a little concerned about for the future; the documentation and trying to get a way to continue to staff it.

Scott: I don’t think that’s unique. Part of the problem is the people who really, really work on it, don’t really need the documentation. So, they don’t necessarily feel the need as much as the people who aren’t as active in the community itself.

Robert: Right. But the problem is that it’s hard to get new users on board when you don’t have good documentation. That’s really the challenge.

Scott: Right.

Sean: Well, Bob, I really want to thank you for taking the time. I couldn’t agree more that it was a great interview. It was really, really nice perspective we got.

Bookmark this:

Interviewee: Jay Pipes

Jay Pipes

In this interview, we speak with Jay Pipes North American Community Relations Manager at MySQL.

We talk about:

• How MySQL Community and Enterprise servers are nothing like RedHat Fedora and RHEL.
• The nuances of MySQL licensing.
• It’s built mostly like traditional software, with 120 developers on staff at MySQL.
• But that’s changing, the worklog system has been opened to the community and they’re starting to take contributions.
• For that to happen, MySQL has to become more modular and support plug-ins.
• MySQL supports so many platforms through a portability layer, which was a lot of work initially, but now it’s pretty well baked.
• Bugs = Gold.
• The tenets of MySQL: performance, reliability, and ease of use.
• It’s easier for proprietary software to make money, it’s easier for open-source to get work done.
• And finally, how to get involved.

Jay Pipes: My name is Jay Pipes. I work for MySQL. I’m the North American Community Relations Manager. Part of our job as community managers is to monitor, encourage, and grow the external MySQL ecosystem as much as possible. [This involves] responding to concerns of the community, pushing those concerns internally, being an advocate for the community within MySQL—being a liaison between free and open-source projects and companies and MySQL. Also, each of us has different responsibilities. I do a bit of development work for the MySQL Forge website. I work a bit with engineering. I do a lot of conferences and speaking engagements on performance tuning, and blogging, and writing, and that kind of thing.

Scott: I get it. You’re greasing the wheels between MySQL and the community at large, and trying to keep the information flowing friction-free in both directions.

Jay: Yes, I want to remove the friction points between contributors to MySQL and users of MySQL, developers and DBAs, and the flow of information from MySQL to make sure it’s accurate—that they understand. Especially with MySQL, we’re in a unique position. We provide this open-source software and we have this enormous, ubiquitous user community, but at the same time we’re selling products. So, because it started out as an open-source project, and the community knows that, a lot of people don’t even realize that MySQL is a company.

A lot of times we have to explain that MySQL as a company has to do certain things to provide revenue that will in turn benefit the open source community. Sometimes that’s the challenge. There’s this push and pull between commercial and the community. And the community team at MySQL, including me—a lot of times we get asked to advocate for the community within the company, but we also have to have an open mind. We are a revenue producing company, and at the same time, we want to be good for the community. That kind of balance is most of what my job is.

Scott: It’s interesting that you bring that up. There’s still this feeling like it still works the way it did six or seven years ago where open-source was people working on stuff in their basement. But if you look at any of the major projects out there, the model has really shifted. If you look at the Linux kernel itself—that’s a lot of engineers at Red Hat, IBM, Intel, and various different companies that are really doing the bulk of coding. If you look at something like RedHat/Enterprise Linux—again, this is a commercial enterprise, right? This is an ongoing business that has come up with a business model around something open source. And so, with you guys—help me understand it a little bit…

Jay: I will try and clarify something. A lot of people say, “Well, when MySQL just recently (in September or October) split between the community server and the enterprise server in MySQL…“ A lot of people equated that to RedHat/Enterprise Linux split. One of the things significantly different between MySQL and RedHat, is that within RedHat does produce its own pieces of software within the RHEL stack and the Fedora community packaging, it doesn’t have total control over its software — their software stack is dependent on upstream committers. Obviously its engineers contribute to the Linux kernel and various other things.

But MySQL has always been 99% completely written by MySQL employees. So, it’s a different model in that RedHat (until more recently where they’ve started to create more of their own software) has been more of a packager than it has been a producer of software. And there is a difference there. It’s what’s made it difficult for people to say, “Well, MySQL has done the Fedora versus RHEL split.

Scott: OK.

Jay: It is different because we are producing almost 100% of the code that we’re selling in the enterprise and then giving away in the community. Does that make sense?

Scott: Totally. And so, I’m sure you can explain this better, but if you’re using MySQL as part of an open-source project, then MySQL can be freely distributed with that project. But if you’re building something commercial on top of MySQL, then my understanding is that’s where you actually have to purchase a license. Or do I have it all wrong?

Jay: Well, it depends. A lot of people are confused by the licensing. Some of the confusion stems from the fact that it is GPL. Some confusion is that we sell commercially-licensed MySQL for those OEMs and ISVs that embed MySQL within their product and then distribute it. So, GPL is all about distribution—reciprocity and distribution.

If you distribute your product with MySQL, and your product is not GPL or a GPL-compatible license and is not an open-source product, then yes, you are required to pay a licensing fee to My SQL. However, a lot of people will say, “I’m distributing my product as a non-open-source piece of software. it can connect to MySQL.” So, if you have a MySQL server running on your network, my application can connect to that server and run against it.

There are different ways of using MySQL, but a lot of where the licensing comes in is when you’re an original equipment manufacturer and an independent service vendor—or whatever ISV is anymore—and you’re embedding and distributing MySQL as an integral part of your application.

That’s where the licensing comes into play. But not for web applications, when you have MySQL installed on the server and you’re a service-oriented application provider. That’s not where the licensing comes in. That’s where we sell the MySQL Enterprise edition, which is the support and services offering.

Scott: Got you. So, just to make sure I understand that: If I’m an ISV and I’m selling my software, and I package MySQL with it, I have two options. I can distribute my software under the GPL license, or I can pay a licensing fee to MySQL and distribute my source under a proprietary license. Is that it?

Jay: That is correct.

Scott: OK. OK, good.

Jay: And in the past, there’s been confusion about the protocol, but that’s not the issue. I think people have blown that up out of proportion. What you just said is exactly what it is. If you are distributing your application packaged with MySQL and you’re not releasing under GPL or a GPL-compatible license, you have to buy a license for each copy of your software that you distribute, because you are then distributing MySQL. Now, the licensing costs have also been blown completely out of proportion by a lot of people. People say, “Oh, it costs $595 per distribution.” That’s not right either..

The pricing depends on whatever the sales team at MySQL and you agree on. It’s just like any other company. But that is the case when you need to buy licensing, when you embed MySQL and you do not want to release your source code as GPL or compatible licensing.

Scott: Cool. And I’m just not involved enough in the community to really even be aware of what have obviously been contentious conversations, probably over the years. So it’s just me trying to understand it, coming…

Jay: No, no. I understand where you’re coming from. I was referring to this kind of myth that MySQL licensing is overly complicated. I work for MySQL, so I’m biased. But I don’t particularly think that is a complex process to understand. I think that the complexity really stems from the fact that there are just a ton of open-source licenses out there that all have these weird idiosyncrasies to them. And I think that complexity lends itself to, “Well, MySQL is open source, so it’s going to be complicated.”

Scott: Right. So, the main thing we’re focusing on is how software is built. We’re looking at things like Apache and the Linux Kernel. From the outside it kind of looks like they’re built over a mailing list. In other words, there are these core mailing lists, people post code to them, it gets reviewed, there’s a maintainer who decides whether it gets checked into the main tree or not.

Jay: That’s absolutely true, with Linux, Apache and Eclipse. Eclipse is more bureaucratic than that; they have hierarchies and procedures and policies and incubation periods and all that stuff—as does Apache.

Scott: With you guys, it looks more like a traditional proprietary shop. I’m guessing that you sit down and have meetings. You discuss what features are going to be slated for the next release. You come up with project plans; you come up with specs.

Jay: If only it were that simple. Yeah, on the outside is does look like a traditional software house, in that we have maybe 120 engineers working on various teams, from people that work on the connectors and the GUI tools to people that work on the server runtimes or the backup and replication folks.

Scott: Which is a lot. When you talk about 120 engineers,that’s a mid-size software company.

Jay: Absolutely. Absolutely. And yes it’s true that we have scrums internally. We have internal roadmaps. And up until, I would say, December or January of this year, what has been more of a cathedral-type model, meaning more of a closed-source model for development at MySQL. It’s now starting to open up significantly.

Recently, we opened up our internal work log system, which is as close as you’re going to get to a list of roadmap tasks that we’re working on. This is for anything from MySQL 5.2 and up to MySQL 8.1 and beyond from all sorts of crazy wish-list ideas to stuff that’s actually going into the code at this point. We’ve opened that up publicly on our MySQL Forge (http://forge.mysql.com/worklog/). People can comment on these tasks, provide suggestions and vote on things that they’d like to see. We’ve also started accepting more contributions from the outside community. So, it’s starting to be more of mix of an open-source project and a commercial company model. We’ll see how it goes. I’m obviously pushing for more of the open-source development model, having more outside committers and contributors that are providing both external tools to the MySQL server, but also fixing bugs and provided patches for small features within the server itself.

Scott: That’s interesting that you’re transitioning from the cathedral to the bazaar, to some degree.

One of the things that I don’t understand about things like the Linux Kernel is how really big sub-systems get built or worked on. At the point where IBM decides, “OK, we just need this in the kernel.” They don’t ultimately kind of get to say, right? If Linus doesn’t like it, it doesn’t make it in. At the same time, corporations are sort of doing the bulk of the development.

So, with MySQL, how do you see that shaking out? Certainly MySQL, the company, is going to control the direction of the product, you’re opening it up to take more community input both in terms of suggestions and ideas and in terms of actual feature code and things like that. But I’m guessing there will still be a pretty significant section of the product that will be spec-ed out. A team of engineers will be put on it to build out a feature. It’ll go through…

Jay: I think it will be a mix of both. And we’re still going through these growing pains of figuring out how this is going to work. The community team is going to be pushing more and more contributions from the community, and MySQL doesn’t dictate those in any way. Someone can hop on there and say, “You know what, I want to implement check constraints, and here’s the patch for it.” What will be the issue is which version of MySQL will that patch make it into. And will it be a module that will be marked as experimental? Will it be something that will be patched into the core kernel?

That’s the process that we’re currently going through. We’re still in these growing pains. We’re still really just now figuring out how to handle these kinds of contributions. So, over the next year or two, I think we’ll start to hammer that out, and understand, “This is going to go into the community server, and this is going to go into the core kernel.” I think, as we make the core server more modular, that issues like that are going to start to disappear a little more, because someone can provide a module, just like mod_ssl for Apache.

Scott: Right.

Jay: It can be a self-contained component that isn’t necessarily going to kill the main, core kernel of MySQL. And so it’s not going to be as big of an issue, because we can package and version up that module separately from the core kernel, and the community person can have it out there. Until we get to that modular core piece of MySQL, it’s going to be a little bit of a difficult road, as we decide how to patch that stuff in. But, on the commercial side, we’re always going to have companies that will provide us with what we call NRE, non-recurring engineering.

Scott: Right.

Jay: Which is basically, someone’s paying us to put a feature into MySQL that is vital, or mission-critical, for their business. So recently, a lot of that type of work has gone into the NDB Cluster tool, which is our high-availability tool. Telecom companies that extensively use MySQL Cluster would like certain things, and they’re paying for those things to get included. And we’ve got those type of projects going on all the time. In the next year or two, we’ll start to see a bigger balance of community driven activity, in engineering, and commercially driven activity.

Scott: Looking at successful open source software, I think you’re exactly right. Modularity seems to be completely essential. I might have trouble getting something into the Apache core, but I wouldn’t have any trouble writing a module and just putting it out there, and if people like it…

Jay: Absolutely, absolutely. And that’s the key to the community driven coding, is that once we get that architecture completed — the plugin interface — where people can write add-ons and extensions to MySQL – that problem of, “OK, which version of the server? Can we put this in there without destabilizing the core runtime?” Those kinds of questions will cease to be an issue. And so will packaging issues, because the community person can put it on their website: “Hey, this is my module for MySQL. Go download and install lit.”

Scott: Right.

Jay: Just like you would with any of the weird Apache modules that are floating out there.

Scott: Right. Apache, Eclipse. All of these things have a modular architecture. To be an open-source project that’s taking community contributions, it seems essential to have that modular architecture.

Jay: I think it is, yeah. It’s going to be a long ways to go. From my understanding, from the engineering team, it’s not something that happens overnight and, certainly, is going to take lower precedence to some of the commercial work that we need to get done on the server. And obviously, our roadmap is years ahead of time. [laughs] The stuff that’s going into MySQL 6.0 and 7.0 is already on the block.

Scott: Right.

Jay: Bringing stuff up like the modularization of the core kernel, we’re looking at two years down the road. But it’s still, in my opinion, vital to start thinking about this now if we’re going to really get to a point where a community is actively contributing to MySQL.

Scott: So, from a software development standpoint, that seems like it would be a particular challenge for MySQL is just that MySQL runs on so many different platforms. You guys have, I don’t know how many kind of distros for a given version. You run on Windows. You run on Mac. You run on a whole bunch of different Linux flavors. How much of the engineering effort do you feel like goes into features themselves, versus how much of the engineering effort goes into making a distribution that runs on such an enormously wide variety of platforms?

Jay: Yeah. I’m not privy to the exact numbers. I can take a guess, though. I would say that the portability layer that allows us to run on these various platforms is fairly stable. Not that it runs perfectly on all the platforms, but that we do run on all the major platforms. And the reason we can do that is an underlying subsystem that takes care of the portability between those systems. That’s been around for a while, so, unless we’re talking about newer things, like Windows 64-bit running on Falcon — which is our new storage engine coming out — I think a lot of that’s already been done. So, most of the work is really in the features, and a lot less in the portability layers.

Scott: OK.

Jay: And I would say that is the case with, say, PHP or Apache or Python, or many of the major open source projects. That core portability layer was a key thing early on, and it’s stabilized pretty dramatically recently, so that’s not really what people are working on; it’s more of the feature-wise stuff.

Scott: Got you. Got you. So, the only time the portability layer really needs significant engineering, like you said, is if you’re porting to a whole different architecture, like 64-bit, or something like that.

Jay: Or when you’re specifically profiling bottlenecks on a specific architecture.

Scott: Right.

Jay: But that’s more of a performance thing and less of, “Will it work on the platform?”

Scott: So, talk to me about some of the other stuff that goes into building a product, things like testing and QA. How does that work? MySQL, I’m assuming that it’s got pre-release beta builds, or daily builds, or things like that, that you can pull down.

Jay: Yeah. In fact, the release schedule of MySQL, on the way it’s built, I don’t think is going to be much different from most other open-source projects. We have an internal build team, which I think there’s four or five people on it, maybe. They are responsible for the overall release management: making sure that the builds compile on all platforms, that the binaries are stable on the major platforms. And also, building up the release notes, making sure the flags and switches that are relevant for each platform are turned on or off depending on what’s needed, and that everything runs through our internal push build system, which, essentially, is an automated system that says, “Will this build on this architecture?” And then, we also have a QA and testing team.

It used to be a single team. Now, we have one man, Omer Bar Nir, who’s the QA architect over the whole thing. But we have QA engineers, now, attached to each of the development teams. And so, they are focused specifically on the QA and testing of, say, backup and replication, or the storage engines, and things like that. So, where it used to be that the QA was across the board, now they’ve split up and focused on specific pieces. And I don’t think that’s very different from any other open source project. The way we release is, we use a tool called Bit Keeper for our source control, and we do nightly or daily snapshots from that, which you can take and build the source code yourself.

And then, once in a while, we’ll package up the source code into tar balls, or zip files, depending on what platform you need. And then, depending on what version of MySQL, whether it’s Enterprise or Community, they’re built into binaries and then distributed. All the distros that I know of don’t use the binaries at all. All the Linux distributions, they actually take the source, from either BitKeeper, or from the source tar balls for a release, and then modify it to suit their needs, mostly by where the configuration files go on install, what’s in there by default, all that kind of stuff. And then they package it up into a. DEB or an RPM, or whatever it is.

Scott: Right.

Jay: Now, for Windows folks, the vast majority of Windows users don’t have the ability to compile software locally on Windows. So, it’s much more important that we provide binaries for MySQL on Windows than it is for the Linux folks. So, that’s most of why MySQL has been providing binaries for so long. Also, we say, “If we built the binary, we’re assuring you that it’s stable on that platform.” And to be honest, most of the Linux distros are very stable as well. The same goes for Mac and Windows. We built the binaries so people can download them and install them.

Scott: So, what percentage do you feel like of the QA or of the bugs that are found and posted for you guys to fix, how much is found by internal QA versus how critical is the community to wringing those bugs out of the product while you’re posting the daily builds, moving towards release?

Jay: That’s a good question. I would have to refer to Omer and the MySQL group, they kind of have these stats. But I would say that, internally, probably 10% to 20% of the bugs are found by MySQL engineers or support engineers. And then you’re going to come across this gray border between who’s a user and who’s a customer.

A lot of users are also customers. Sometimes we’ll get a fairly large installation, say, Yahoo Finance or Google, that submits a bug on a specific version of MySQL. But a lot of times we’ll get larger installations from users as well.

We also have something called the Quality Contributor Program, which is for users that are really our bug seekers. They’re actively trying to find edge cases where stuff just blows up. And so we have a program for people like that. But overall I’d say it’s fairly spread out between internal folks finding bugs, customers finding bugs, and then the larger communities finding bugs.

Scott: Got you.

Jay: But we do get a ton of bugs. The majority are small. In other words, documentation type stuff. Whenever I’m giving a talk on MySQL, I talk about community. And I always say that bugs are gold to MySQL. We value them just as much as anything else from the community—especially a reproducible bug case.

Scott: Right.

Jay: Because it saves so much time for the engineers. Run this code and there, it crashes. That kind of thing is gold to MySQL. So, I’m always encouraging people, “If you ever find a bug in MySQL, don’t ignore it. Send it in.”

Scott: That makes perfect sense. Talk about the work that MySQL—I mean, obviously it’s used so pervasively now, and lots of mission-critical stuff is built on it—what things do you do around security, reliability, all of the “itys” that people talk about with software?

Jay: All the “itys.” [laughter]

Scott: Stability, reliability.

Jay: That’s a good… Performanceability. [laughter] Usability.

Scott: Usability, scalability.

Jay: The three things that MySQL always strived for are performance, reliability, and ease of use. Those are the three binding principles of how our engineers kind of evaluate how well we’re doing. Is it easy to use? Does it perform well? Is it reliable? As far as security and stuff, as an open-source project we tend to worry a little bit less about security. There are just so many people looking for security holes in the software, because they can see the source code and look at it. They can see major problem areas and we usually get notified quickly and respond very quickly to those kind of things. Let’s see… Scalability. I’m biased, but I think we scale very, very well. And it’s always something we’re thinking about internally, because performance doesn’t necessarily mean scalability. You can get a hundred concurrent connections for doing web pages or responses at half a millisecond, but if you can get 10,000 concurrent connections at 0.7 seconds, it’s less performance but better scalability. And there’s sort of this constant refactoring process going on. How can we make this better? How can make it scale better. All that kind of stuff.

Scott: Cool.

Jay: Which I’m sure is the same with any closed-source software house and any open-source project as well. You’re always thinking about all those “itys.”

Scott: Right, right. Well, you’ve got a mature product, so you’re not engineering it from scratch to have all of those, but you’re evolving it, and a lot of the work now is more in terms of…Either there’s a well-defined opportunity to rewrite something and increase performance and scalability, or you’re really just — as you add new features — trying to make sure you don’t negatively impact those areas that are already good.

Jay: Right. As you increase the features in the code base, you increase the code complexity, and you always look out for performance regressions because of that. And there’s a way to combat that, but the general rule of thumb is, the more code you add to something, you’re going to impact the performance. So, there’s always this balance. Do we need this feature? Because the last thing I think MySQL wants to become is—no offense to Oracle or PostgreSQL—a database that has a million features that no one uses.

Scott: Right.

Jay: And that’s something that does go through the mind of the software architects at MySQL. Is this a critical functionality that the majority of users are going to use and are going to value, or is it going to be passed over and just slow down the code?

Scott: That seems to be a key challenge of closed-source proprietary companies, is that they really have to guess. They have to shoot in the dark in terms of… First they have to come up with big features, because if they don’t, they can’t compel people to buy the upgrade.

Jay: Right. Which is the opposite of how MySQL sells our stuff. We’re not trying to be this enormous feature-rich database piece. We’re trying to be the best and fastest online database. And so the features that we’re adding are designed for highly-scalable web applications and online databases.

Scott: And I would guess, too, as you open it up to more and more community input, it will become easier to identify features which will be widely used. Because, the worst thing in the world is to write a feature, nobody uses it, but you can’t ever cut it because it’s actually not that nobody uses it, it’s that three people use it.

Jay: Yeah. And this goes back to that, what’s commercial versus what’s community? And I think that actually closed-source software companies have more of a problem with this, in that a large customer really, really wants this feature in there. And it’ll be added into the code base and sometimes significantly affect performance of another piece, but it’s been bought and paid for by a customer and will stay in there. And unless the software is written in a modular fashion, it will impact adversely everyone else who will never use that feature. And I think the open-source model, which tends to lean towards a more modular architecture, can handle that dilemma better.

Scott: And also, in things that are very, very open source, where most of the code is coming from community contributors, there are no features that are being written because somebody thinks someone else will want them. The only features that are being written are, “I’m writing this feature because I need it.” So, that’s at least a little validation that the feature is needed by somebody.

Jay: Sure.

Scott: But, doesn’t MySQL have just exactly the problem you talked about? Because you mentioned that you guys do some nonrecurring engineering.

Jay: Yeah. And that’s the dilemma that all commercial software companies are faced with. Which is why I’ve said as we move more and more into that mix of community input and also making that core kernel much more modular, I think that we can significantly offset the disadvantage of that, or the drawbacks of having nonrecurring engineering work done.

Scott: So, initially, I’m guessing the main driver of having MySQL be open source was just so that it would be used and accepted. In other words, it’s a much more difficult proposition to sell something that’s completely closed-source proprietary that targets Linux as a primary platform. And so, it seems like a lot of companies open-source their software and they derive their revenue off other things, support and that kind of stuff.

Jay: And packaging.

Scott: Yeah, and packaging. Otherwise you’re just not in the game.

Jay: Right. Well, one part of it is the revenue, right? When you look at open source versus commercial, commercial has just an enormous advantage from a revenue perspective because of their control over their product, right? The open-source company doesn’t necessarily have that. From the exact opposite end of the spectrum, the open-source company doesn’t have nearly the amount of cost involved in R&D, QA, and testing that a closed-source company does.

So, the big shift that’s happened is that you’re going to see closed-source companies start to open source products that they are tired of spending money supporting, and let the open-source community take on the cost of that support. Now, I’ve read recently that Microsoft is open sourcing Visual FoxPro, which I thought was awesome. And then I started thinking about it. I’m like, “Well, they’re probably just tired of supporting it. And just give it to the open-source community and see what they do with it.” And I think that’s where we’ll start to see the first major shift with closed source companies that open-source products because they realize that the cost benefit of doing that, and letting that source out there to the open-source community to test and QA and support, is just so much more worth it than keeping an older product in-house that’s really is not making any revenue.

Scott: Well, and one of the places where you see something similar to that is Adobe open-sourcing the Flex SDK. And to them it makes sense because it wasn’t something that they ever sold anyways.

Jay: Right.

Scott: So, it was free to begin with. There doesn’t seem to be a lot of downsides in open sourcing it.

Jay: And there is a difference between free as in no-cost and open source. And MySQL is open source, and free and open source as in free as in freedom. But it doesn’t necessarily mean that just because something is GPL or is open source that it’s free of cost.

Scott: Right.

Jay: And the original definition of free and open-source software really had nothing to do with cost.

Scott: Right.

Jay: Right. So, when I talked about Microsoft or other companies open-sourcing and making free software, I meant it more in the sense of free as in freedom, so that the developers can get their hands on it, and tool with it, and tweak it, and completely change it, and support it themselves. It had less to do with charging for it.

Scott: But it seems to me like a place where companies are looking at open-sourcing products and making them free as in freedom, are places where the product was already free as in cost, or maybe it wasn’t free but it’s just not generating a lot of revenue.

Jay: Exactly. It’s costing more for them to support it than it would the open-source community. And that’s where I think the first round of closed source becoming open source is going to happen.

Scott: So, MySQL started out as being open-source, but developed by a for-profit company. And at this point you’re moving to more of a traditional open-source model. And I’m guessing it’s because as the product has grown, and as it’s gotten more complex, and there are more and more features to it, there’s a need to kind of force it to be more modular. There’s a need to get more community involvement in the development…

Jay: Yeah, and input. Right.

Scott: And just to really shape the direction because it’s not just a single little standalone database anymore, right? I mean, this is a pretty massive product that you have. And, you’re at that point where you need that cost savings that the community can provide. And you need the input, and direction, and expertise that the community can provide to really move the product forward in the best possible way. Am I summing it up correctly?

Jay: Yeah. Yeah, although I don’t think the plan was ever to push it towards the Apache or Linux model. But we want more of a balance, just like you said, so that we can get the benefits of the community. But also, so we can give more back to the community, and have them happier with the product that we’re providing.

Scott: Sure, sure. There’s more buy-in when it’s something that you can actually work on.

Jay: Right.

Scott: And there’s more buy-in when you touch the product, so to speak, I guess.

Jay: Right.

Scott: OK. Well, I guess, what else would you like to say? I’ve gone through the questions that I had. I’ll go ahead and hand you the microphone. And any message that you think is important to get out that I didn’t cover, feel free.

Jay: Yeah. Well, I definitely did want to point out that the MySQL community headquarters is becoming the MySQL Forge website. And that is http://forge.mysql.com. And, right now, what we have in there is a list of open-source and MySQL-related projects in a project directory (http://forge.mysql.com/projects/). A whole directory of code snippets in various languages on how to use MySQL, or coded in C++, PHP, Perl, SQL,.NET, you name it is in there (http://forge.mysql.com/snippets/). And then we also have our public worklogs, which I mentioned earlier. It essentially represents our big roadmap. And it’s broken into specific tasks that are unassigned or assigned to a specific developer. And the developers really want input from the community. So there’s a way of commenting on those work blogs that I highly encourage people that are interested in MySQL to go and give your input to the developer of that specific piece, and let them know what you think about it. So, forge.mysql.com. And then there’s also a huge Wiki that we’re developing as well. That would be the last thing I’d say.

Scott: So then, if people want to contribute code to MySQL, if they want to actually work on it, how set up are you to take community contributions at this point? Or what do you think the timeline is to where you’ll really…

Jay: We’ve accepted, I think, about 80 contributions already this year—contributions meaning one-line patches to semi-large features. So, we’re already set up to do that. If anyone is ever interested in doing that. I would highly suggest going to http://forge.mysql.com/wiki/Contributing.

Scott: Got you.

Jay: And, that is how you can get all sorts of information on the various ways you can contribute, the different IRC channels where the hackers hangout, and where you can get help, how you can build a test case, all that kind of stuff. How you can build locally, and all that kind of information is all in that page?

Scott: Jay, thanks for taking the time to chat.

Jay: Thanks.

Bookmark this:

Patrick Hogan

In this interview, we spoke with Patrick Hogan about open source at NASA. Patrick has been managing NASA open source projects since 2002, incubating competitive technologies to deliver scientific content. The goal has been to engineer open source solutions that leverage open data standards for sustainable technologies that can be extended in both open and proprietary ways. Several successful projects have come out of this program, including a virtual scanning electron microscope, software that allows the blind to aurally visualize mathematical equations, and the very successful NASA World Wind, a fully navigable 3D geospatial data visualization platform.

In this interview, Patrick talks about:

• The NASA World Wind open-source project
• Preventing a strong fork
• Spec’ing features on an open-source project   
• World Wind extensibility, even for non-programmers
• Not everything the government writes is open-source and/or public domain
• You add value on open source, not in open source

Scott Swigart: Hi, Patrick. Thanks for taking the time to chat. You head up World Wind, which is an open source NASA project. Can you tell us about World Wind?

Patrick: NASA World Wind began with a program called NASA Learning Technologies, and was purposed to get NASA content into the classroom. NASA has lots of data, but we needed continuous dynamic access to deliver that information in a manner that was video-game-like. So World Wind began as a skunk works project to build this 3D geospatial visualization technology. And that’s where World Wind came from — out of NASA Learning Technologies.

It was successful enough that NASA independently funded it for about a year. Following that, the Department of Energy has gotten behind development of this project with a cross-platform approach, so now we have World Wind Java.

Scott: Talk a little bit about how the community integrates with it. Do you find the value of open source to the project to be that people can look at the source code, learn from it, and better utilize it, or do you get a significant amount of contributions from the external community at large?

Patrick: World Wind Central grew out of an IRC gathering of hobbyists who wanted to participate with the NASA World Wind effort. That interest evolved into what you see today at www.WorldWindCentral.com. They formally call themselves the Free Earth Foundation, but World Wind Central is their website.

Together they have made significant contributions to World Wind. These have improved World Wind’s ability to deliver pixels and greatly expanded its functionality. If you visit that website and look at the full add-on list, you’ll see hundreds of additional functionalities.

Scott: Does this run like a typical open source project where proposed changes are posted to a mailing list, they get scrutinized, and people post feedback on them?

Patrick: Right. The code is on SourceForge, and there are branches – these tend to coalesce when we get to a release. And then it starts all over again with sandboxes where people are playing with new ideas. And some of these ideas get put back into the main branch, and it goes from there.

I will say that as a NASA World Wind release, NASA gets to ‘approve’ the official release, probably similarly to the way Sun runs Java. Everybody can make a contribution, but still some entity is trying to focus that effort.

Scott: I think all open source projects run as enlightened dictatorships. Even in the case of Linux kernel, Linus decides what’s in and what’s not in.

Patrick: We have that here, too; it’s the same circumstance, and I think you’re right. The real art is in being able to prevent a strong fork — to have enough confidence within the community that they stick around and focus on a singular effort. As long as we can maintain that, I think we’ve got a good future.

We have a really active and very supportive community.

Scott: What about the modularity? You mentioned that there are hundreds of add-ons. What does the extensibility story for World Wind look like?

Patrick: Both the idea of a NASA open source project and the visualization technology itself have evolved. We’ve learned a lot from when we first started this project in 2002. The API for World Wind .NET was built after the basic code was established.

It wasn’t originally built with the architecture of having the API at the core, which is what we’ve done now with the Java version, and which is what we would love to go back and do with the .NET version.

Open APIs allow people to add things and leverage it in ways completely independent of the actual — for lack of a better word, we’ll call it an application — whether that’s a free add-on, plug-in, or whatever, or whether it’s an actual marketed vale-added functionality.

NASA World Wind is being used internally by the government and it is my understanding that it’s also being used internally by some private companies. These entities are doing things with this visualization technology that services their particular needs. I’m sure that many of these activities qualify as proprietary, even if it’s not something that they’re out marketing.

Scott: Does the licensing of it allow you to distribute a closed-source proprietary program that includes World Wind?

Patrick: Absolutely, and it’s the same as with the HP security module for Linux. You have to put some additional code into Linux to get that to work the way it needs to; I think it costs a few grand a year.

Scott: I thought GPL required anything that uses it to also be open source?

Patrick: Let’s say you need a special hook for your independent application, and World Wind doesn’t already have the hook that you want. So, you put that hook into the open source code. That hook would fall under the NASA open source agreement license.

Scott: So if you have an application that does a whole bunch of stuff, and World Wind is a small piece of it, you’re not required to open source your application if you make extensions or derivatives of World Wind as part of integrating it into your application? That small piece has to be open source, but not the entire app?

Patrick: Right. You can distribute World Wind, with that additional code if you want. The optimum idea would be to participate with NASA to get that particular need included as a standard part of successive releases. That way, whenever a new World Wind comes out, your particular module, application, extension, whatever you want to call it, is still viable. That’s the advantage of working with us, but there’s nothing to prevent you from doing this independently.

Scott: So, I see you’re on version 1.4. Talk a little bit about how features get spec’d. How does that work where you decide what’s going to go in 1.5 or 2.0 or how features go from being concepts to being actually in the product?

Patrick: The World Wind central group organizes meetings that we participated in prior to that release, with a wiki for the agenda to determine what the next release was going to have.

Everyone supplied their wish list and then everyone made comments about those particular features that were proposed for inclusion, in terms of what they think is important, what they think is not so important, and what they think is really cool. Then we would IRC chat over these issues and discuss on the forum.

For release 1.4, after we all had made comments on what we liked or didn’t like about the features that were proposed, we had meetings to hash it out. Then we agreed on a release date, and certain things — if they weren’t buggy — were included. And if they were on the final list to be included but were buggy, they were not included. So it’s been a community effort.

Scott: A lot of times in the open source world people can propose an idea if they’re able to propose the code that goes along with it. So, in other words, the way an idea gets proposed is that a developer codes something up and submits it. But it sounds to me like you have a process where you wouldn’t necessarily have to be able to code it to request the feature and maybe get it slated for development.

Patrick: Right. If it’s a good idea and you can inspire someone to deliver against it, then that’s definitely part of the mechanism. So, that’s true, although it really does boil down to having somebody who’s qualified to pick up that branding iron.

But somebody might also have a really good idea and kind of teach themselves, even if they’re not quite developers. There’s a difference between add-on and plug-in — a plug-in is a little more serious, and an add-on is often based on an XML file. There are people that have looked at other XML files and used them as templates. They make a template of other add-ons and just tweak it to better serve their particular interests.

So you can make an add-on without being a programmer, but to actually make changes to the code to help that particular add-on do what you want it to do, that’s some of the stuff where this community has really provided technical expertise in concert with other folks who have a good idea and a lot of energy to gather and organize the necessary data for the add-on.

The two analysis tools for earthquake events were developed in that kind of mode. In this case, Chad Zimmerman assembled the data and designed how the functionality would work, he may not be a developer per se, but he’s built the necessary XML files. Then he got someone else, a ‘real’ programmer, to help make the necessary changes to the World Wind code.

Scott: So, what are some of the advantages you see? I imagine it would be difficult for NASA to do something kind of closed source proprietary — just release the lib but not the source. But that said, what do you see as being some of the advantages, what are some of the strengths of building this as an open source?

Patrick: Actually, if someone wants to work with NASA and build something of an intellectual property nature, they are certainly able to do that.

There’s an application, The Morning Report, just recently licensed to the FAA that was built here at Ames Research Center. This application analyzes logs on commercial aircraft for anomalies that are not normally visible — not the expected anomalies, but anomalies that are very subtle. And this software analyzes for that.

The visualization tool for that is World Wind, so they just include World Wind with it, but the rest of the technology is very much a licensed intellectual property-based technology.

NASA has also developed things that remain intellectual property, that were developed internally. This can be done with NASA developing the licensable technology, and it can be done in concert with an outside entity who wants to partner or collaborate with NASA.

Scott: Talk a little bit about what you think some of the decision points are when an organization like NASA is looking at writing code. What are some of the decision points around whether it should be open source or not?

Patrick: Well, I think right now, NASA World Wind pretty much defines what should be open source — the ability to deliver pixels. NASA World Wind is visualization technology, it’s infrastructure.

NASA World Wind would like to increase the opportunity for others to continually take advantage of the latest video technology, which as you know, is constantly changing as video cards get more sophisticated. We want to optimize the delivery of a pixel, and leave it to others to provide the information intelligence that’s needed for people to understand ideas, tell stories or whatever.

I think the opportunities for taking advantage of this medium are pretty much wide open and unlimited. NASA World Wind really wants to stay out of that fray and just operate at the pixel delivery level.

Scott: At some point a technology becomes something that you want to build on top of. You expect that it just works — it’s sort of a piece of infrastructure, and not so much a piece of intellectual property anymore. In that kind of a case, a lot of times you see things become open source.

For example, at some point you just want an operating system, and you’re not really looking at building a lot of intellectual property into it. You just need something that will let you talk on the network, serve up files, run a web server, and so forth. A web server is another example where there doesn’t need to be a ton of IP. There doesn’t have to be a ton of IP in a web browser. And in your case there doesn’t have to be a ton of IP in serving pixels.

What you’re doing is you’re pushing people up the stack where you’re saying, "Instead of inventing your own rendering and instead of doing all this stuff, just consider it to be a piece of infrastructure and build on top of that." Right?

Your value isn’t going to be in rendering a pixel better. Your value is going to be on what is the data, what is the analysis, what’s the meaning that you’re trying to derive from this data? And the rendering of it isn’t probably where you’re going to move the world forward.

Patrick: Right. And we want something that is in a sense standardized, so that it’s a huge market that you’re dipping into. As you build that intelligence, anybody anywhere can run it because they have that basic, as you say, infrastructure.

Scott: Some of your funding comes from Congress, and you’ve mentioned too that Microsoft, and it looks like Sun are also providing corporate sponsorship. Talk a little bit about corporate sponsorship. First of all, are there other corporate sponsors? Do you have Intel and AMD doing processor optimizations? Things like that. And in general, what do you feel like the importance of corporate sponsorship is around open source products, especially those of this scope and complexity?

Patrick: Right now we’re benefiting from the support provided by corporate sponsors — Microsoft, AMD, Sun, Isilon, C-Map and others. In each of these cases, mostly it’s been hardware and consultation.

I-Cubed, www.i3.com, recently provided NASA with a Landsat set for planet Earth of Landsat 7 — the best I have ever seen. Normally, Landsat imagery is a mosaic of different days — the most cloudless day possible — and so, its imagery taken at different times of the year. You zoom in on Earth and you see what looks like assembled puzzle pieces and this can take away from the experience.

What i-cubed did was to put into the public domain an extraordinarily beautiful and color-balanced Landsat dataset that they had developed in concert with Isilon. Now, that type of gift, which normally is worth a quarter million dollars just for internal use, is now available for the world to share. This gives the world a wonderful base-level set of imagery for Earth.

The government has begun to recognize the value of open source visualization technology, which is why these different federal agencies are now supporting us. Open source visualization technology allows them to build all kinds of specialized functionalities. They’re able to take advantage of the world community pushing this technology, evolving this technology, advancing this technology, and they really appreciate benefiting from that opportunity. They also appreciate the synergy of being able to easily share these developments between federal agencies.

There’s a real benefit to the government in terms not only of cross-agency ability to communicate on a standards-based visualization platform, but there’s also the ability to save lots of money. Because if they have a need, they can either build it internally or they can spec it out then put it out to bid and say, "Who’s going to build this for us?" So, it saves the U.S. taxpayer a huge amount of money as well as making the government more efficient and more effective.

The open source model is being appreciated for its value, not only to government, but for its ability to stimulate new ideas in the marketplace as well.

Scott: Patrick, thanks for taking the time to chat with us.

Bookmark this:

Interviewee: Marc Miller

In this interview, we spoke with Marc Miller about his views on the current state of open source software.  Marc works for Advanced Micro Devices (AMD), and in January, Marc took on a role as the open source software evangelist in the AMD Developer Outreach organization enabling Linux kernel and application developers to develop optimized code using both AMD and 3rd party tools and resources. In his role as a software Alliance Manager for AMD 2001-2006, Mr. Miller played a significant role in developing a Linux marketing strategy with a focus on integration of AMD technology with software tools developed by the open source community and industry partners. Throughout his career at AMD, Marc has been a key contact for open source developers wishing to work with AMD, and has been an open source ambassador for AMD, helping to coordinate outbound and inbound communication between AMD and Linux developers.

In this interview Marc talks about:

• Challenges that open source companies have when competing with proprietary software
• Advantages of open source
• Choice also has disadvantages
• Is open-source anti-capitalism?
• The PR value of being open-source
• Open-source is less idealistic and more pragmatic today
• The future is blending open-source and proprietary software

Richard: First off, I have to ask what you think about the idea of Microsoft funding a blog that looks into the differences in delivery model between open source and closed source. Do you think it’s possible that that blog would be unbiased? I mean, we’re doing our best to make it that way, but I’m wondering if you think we can do it.

Marc: There’s a lot of suspicion right now in the open source community. In the aftermath of the Novell-Microsoft pact, it appears that nobody quite understands what Microsoft is up to and what their game plan is. Certainly the community is extremely skeptical about anything that Novell or Microsoft does at this point based on past experiences, for example with the Samba and Mono communities.

That said, while a blog, focused on open source and sponsored by Microsoft would be treated with skepticism by certain segments of the open source community, there would also be people interested in reading it. Everybody wants to understand Microsoft’s angle on Linux.

Richard: I figured that. I figured there’s going to be healthy skepticism.

Sean: When you speak, Marc, about what the angle is, are you asking what Microsoft’s angle is with Novell?

Marc: Yeah, and the way you look at it will depend on who you are. If you’re a large company like AMD that values both their Microsoft and Novell relationships, we’d be looking at that blog for opportunities where our customers can benefit from the marriage of the two.

One of the key things about the Novell and Microsoft pact was Steve Ballmer getting up on a box and saying that this is to protect Novell’s customers from being sued for use of Microsoft patented IP. That has a lot of people very frightened.

For example, people will be looking in those blogs for what the caution areas are. What products might I get burned on if a court decides Microsoft’s claims on software IP are legally defendable?

Richard: That’s interesting. So, with that in mind, I have a few basic open source questions that I hope you’re willing to address and give us your opinion. One is, what are the challenges an open source company faces when competing with closed source companies?

Marc: Very often the open source companies are trying to profit from something that’s not the source code itself.

Take a look at someone like Canonical. All of their software development efforts have been to get Ubuntu into the hands of as many users as possible. They make Ubuntu Linux available at no charge, under open source licensing mechanisms. Their profit model is currently based on support services. For someone like Canonical it’s a question of does the open source community develop products where the user might be willing to pay something for a service like support?

Richard: So, it seems to me that the biggest asset a software company has is the source code. What traditionally has been called trade secrets, intellectual property, or secret sauce.

Marc: In the closed source case, yes.

Richard: It’s an interesting idea to make money by giving your main product away.

Marc: The GPL completely allows for an open source program to be sold for profit. But it also allows multiple copies of that to be made and redistributed without royalties. So, it’s a real interesting area. Since it has been difficult to turn a profit from just selling a GPL application, traditionally most companies that back open source projects use the software itself as an enabler for whatever service or subscription that they plan to sell. In some cases these companies also sell a closed source product that compliments the open source projects.

Richard: So, what do you think are the most compelling advantages to open source over closed source? What is open source able to do much better than closed source can do, in your opinion?

Marc: Open source allows for freedom of innovation in the absence of a driving business case in a lot of cases. Take the Debian community, which has emphatically refused to adopt the same way of doing AMD64 architecture support the way that other Linux distributions have. To give you the ten-second synopsis, other Linux distributions support running 32-bit and 64-bit code in the same environment. Debian felt the way the directory structure was handled in other AMD64-supporting distributions was the wrong approach, and they felt they needed to adopt a structure that would prepare Debian Linux for future innovations. So they elected to support an approach that was different from what other Linux distributions were taking and refused the quick fix implemented for business reasons. I use this example to illustrate that, in the open source community, everybody has the freedom to innovate according to their own interest and ethics, business justification or not.  I respect the Debian community for holding to their values. In the closed source space it’s a much more restricted audience; the freedom to do what a particular developer feels is the right thing just isn’t there. People are still confined by ethics, but those can be overridden by corporate objectives and customer requirements.

For example, products sometimes are shipped before they’re truly ready just to stay on schedule. Debian is famous, if not notorious, for delaying a release until the software is really ready and no known critical bugs exist. If a bug is missed prior to release, open source methods allow opportunities for the community to fix the bug after the fact.

In open source software, anyone can disagree with a corporate direction and then redistribute their own version which has things written the way that they think it ought to be. This promotes a freedom to innovate that goes beyond what you can often find in the closed source model.

Richard: How does open source keep from having endless slightly different distros?  Is it Darwinian? The things that really work and get picked up live on, and those that don’t just sort of peter out?

Marc: The biggest strength and the biggest weakness of open source is the freedom of choice. Michael Dell has been quoted in the press prior to the May 1st announcement that they were going to start shipping Ubuntu. Each time he’s quoted about responding to when are you going to start shipping Linux, when are you going start supporting the community, he says, "I’m fine with supporting the community. Which one? Which one do I pick?"

Every Linux distribution has a different philosophy. Look at Red Hat; they have had a very long history of preserving backward compatibility and ensuring stability of the products such that when their enterprise customer upgrades, they’re not going to have to recompile their kernel and reinstall a bunch of apps.

Look at the SUSE case; they try to take a much more balanced approach to include innovation in cases where it might be a minor inconvenience, but they believe that the customer will appreciate the value that they get out of the product for that minor inconvenience.

Now look at Debian; I once suggested that Debian try to encourage enterprise ISVs like Oracle to integrate their software with Debian Linux as a way to expand their visibility and market share. A Debian developer responded, "Who cares about market share? What we care about is open source." Everyone on that discussion list thread agreed there’s no reason to make Debian more compatible with Oracle databases because Oracle isn’t open source. Debian’s mission is an open source revolution across all software segments.

So each Linux distribution, besides having a different code base, has a different philosophy behind it. And that creates a lot of choices for the user and that’s a big strength, but that choice complicates the decision too. There are two main product lines in the Windows release – a desktop/workstation line, and a server line of products, and one can even argue that those two product lines are more similar than dissimilar as they leverage many of the same operating system components. There are 359 open source operating system distributions according to Distrowatch, most of which are Linux.

Richard: It seems to me in doing research on this subject, I run into an awful lot of open source advocates who seem to be open source advocates more because of the anti-capitalism sort of flavor to it, at least in some corners of open source.

That seems to be a common voice, although it doesn’t seem to be a common motivation of all of the open source projects I’ve looked at. It looks like there’s an awful lot of people writing open source trying to figure out a way to make money on it. They believe in open source for maybe collaborative reasons, maybe for development cost reasons or whatever, but they want to make a profit.

But this anti-capitalist message seems to be sort of pervasive in one from or another through almost everything I’ve read. Do you think that that hurts the open source community in the eyes of people who are agnostic in terms of development paradigm? Do you think some people are turned off to open source because they see it as sort of anti-business, and they aren’t?

Marc: The attitudes of the open source community have changed over the years. If you look at the Linux community of the late ’90s, you’ll find that’s very much the case, to the point when we launched x86-64.org I asked that all of the corporate logos be removed from the website. Since then we’ve brought them back because the open source developers realized that without a corporate motivation and corporate sponsors that there’s a lot that they’re prevented from doing. Many open source applications are hosted and mirrored on either boxes in a corporation’s data center or using corporate sponsorship money. Linux Symposium wouldn’t happen without corporate sponsorship. Linux-supporting companies look after the interests of the open source communities in standards bodies and industry forums that developers don’t usually have access to.

Also, a lot of corporate sponsors pay developers to do what they would do anyway. And those developers have become grateful to those corporations in a way that has lessened the anti-capitalism trend.

Sean: Let me ask you a follow-up to that, and I know exactly why Richard called it an anti-capitalism bent, but do you see that as the most appropriate phrasing for it? I mean that’s kind of the quick draw phrase that everybody uses, but is it anti-capitalism or is it anti something else?

Marc: I’d say it started off as anti-capitalism because the feeling was that most corporations A) produced strictly closed source software and B) didn’t understand the open source model well enough to be any good to what people like Richard Stallman were trying to promote. So, it was an anti-capitalism setup.

But as I say, modern-day, most of these well known developers are sponsored by a business. So, a lot of that trend has died down.

Sean: So what do you think it’s kind of evolved into? Is there still kind of an anti- bent to the whole community? I mean is it kind of anti-control, or anti-throttling of innovation, or what would you stick there?

Marc: The Linux community is a lot more pragmatic about software IP than they were 15 years ago but long term, open source enthusiasts still are driving toward elimination of software as a proprietary product. It’s a freedom of knowledge movement, and much in the same way that you can go to Wikipedia and read about any topic that someone has contributed knowledge to. In the software case, that freedom of information is creating software instead of encyclopedia articles, and the content creates a new market. You can still make money off software creations, without making money from the software itself. Perhaps that’s services for the software, perhaps it’s a subscription to access premium content.

Open source enthusiasts feel that they are resisting controls over innovation and strive to obtain freedom to innovate in a way that’s beneficial to the customer, regardless of business case. It’s not anti-business but does challenge many existing business models.

Richard: In other soft product areas, like music, publishing, and stuff like that, you have a war between the copyright holders and those who think that stuff should be free. Do you think that closed source companies have a right to protect their source, or do you think somehow they’re subverting the greater good?

Marc: My opinion is that they have every right to do with their products whatever they need to do. If the software author also supports an open source movement, that’s a great thing but the open source model is not for everyone. In Eric Raymond’s "Cathedral and the Bazaar," there’s a whole section about reasons to open source, and that if your product does not meet all those criteria, then maybe you are better off keeping it closed. Often a solution stack is incomplete without certain pieces of software that happen to be proprietary.

For example, people buy a solution stack that includes Oracle and SAP because they derive value from the closed source applications, not because it’s built on top of an open source infrastructure. The open source operating systems (such as Linux) offer great features too, but today, most OS licensing and services revenue is driven by the application workload inherent in the solution stack the OS was bundled with, not necessarily the OS or benefits of open source.

Richard: So, you don’t think open source is in opposition to closed source, but rather just a different paradigm?

Marc: Yes.

Richard: OK. That’s cool; that’s interesting to know. I was trying to run down project-development leads on OpenOffice.org, and I think a little bit over half of them are employed by Sun. I’m curious, speaking non-altruistically, what does a company gain by sponsoring open source?

Marc: Besides the points I’ve already covered, they gain favoritism as an open-source-friendly company, that developers will even develop for their closed products because they believe they are contributing to the greater-good effort.

Richard: So, it’s sort of developer PR at some level? I’m sure not all of it, but there’s some, "We’re good guys, look, we’re sponsoring."

Marc: From a business perspective, that’s right – that’s what many of them see. The ability to leverage innovations from the community is also attractive, but the contributions aren’t guaranteed in the same way that the developer PR is.

Richard: That’s interesting.

Sean: For about seven years I’ve been involved in deep, technical efforts around evangelism, and I’ve certainly seen closed source companies realize and economic gain from sponsoring and supporting a community.

For a company that’s got a mainly closed source model, and they embrace open source model on the side, do you have any kind of anecdotes or examples of places where the company looked back and saw gain?

Marc: Probably the closest thing that I can think of is Adobe’s decision to allow projects like OpenOffice.org to include PDF generating tools, and their decision to make PDF a completely open specification that they encourage other people to put into their products. This has benefited Adobe’s PDF software products, even though Adobe’s products are closed. So, by creating an open standard, they’ve enlarged the market for themselves, and by supporting open source efforts, they’ve gotten a lot of mileage in the market that they sell into.

Richard: Last week Adobe open-sourced Flex, which was a free product. What, do you suppose, they gain by open sourcing it? Do they gain lower development costs, because they’re going to get free collaboration?

Marc: Well, I’m not very familiar with Flex, but suppose that they did the same thing with Acrobat Reader, which is another free product that they give away. They have already developed the market to the point that PDF is ubiquitous. If you want to make a document that is guaranteed to be readable on just about any platform out there, whether it’s x86, Apple, or a smart phone, you generate it in PDF.

If they decided to make an open source Acrobat Reader, that allows a lot of projects to tap into it and borrow code from it; it promotes the whole integration story of plugins and addons that go beyond the original plugin interface for the application. By giving their developer community the freedom to innovate, the development direction would be managed by the community.

You asked about lower development costs. The cost only goes down measurably when the community contributes something that the “copyleft” holder was going to pay for. Open source software lets a lot more innovators in the door to help with software development, but sometimes additional resources have to be found to do the work that no one contributes for free. The point that the open source community is trying to make is that open source innovators are shared among many open source projects. It would be expensive to recruit each of those developers under contract, and even more expensive to maintain that software over a long period of time.

Indeed, many projects have benefited from the open source model. Open source is not a guarantee of success any more than having great technology. There has to be demand for it, and from the right audience or no one will show up to do the work.

Richard: I see. A couple weeks ago, I ran into the CSI company that was started by Stuart Cohen. It’s sort of open source. The paradigm is that CSI hires developers, and managers as part of a geographically centered development teams. Companies who want a particular piece of software pay for CSI to manage that effort.

The idea is you might have half a dozen companies who want to have some software package. They pay CSI to develop it, and they also may make some of the developers on their staff available to the management of CSI to contribute to that project.

However, the thing that interested me the most is that was that once the software package is developed, it may not be re-licensed as open source. Rather, it’s going to be the decision of the contributing companies how to re-license, or even whether to re-license it. The presumption, I guess, is that they’ll go ahead and let it be available through some sort of open source model, where it can be modified and redistributed without cost, but they don’t have to.

It looks like a blending of open source and closed source. You’ve got this shared cost and collaboration of open source, but you don’t necessarily have the other main underpinning of open source, which is that the source code is available to everybody.

Do you think that that’s where we’re going, into different sorts of paradigms that blend proprietary and open source methodologies, and do you think that’s good or bad?

Marc: Though they would prefer everything to be open source, open source developers are beginning to accept that there are certain things that are difficult for a company to expose without getting them into potential legal trouble. They don’t like it, but they’re understanding. Overall, I think it’s a good thing.

Richard: So, it sounds like open source is drifting away from highly idealistic to more pragmatic; still holding onto their ideals, but being more pragmatic about attainability in certain situations.

Marc: There are some exceptions, but for the bulk of the open source community, yes, they’re becoming a lot more pragmatic about things. They still prefer to hold onto certain ideals, and continue to direct everything in that direction, but most developers are fairly pragmatic about it.

Richard: So, I had a couple of interviews with some guys at Microsoft focused on ensuring security for Microsoft products: Michael Howard and James Whittaker. Both of them, separately, said that the biggest advantage that closed source has, in almost every area—architecture, implementation, quality assurance, and documentation—is the ability to mandate a process. Microsoft has the ability to say, "This is part of the job, and you have to do it," whether it’s unit testing, adhering to coding standards, security reviews, or whatever. The implication was that would be hard to put together with open source because all you have is the code, you aren’t certain of the steps the developer went through to produce that code.

Is there good adherence to process in open source projects? Is it something that’s improving? Can you comment on that area, in general?

Marc: The open source community has code control procedures and models of their own. For example, in the LinuxBIOS/OpenBIOS project, you can submit a patch to the mailing list only once you’ve gotten one of the key maintainers to review and approve the code. So, there are still very much the rules for developing that code and processes to be followed. Some of these methods probably emerged from source code control models originally used in closed source settings, but in the spirit of open source, why change or reinvent what’s already working well?

The rules for submission seem to vary a lot for each project. You can have quite a range of different quality control mechanisms. But, usually, there is a quality control mechanism there.

Richard: Yes, there has to be. You can’t release software unless you have some kind of quality control. It seems like you have to have some sort of gatekeeper over the code base, or it’s going to deteriorate in a hurry.

Marc: Without someone moderating the process, you end up with wars of someone changing a bit of code, and another person changing it back. Endless back and forth. Code control also guides the direction of development so that community goals can be met.

Richard: Yes. So, what to you see on the horizon? What do you think is in the near future, generally, in open source?

Marc: A lot more of, as you put it, the blending areas particularly as Linux becomes more widely used in different server and desktop environments. For example, in multi media areas we’re seeing a lot of companies that create set top boxes that use a combination of closed source and open source software in embedded devices. It’s an area where businesses are realizing there’s money to be made with the combination of open source software and closed source IP.

Virtualization is a big area of expansion. A lot of companies want to make money off of virtualization, sometimes for security, and sometimes it’s to increase the range of compatibility across operating system environments. There are both closed and open source options, such as KVM, VMWare, and Xen. Each innovates using different philosophies. So, this is an area that I’m watching with interest.

Device drivers are also an area of much attention. The Linux kernel community has been very emphatic that they will not accept any closed source code into the Linux kernel. And yet the way that you develop drivers for Linux is to include it in the kernel.  For example, the FCC has given 802.11 wireless networking devices a certain range of frequencies that may be used. Many of those devices are essentially programmable radios. The hardware vendors don’t want to open source that because that allows someone to then misuse the product and gain access to frequencies that the FCC wouldn’t approve of. Sometimes in order to reveal specifications, licensed IP used in the hardware design has to be revealed.

The drivers I’ve just described can never be part of the Linux kernel, and yet Linux users want to be able to use all these devices. In recent months, just really January of this year, certain key people in the Linux community are starting to realize that this prevents a lot of hardware companies from participating in the way that they would like to, and that though hardware vendors have nothing against open source models, for one reason or another they simply cannot release their code. I’ve been watching a number of efforts to make it easier for hardware developers to produce Linux drivers. I hope this topic is discussed during the next USENIX Kernel Summit and that the maintainers find a way to prevent Linux from being stifled further on the basis of compatibility problems with new technologies.

Richard: Do you see Linux making inroads into the desktop market? Do you think that they can gain a significant part of that?

Marc: Client computing models are changing from traditional desktop and laptops, and with these changes there are new opportunities for Linux. Different types of desktop applications or use models also offer an opportunity for Linux.

I think in time we will see Linux emerge in the same way – having strengths in certain types of client devices. I don’t know what that strength is going to be yet for Linux but many people in the community are exploring what it should be. Today, most ISVs only want to deal with one desktop operating system, and that’s Windows.

Today there are a number of ways, including virtualization, that you can run Windows apps in a Linux environment. As that improves, I think we’ll see a lot of ISVs soften to that idea. And, as they look at some of the things that we’re developing for Linux, they may decide that it’s better to develop a Linux app that will also run under Windows than it is the other way around.

As the Novell-Microsoft Interoperability Lab continues to grow their efforts I can definitely envision Microsoft having to run Linux apps inside their environments much the same way that we have those today in Linux.

Richard: It seems to me that products like OpenOffice.org and other open source products that are becoming strong, mature applications in the traditional desktop applications space have to work to mitigate Microsoft’s grip on that market. In the past, when Windows first started to take over the desktop, it was because that’s where all the applications were. You could put Linux or something else on there, OS2 or whatever, but all the mature applications were Microsoft applications, and they had to run on Windows. And I think that’s sort of how they got that grip on the desktop market. It seems like the first step toward making that a more competitive market is through the applications phase.

Marc: Yeah, I agree with that.

Scott: It looks like the TCO battle has been fought to a stalemate.  Microsoft has Microsoft sponsored reports saying they have lower TCO.  On the Linux side, there are reports sponsored by Linux supporters saying that Linux has lower TCO.  Do the communities find any of these reports credible?  Have you come across any truly independent TCO reports that you’ve thought were especially good?  Is it even reasonable to expect that the finding of some TCO report will be borne out in the specifics of your organization?

Marc: In the independent studies I’ve read, per machine, the costs are pretty much the same for a supported machine if you compare Windows to a truly equivalent Linux implementation, but this fails to take into account the workload processing power costs. In the open source model, I have visibility into what code is slowing down my e.g. mail server, exclude the things I don’t need to have running, and make the machine more efficient. Without that optimization, more machines would be required for the same workload regardless of the operating system in use, which adds to the cost.

Richard: Marc.  Thanks for taking the time to chat with us.

Marc: Thank you.

Bookmark this: