How Software is Built » process

Interview with Britten Martin – Group Manager – Customer Service and Support – Microsoft

campsean — Fri, 21 Dec 2007 01:24:34 +0000

Interviewers: Scott Swigart and Sean Campbell

In this interview with Britten Martin we talked to him about:

Scott: Britten if you could just introduce yourself for the record. That is usually a good place to start, and then I’ve got an initial question after that.

Britten: Cool. So, my name is Britten Martin, I am a group manager within CSS which is Customer Service and Support. I’ve been with Microsoft since 1999 where I came in as a support engineer supporting Windows 2000. So I have been a support engineer, I have done some work with our outsource partners in a partner technical lead role, then I moved into frontline management and did that for about three years. In my current role I am a manager of managers, I have an organization of about 75 people and I am the global sponsor for System Center Support, which is your SMS and MOM products.

Scott: So talk a little bit about what falls under the scope of support inside of Microsoft. I mean, engineering rather has its domain, marketing has its domain: what is support defined as and what are kind of the responsibilities roll up under that inside Microsoft?

Britten: That is a good question. It really depends on which segment of support. It is a great clarifying question, Scott, because I am within Commercial Support which is the enterprise-based support, your back‑end, your servers, many of your big customers that buy dedicated support contracts, the government, many of the major companies around the world. We also have certainly have a consumer end which is your home user running Vista, Office, also Xbox, Zune. Any product that you can think of that is released by Microsoft has support. So really what it is, the thing that separates, I think, Microsoft from other companies in that, is that no matter what’s out there, you can pick up the phone, call 1‑800‑MICROSOFT, tell them what your product is, and you’ll get through to someone to support your issue. Although it certainly depends on the customer segment that you are in, who you are routed to, where you go and how much it costs.

Scott: Sure.

Britten: In Commerical Support, we support customers who have technical how-to questions about the products and certainly those who are getting error messages.† Also, if you need help setting it up, if you need more consultative type services, we do that as well.

Scott: OK. Then one other thing I was somewhat curious about too is ‑‑ one thing I have always been curious about I guess – is how a problem works its way through the cycle.

Britten: OK.

: So, you start out when the phone’s ringing and you have one customer, two customers, and three customers: they seem to be having a similar issue. How does it work its way through the process from the technician helping them on the phone, to maybe later on deciding it needs a KB on it, to maybe a hot fix, to maybe being rolled up in a Service Pack?

Britten: That is a good question too. OK. I will speak in today’s terms because major things have shifted over the last couple of years. There used to be a lot of process behind just what you were saying, which is, you know: "When do we do a KB? How many issues before we make a hotfix?" Nowadays with the advent of blogs, which is kind of, what you are doing, you know, what this interview is about, in support we have become a lot more proactive with getting immediate information out to our customers. I will give you an example from my group, within System Center: last week we had an issue with SCCM, the new version of SMS. Just like you described: one call, two calls, three calls. All of a sudden we are like, "Wait a minute, there’s something going on here!"

So we got some information together, dumped it out into our blog so it was out there. On the back‑end our support engineers, who are your front‑line resources, are working with the escalation team, our escalation engineers who do the debugging, who look at source code, who work with the product group. They are debugging the issue, and they get it down to, "OK, this is clearly a bug." At that point, that escalation engineer has a hotline per se to the product group, they pass along all of their research, they are reproducing the steps, reproduce the issue, send it on to them, all the documentation.

Then it is kind of in the product group’s hands to decide, "Is this something that we need to fix now. Do we need a hotfix, is it just something that we can put documentation out." You know kind of that form of decision making process. In this case it was a major issue, it was a deployment blocker, huge customer pain, so they released a hotfix. Then once again, depending on the urgency, is it a publically available hotfix or is it something that you have to call support to get a hold of? In this case, we knew this was a major issue so we made it available out on Microsoft.com, put a KB article out there, and linked the hotfix to it. Therefore, customers could immediately just go and download that hotfix.

Down the road certainly that’s going to be in the Service Pack 1 or similar, so you can pretty much figure for all of our major products, any bugs that we fix, or any hotfixes we release up until, there’s a cut‑off date, will make it into the Service Pack.

Scott: And then talk to me a little bit about the timeframe, right, between the time where maybe the first few calls came in, until the time that hotfix and that KB article are out there.

Britten: Well, once again it is very, very dependent upon the criticality and the widespread nature of it.

In this case from first call to KB/hotfix was about five days. In addition, some of those delays ‑‑ they had identified the code, they had written a hotfix within about two days, but there were some delays, you know, going through legal on official KB review, and getting everything up there and replicated. So from beginning to end, about five days.

Scott: Yeah, that is fast. I do not think most people would ever think of Microsoft turning something around that quickly.

Britten: Some times, I think Microsoft is seen as very slow to respond, but in reality, itís often not the case.† †If it is truly a deployment blocking issue or something that causes widespread customer pain you will see a quick turnaround on it and you will definitely see the documentation out there in a timely manner. I have been here eight years, I have seen issues where it was tough to figure out what the fix was, what the resolution was. Where it was something that would hide itself into multiple components so it was a complicated fix to write, it takes a little longer. However, if we can identify it, if we do a good job on the support side with identifying it, providing reproduction steps to the product group, you know, they can turn it around quickly.

Scott: So one question then too, would be, what does the triage process look like if after reproducing the bug you quickly realize that it has security implications versus something that’s a deployment blocker, which might be the next layer and then further down which are just kind of usability or general problems in using the software?

Britten: Yeah, I think, once again when you are running into security type issues what you get with that is even more resources thrown at the problem. We have security aliases internally, they have product group members, and they have certainly a high-level support presence as well, that if something comes out we can fire it up and quickly. There †is a whole process that is enabled. Let us say there is a new Sasser or Blaster or any of those. We have processes in place that will quickly call together members of support, product group. We have open lines 24 hours a day. It is kind of the war room type mentality.

It has product group members on it. It has support members. So people on the field can call in and say, "Look, I have got an issue with this bug or with this security hole and just want to make sure I have the up to date information". So they can call into this grid, they can get the information.

On the back‑end, it is hard to say how long it is going to take, depending on the fix but you are going to have even more resources available. You will see a fix as quickly as we can get it out there. After the Sasser/Blaster/Nimda pain, we really turned the corner not only on product development but also on the support side to ensure we could have a quick turnaround.

: What do you think the level of community involvement is in the support process because obviously that is a huge piece of the open source model. Right down from many eyes looking forsecurity bugs to a very well populated mailing list where there is a high discussion level, traffic, and things like that. However, at the same time with the closed source support process while the team is staffed potentially adequately with full time employees, there are general concerns out there with a closed source model in terms of how much the community can get involved.

So what are the avenues the community can actually affect the support
process at Microsoft and/or see what is happening under the covers of the support process?

Britten: Yeah, I will tell you that it has really been something we have been working on for the last few years. The product groups and the support teams have been trying to find ways to tap into that community model because there are many folks out there that do our support day in and day out with customers that have a lot of knowledge that could be helping the broader community.

So, one of the things that we have is the MVP program, the Microsoft Most Valuable Professional. One of the things that these folks do is to actively take part in the public newsgroups for our products. I can point to a couple of products ‑ Small Business Server is the poster child for that. It has a very strong community. Customers can go up to the Small Business Server forums and post problems and the MVPs will quickly get involved to help resolve the issue.

There is a lot of energy around building a stronger MVP community and empowering the MVPs to help solve customer issues. Many of the product groups have CPE, Customer Partner Experience, teams, that are really focused on tapping into that partner community.

Another example from my world System Center Essentials, or SCE. We actually launched support for that on a complete forum based model, where we had engineers from CSS monitoring the forum, and we worked to build strong partner participation. If the partner could not answer the question, or did not answer the question within 24 hours, someone from support would jump in. We are trying to grow more community presence around support.

Globally, there is more partner presence out there as well. A good example is SoftGrid or Softriciy, a recent acquisition. In Europe, before Microsoft took over, there was not really a ton of dedicated support. It was all done through the community. Therefore, we have been looking and trying to learn from them on how they involve the partners, and how we can keep that type of interaction going. I know there is a log of energy around this and it is a
huge initiative.

Sean: Well, one question which is really back to the second part of the original question which, is that a lot of the product groups in Microsoft are being affected by the open source model.† For example recently we talked to Shawn Burke who has been instrumental in getting the source code† for the framework out the door. How much of push to learn from the open source community is affecting the support team in general? For example could you see a day where the status on bug fixes is more open and less a black box until the actual fix arrives out on the web?

Britten: So thinking more about ‑‑ I am partner, I am out there. I find a bug or a hole, I put it out there, and really being more transparent if that is going to be fixed. Is that kind of…?

Sean: Yeah, exactly. I realize it’s not going to be a plate glass window ‑ it never is. But at the same time I think a lot of people will look at support right now and say, once I send in my support request I don’t really have a lot visibility of how they are stacking it, ranking it, prioritizing it, and where it goes, until maybe I get an email that says you have been deferred. Right?

So I am just curious, I know a lot of the product groups specifically are trying to expose a fair amount of their development processes, so I was curious if maybe that was making its way over to the support side.

Britten: You know it is a fair point. Let’s say you are running a beta version of the product and you are involved in the one of the official beta programs, such as TAP ñ our Technology Adoption Program.† †The beta version is fully supported and you can actually go and directly file a bug. In addition, I know that they try to make a lot of that process transparent, so you know if the bug is a duplicate. Is it being fixed? Is it being deferred to the Service Pack 1…whatever. So actually, I know on some levels we do that.

On the support side, I think we make good use of our blog. There is a huge blogging effort, because there is a lot of knowledge within our support engineers and within our internal databases. For one reason or another that has never been made public. So I think we try to do some of that as well. But I can see the point. I mean maybe someday and maybe that some day is not too far away where you can log in and say, here is a bug that I submitted. Alternatively, even better for Windows, here is the bin of things that are going to be fixed in the next two weeks, months, whatever.

Sean: Exactly

Scott: Well, and so you know not every support request anymore comes in through a phone call, right? There are things like the Connect site. There are things like Windows Error Reporting services. I guess if you would not mind talk for a minute about some of the different channels that, a problem can kind of come in through and… Go ahead.

Britten: No, no, it is fine. So today, and if I use Americas Commerical Support as a kind of a measuring stick, somewhere around 85% of our support issues come in via the phone ‑ very traditional means of operation. We are really trying to push more of our online type offerings. One is online submission where you can go up and create your issue; you can attach data, submit it to a support engineer. That support engineer reviews that data and calls you back, hopefully with an answer.

The Connect site is a lot more beta focused at this point. I am not an expert there, but that’s where you get into many of the cool things where you can actually submit bugs and all that. We also have the managed forums, which we talked a little bit about. The managed forums are a completely different avenue. Most of those are staffed by Microsoft support engineers as well as product group representation and the partner community.

So you have an issue with Windows, you can go up to a specific Windows forum, post a question. You might get an answer fromsomeone in the product group, a partner, or a support engineer. Therefore, you have those as well.

On the consumer side, they do really cool things with automated responses. Therefore, you submit an issue and it checks your data, checks your error log to see if it is a known issue. Before you would hear back from the support engineer, you might get an automated response with, you know, try these two things. If this does not work respond back and the engineer will contact you, then as you said, you had the whole Watson or error reporting. The error reporting stuff is very cool, because regardless of bugs filed or anything else that goes on, all the product groups take all of their Watson or their error reporting data and they make sure that as they look to roll out service packs, with new versions of products especially, they want to handle those top 15, top 20, top 50 Watson issues. They want to make sure that they are knocking those off.

Therefore, it is an interesting mix of many different avenues for customers to submit data, and in some cases, a Watson log, the error reporting, customers might not always think they are really contributing to the next version of the product, when in essence, they truly are.

: I realize, too, that you are right, that is largely a mechanism for when a product is in Beta. I guess, I have two questions, but one is a follow up on this thread initially. Talk a little bit, I guess; about how a product is supported differently while it is in beta, versus once, it has been released?

Britten: That is a good question. We have a few different types of beta opportunities for customers.† There are more formal relationships, such as TAP (Technology Adoption Program) and RDP (Rapid Deployment Program).† With these formal programs, customers agree to deploy this product into some type of production environment. In return, they get access to submit bugs, they get access to product group members, and they get
access to dedicated support engineers.

From within support, let us say we have a new version of Windows, we might have three or four support engineers in the US dedicated to this beta support. These engineers get a hold of the product early, they get access to product specs, they visit the product group, and they may even get to go out on site to customers who are deploying this product so they get some real world hands on experience.

Now, that is for the structured beta program. If you’re not a customer in one of these programs, you likely can still download beta copies of the software. Even in these situations, there is support out there, through the forums that we talked about before. Once again, you probably get answers from those same beta engineers and some of the product group folks. You just do not have that one-to-one interaction in a lot more, you know.

Does that make sense?

Scott: Yeah. Yes, very much so. One of the other things that ‑ I will kind of circle back to something you said earlier – If it needs code to solve the problem, you know, if it is not just configuration file that was messed up or any one of the myriad of things that could cause a problem. You know, some registry key got corrupted or whatever. However, it actually needs the coding hot fix, you could hand it over to the engineering, send it over to the product team and they triage it, compared to other stuff.

I mean, have you ever run into situations it seems like this would inevitably happen where they’ve got their engineers dedicated to building the next version of the product, and meanwhile, there’s an endless list of bugs in any complex software that they could be tasking engineering resources with. In addition, it seems that they kind of have to make that call about pulling people off engineering new features to work on a hotfix or something like that.

Britten: Yeah.

Scott: And how does that process of triage work? Because support is kind of on the front line with the customers, it seems like sometimes you guys might have to go back to them and say, "Um, you know the time frame that you’re looking at fixing this in probably isn’t really going to work, so think about it again."

Britten: Right. In addition, I will tell you, it is really dependent upon the product group. You look at something like Windows. Windows has a whole team, and you hear it from time to time called Win SE. ,Windows Sustained Engineering. They have a whole team dedicated to current version, and whatever supported versions are out there, on the hotfix front.

With some of the smaller products, you may run into situations where they have to pull developer resources off the next version to fix current version or legacy version problems. However, I tell you, in my time here, I have never seen that as being a blocker in getting something fixed. If there is truly an issue that needs to be resolved in the current version of the product, especially if its security related, you can be assured that they will find someone to handle it.

However, as a general rule of thumb, all big product groups have sustained engineering, folks that are dedicated to current version problems, or legacy version, so it is not necessarily that huge resource problem that you would think.

Scott: OK. Therefore, many times, they are not pulling people off features. They have people who are dedicated just to the maintenance and hot fixes.

Britten: Right. That is your sustained engineering folks, for the most part. As I said, I think you would run into some smaller product that might not be in that same boat, but the major ones have sustained engineering.

Scott: It seems, too, like support, it is a little bit of a difficult situation, because, I guarantee, if I call with any bug, it is a critical show‑stopper. The world will end if it does not get fixed. So how do you kind of manage customers, who are generally going to feel the importance of their bug is high, but Microsoft does not necessarily determine it so in the grand scheme of things?

Britten: Yeah. I tell you, our people are experts at conflict management and really setting customer expectations, and certainly the empathy piece as well. I mean, you have to be empathetic with the situation, but you also need to be very transparent and upfront.. It is saying, "Look, here’s where the bug is at this point. It is something that may or may not get fixed. Here’s the timeline."

With many of our customers in the Commercial space, these customers also have account managers that are Microsoft employees that really serve as the liaison between support and the customer. Therefore, we do a lot of work with the account managers to help us on the customer maintenance side of things, when maybe the messaging isn’t as good as we’d like it to be.

However, frankly, all major problems end up getting fixed. They might not get fixed in the period the customer would like to see it, but, between a hotfix, the next service pack, and the next version of the product, all issues do end up getting resolved.

Once again, it is really just dependent upon how many customers are having the same problem. Is it something that other customers could run into? In addition, really prioritizing, like you said before. I mean, the product group, while they have a sustained engineering team, has a limited number of resources, and they need to prioritize what are the big-ticket items that need to be worked on.

However, that skill of conflict management is one of the key pieces for successful support engineers, because, every day, when we answer the phone, when we respond to an email, that is generally that networking administratorís worst day of the year. The servers are down. They are catching heat from their management.† Something is going on, they are being yelled at. Our engineers deal with this on a day in and day out basis and as a result, they get very good at handling the heat.

: So one question I am curious about is, how do you deal with it when the support engineer knows that it is really a usability issue and not a bug, sure there is a sequence of phone calls and bug reports, but it is pretty clear that this isn’t a bug, it is more of an issue of usability. And it might be in the land of the undiscovered feature right? We were told, if I remember, some person in the office team told me that something like 90 percent of the feature requests for office, are already in the product, but nobody knows the feature is in there. So how do you people educate the product teams and what does that flow look like? How do you finally classify that, "OK this isn’t really a bug, it is an issue of usability?"

Britten: That is a good question.

Sean: Because obviously the product team, like everybody else is blind to what they build, right? Of course, it is useable. Because I built it, I mean every developer has fallen to that problem and we all have seen some pretty horrible UIs over the years. So how does the support team educate the product teams on this point?

Britten: Yeah. In the past, if you back up five years, 10 years, support was as you said an island. We were out here, taking all the heat and there was limited interaction with the product group. About five years ago, things drastically changed.† we started doing what is called supportability in our world. We review what are the top issues, how long does it take us to solve those issues and how much do they cost the company for us to support these issues?

Therefore, we came up with a list of the top 10 issues. We then present that back to the product group. Now the interesting thing about that is the product group actually funds us to do support for their product. So, when they see that a problem in their coding is †costing them $800,000. they to take some notice , and say wow OK, what does right look like, what are we missing here?

Sean: Well, just a quick follow up to that. What are some examples of things where you feel like you touched the product, and you made a real delta that the customers ended up bettering from, but at the same time you folks probably would never personally got credit for it

Britten: That is right. I donít have a list in front of me but DNS is one that consistently scored high for Windows 2000 and even some into Windows Server 2003.In addition, there was so much push and so much interaction that there were some major changes that happened later in 2003 and into 2008 that have drastically changed the way DNS works.

Another one I can give you that clearly had all the supportability people driving this was terminal server licensing. Terminal server licensing, when you go back to 2000 and once again, kind of into 2003 with a complete and utter nightmare for something that should be very simple. Consistently that silly process ended up top two, top three on our supportability list for Windows Server. And some very small tweaks based on advice provided by support completely changed this, dropped out of the top 10, out of the top 20 in support issues and saved the company a lot of money.

Those are two examples of technologies I know we beat on. However, on all the products out there, SQL, Exchange, Windows, SMS, MOM, they had the supportability groups that review the top 10 issues. In fact, I was just in Redmond last week, looking for the SMS side and they do cool things with it. I mean they look at top 10 issues for previous version of the product, look at the top 10 issues for current version, and review whether or not they fixed the top 10 call generateors? If not, what do we need to do to make sure it is in the next version that we do? Support is a huge data mine that we are just starting to realize the power of all the data that we have.† We are a great voice of the customer.

Sean: One quick follow up to that and then Iíll turn it back to Scott. I am curious because you mentioned that five years ago there was some type of a change, right? In addition, all of a sudden, you folks were a lot more integrated. What drove that? In addition, I realize with all changes ongoing, there is probably stuff you can say and stuff you cannot, but I am sure somebody reading the interview is going to be like, "Wait, wait, and ask him that." So what drove it at a high level?

Britten: I think that is fair. I wish I had some great and gory backroom drama to share, but if it happened, I just donít know the details.† In my opinion, you know, part of it was just the fact that, wow, especially on the Enterprise side, Windows Server 2000 started to really turn the corner and become a huge enterprise related product. People were out there buying it and implementing it and someone on the CSS side as well as, I can guarantee you, the Windows product group was like "We need to tap into this.î

Around this time, I was a support engineer and I suddenly realized that the product group was really listening.† In all businesses you started hearing about some monthly conference calls where we were there to provide feedback on top issues.

It was a time when we were moving from being just a consumer, home user company to being a legitimate player in the enterprise. In addition, there was a lot of pressure at that point to make sure these products were just better, and I think someone realized "Wow, CSS has a lot of knowledge out there; we need to tap into it."

Scott: Well, how much now do you guys get involved in product development from the standpoint of, you guys are looking at the product, while it is in development and saying, "You know what, if that error happens, there is no way we will able to trace it back and figure that out. We need instrumentation here, we are going to need these things, spit it out to a log file, we are going to need these kind of support features built into the product to be able to kind of quickly
and officially troubleshoot things."

Britten: That is a huge piece of our beta involvement. I know we talked about it before. For each beta, we dedicate engineers from our support teams to the pre-released product. Well, one of the way they make an impact is by providing that ongoing feedback to the product group. They are not only working with beta customers, they are not only answering newsgroups, but they are also messing with the product itself.

Therefore, they filed their own bugs. They have supportability war room meetings where they say, look there needs to be some logging here, or this error message does not make sense, or if this component is blowing up, I need something to trace it back. That is generally our involvement.

We do spec review. So many of our engineers have been on betas before, they are hooked in with the product group contacts. As the next version of the product is being planned, our engineers get involved to review specs, and just generally throughout the beta process from the beginning of our involvement through RTM, we are providing that ongoing supportability feedback.

So not only are we always supporting our customers, but we also look to help the product group on the supportability side. It is like you said, I mean, they are focused on here is a new component, here is a new thing that customers can use, maybe not always focused on how would I support that if it breaks?† Letís be honest Ö if we donít provide our feedback to the product group on what we need, it just causes more pain for us once the product is released.

Scott: One other one question is, so a new product is coming out and how do you get support folks up to speed on it? In the OpenSource model the development team in some cases is the support team and there is a tight integration between that team and the open source community so there isn’t really a need to educate the "support team" in this case. So how do the support engineers come up to speed on a given product?

Britten: This is another benefit of our involvement on the beta side.† †We get access to the source code. We get engineers playing with the product. We get that constant interaction with new builds of the product as it comes out. As a result, months, in some cases years, before the product actually ships, we have experts on this product.

Now, our experts certainly might be great technical writers, but we have a group called Global Technical Readiness. These folks are really program project managers who will say, "All right, we need Vista training for our support engineers." Therefore, they get their list of top issues. They get a list of new components, features, all of that. In addition, they start mining for data using these beta resources to come up with, "Here’s what engineers need to know. Here are the troubleshooting pieces; here are the top issues we have seen with our beta customers. Make sure engineers know this and that."

Therefore, over the course of this beta cycle, we are also in the background writing our own training. So these engineers write the training and as RTM approaches, depending on the training strategy, they might train all their engineers or a subset of the engineers. However, the knowledge is based off spec; it is based off beta engineer experiences, and beta engineer customer interaction.

In addition, the product group, in recent years, has really stepped up to help with training. They will review our training. They will provide insights. They will attend beta deliveries of the training and help the trainer get comfortable with the material and the product.

In some cases, the product group will also do a session of live meetings on, "Hey, here’s how we intended you to support the product. Here are the supportability features that we thought would be helpful." Therefore, you get a lot of interaction with the product group as well. From the beginning of the product to the end, not only are we helping customers on the beta side, but also our selfish win is to get the training that we need.

Scott: So, let me ask the flip side of that, right? So, you mentioned Configuration Manager. You’ve got System Center Configuration Manager 2007 coming out. Everybody wants to learn it…

Britten: Sure.

Scott: …because everybody is motivated. Have do you rather incent people to keep wanting to support the previous version when there is something new, shiny, and interesting.

Britten: At least on the Commercial stage, I do not know that I have ever really run into that problem.

Scott: OK.

Britten: Just simply because, in the enterprise customer segment, what you generally see is not the consumer model where a product releases, and everybody goes and gets the new version. However, it is a gradual uptake, usually. So, say Configuration Manager. Right now, and I can give you the specifics on this one, it is 10% of our total SMS volume. Therefore, engineers get a couple of SCCM cases, but they get the majority of SMS 2003.

In our minds, that is beautiful. Because what happens is, slowly over time they can build their SCCM knowledge while still providing that long held knowledge on SMS. They can help those customers resolve their issues. Therefore, it is a slow mix that generally happens. I do not know, it is ‑‑ they are tech junkies. They love the new stuff, but I guess maybe we all know that we have to pay our dues with the old stuff as well. [laughing] I do not know.

Scott: OK. That makes sense.

Britten: That’s a good question but it never really happens quite that way on the Enterprise side of support.

Sean: What it sounds like for you folks, is that from an enterprise standpoint, you can point to pretty clear statistics on customers who are significant accounts. Alternatively, you could point to customers who, for lack of a better phrase, probably spend a fair amount of revenue, right. In addition, it is probably easy to also look at the segment of the market and say, "We’re not going to give up 15% of SMS just because SMS, the latest version is out" ‑‑or in this case, obviously the rebranded version.

Britten: Right.

Sean: Because you folks can point to real metrics in terms of your percentage of the number of people who are putting in support requests and things like that.

Britten: Right.

Sean: Thanks for taking the time to talk with us Britten.

Britten: No problem.

Bookmark this:

Interview with Bob Bray – Geospatial Architect – Autodesk

campsean — Wed, 12 Sep 2007 17:50:21 +0000

Interviewers: Scott Swigart and Sean Campbell

Interviewee: Robert Bray

In this interview with Bob, Architect for Geospatial Products at Autodesk, we asked him about:

Sean: Bob, why don’t you give us some background on your role with Autodesk.

Robert: I’ve been in software development working for independent software vendors for about 17 years now. My current title is Architect for Geospatial Products. So, I’ve gone through the various ranks, from being a programmer in the trenches, in the cubes, to managing a product development team, to my current role as architect.

So, as I said, I’ve been through various things, mainly in closed source. Until roughly 18 months ago when we open sourced MapGuide. I think it was about six months before that that we really started heavily investigating that idea and started to put the ball in motion to actually make that happen. Basically, I’ve been involved in an open source project very actively for the last 18 months, and I won’t say that I was the key driver that made the decision that we were going to open source this, but I basically took it from that decision point and said, “OK, these are all the things we need to do…” and made that happen.

The product itself is MapGuide.The open source version is called MapGuide Open Source. The closed version is Autodesk MapGuide Enterprise. The codebases are the same. There’s one open source code stream for the whole product. There are a couple of minor functional differences between Enterprise and open source, but they are extremely minor, and we are trying to close as many of those as we can, actually. What we really have with the Enterprise version is basically a commercially tested and supported version. It’s much like the Red Hat model or the MySQL model, in that respect.

Sean: So, just out of curiosity then, if there are some functional differences, what led to those? I mean, they’re minor, like you said, but was some of that just by the nature of having to switch from closed source to open source? Is there some functionality that just couldn’t make it over that transform right away?

Robert: Basically, that is exactly it. We have a commercial desktop mapping product called AutoCAD Map 3D. We used some technology from that in the Enterprise version of MapGuide, and we do that for compatibility between the two products. So, those particular pieces, our coordinate system library, in particular, and our raster image handling library; those are things that we just could not open source at the time.

Sean:So, when Autodesk made the switch, did you model against an existing open source project in terms of how you wanted to handle QA or how you wanted to establish a community feel? You look at different products, like Subversion – I’ve heard they had one vote in the entire history of the project, right?

Robert: [laughs]

Sean: Literally. I watched their presentation at OSCON, and at least that’s the public face of things – that they had one vote. Or, for example, we talked to PostgresSQL yesterday, and they said, “It’s a very communal democracy” in terms of their decision process. They don’t really have a hierarchy. I know it varies across the projects, so did you look at the open source community and say, “Well, now that we’re going to open source this, we’d like to mimic this particular project” at least in some aspect?

Robert: Absolutely. So, when we made this decision, one of the first things I did was take a broad look across all of the projects. I was mainly focused on looking at the geospatial projects, because, if you will, our kin in the community is all of the open source projects – things like MapServer, GDAL, PostGIS… All those open source projects are very close in nature to us, so I certainly did look at those.

I also read a couple of books. The one I liked best was Karl Fogel’s book, “Producing Free Open Source Software.” Of course, he led the Subversion effort, but if you read that book you wouldn’t know there was only one vote in Subversion.

Sean: [laughs]

Robert: [laughs]

Sean: I was a little surprised, too. But, these were two guys from Google, so technically they’re reinforcing each other in their viewpoint, at least. It’s not just one guy saying it. They’re committers on the project…

Robert: I’m shocked.

Sean: Yeah, yeah. I was a little bit, too. But that’s part of the fun part of this investigation is some of the stuff we hear, and then we get to cycle back and say, “Really? So tell us a little more about that.”

Robert: So, we definitely took the approach of forming a community, and the project, and I say not from day one, but within six months, became a community driven effort. MapGuide has always been a product that, it’s almost like there’s a religious following of users out there. Even when it was a commercial product, it had some real hardcore, dedicated users.

So a handful of those users said, “Wow! This is going to be great!” They knew nothing about open source at the time either, but they kind of all jumped on board right away, started testing the open source version, and started getting active in that.

So, when we formed the community, basically, because we were modeling after MapServer and some of the other open source projects on the geospatial side and I really felt that was important we basically created a project steering committee for the product out of that group of people. So that project steering committee, right now, consists of three people from Autodesk and four people from outside in the community.

Sean: Are those people then the chief committers on the project?

Robert: Actually, that’s an interesting thing, because I would say there are a couple of people there that are active committers; the rest of them are users.

Sean: Got you. Because the interesting thing to me about this is that, obviously, a lot of these projects have grown up over time. PostgresSQL has a 20 year history, and to become a committer on their project, they told us, “Well, we wait two to three years to figure out if we can make you a committer, because we’ve been around for 20 years.”

But, if you birth a project out of the ether how do you determine who’s going to be a committer? Because, obviously, there’s probably a stampede of people at the start that say, “I’d like to be a committer, I want to be involved at the highest levels,” that kind of thing. How do you go through that winnowing out process?

Robert: There’s a URL I can send you, (http://mapguide.osgeo.org/psc.html) but basically what I did was draft a set of rules for the project steering committee to follow that include who can be a committer and how they become a committer.

We basically go through a similar process to, I think, Postgres – although we probably wouldn’t be two years in the road doing it. Basically, people start submitting some code from the open source community. They submit it to a developer, we’ll have that developer review it, and then do the actual commit. It’s almost in a patch style, right? Then, once we’re comfortable with them, we’ll vote to make them a committer – it’s a project steering committee vote.

Sean: Got you.

Robert: So, it’s kind of interesting, because, again, the project steering committee is not the core developers; again, it’s a core set of mainly users.

Sean: Once you transitioned over and said, “I want to make this an open source product, ” since you came mainly from a closed source background, what was the thing that most pleased you about being able to transition into an open source development model? What was the thing that you were, I guess, either most afraid of or most reluctant to see transition over? Because, obviously, the models are pretty different.

Robert: I think I was scared to death for quite a while…

Sean: [laughs]

Robert: …because it was completely unfamiliar territory to me. I’ll be honest, right? I’d never worked with an open source project before. I’ve used a few, but I’ve never really participated in an open source project. So, from my perspective, open source projects were these big free for all things, and I’m like, “Oh, my God. What are we getting into here?”

But, when I stepped back and actually looked at it, what I realized is that open source projects…yeah, they run a little differently than commercial projects, but they follow a fairly well-defined process; or at least the large projects do, that are successful. You can see a well established process there. It’s not a free for all, anybody can commit code, and a thing like that. So, that was my biggest fear going in; it was the step into the unknown, right?

Sean: What do you think was the biggest lesson learned from that? Now that you’ve got a couple of years at it, what would be the thing you might’ve done a little bit differently, given an opportunity? It’s apparent you wouldn’t have changed any decision to go open source, and that rings through clear. But, is there anything you would’ve changed during the process of going in that direction?

Robert: If anything, I would’ve been more open with the community right from the get go. I think I had a lot of reservations going into it, and so I didn’t really communicate as much as I should’ve right up front about what we were doing, how we were going about it, and all of that. So, I think I would’ve been much more open around communication right from the get go.

Sean: OK…

Robert: That’s the biggest lesson I think I learned.

Sean: …then what was kind of the biggest positive that you got? I didn’t want to leave that off the table either.

Robert: The biggest positive is that from a community perspective we are getting a lot more up front review; a lot more users using features before they go into, say, the commercial version or whatever, right? We’re getting a lot more early feedback on things that we’re thinking about doing or the ways we’re thinking about implementing new features, lots of early feedback that has a very, I would say, positive influence on the end result of the code that we actually write.

Sean:Well, I’m curious, too. That’s a functional change in the product, but have there been any change to the engineering process on the closed source side of the product? So, you guys open source it, that obviously rattles around and after a couple of years; I’m wondering if that has had any kind of change in the way you guys have developed internally, just having an open source mirror of the project, so to speak.

Robert: Well, it’s not an open source mirror. We only have one copy of the code, and that’s the public one. So, any change that we make to the code, whether it’s for the commercial release or for the open source release, typically gets vetted through the same process. We’ll go to request for comment. That’ll go out to the open source community, because it’s going into the open source version, one way or the other. So, we get early feedback on pretty much everything that goes in.

Of course some features that we may plan are for the enterprise version only, and those of course do not go through public review. Also we reserve the right to decide which features go into which enterprise release and we make no announcements about that because we are a public company right?

Sean: Right, right.

Robert: We don’t comment on that. We don’t do anything like that in the RFC process. But at the same time, the open source community certainly gets to vet all this stuff early. So, any code changes, and basically the whole product’s development process, are morphed into more of an open source development process for both the commercial and open source releases, because, again, we only have one copy of the code and we have one development process.The entire engineering team has basically transitioned to an open process, which is quite interesting.

Sean: Taking it down one level, as we talked to people sometimes as they’ve gone through this change, and the engineering team has a variety of differing reactions to it…various resistance points, or to put it maybe slightly more diplomatically, just things they have to learn as they go through the process. So, anything that would be worth talking about there in terms of lessons learned by the engineering team as they walked through this?

Robert: Yeah, I think there were a lot of lessons learned. Immediately it comes back around to being open about the kind of change we want to make and how you actually communicate with an open source community, because communication within the internal engineering team is quite different in my experience than communication with an open source community.

Sean: Now all of a sudden it’s a much more communal feel and development managers all of the sudden, I assume, had to make the switch from assuming everybody was on the payroll; there had to be a little bit more negotiation that affected the process of getting something developed, vetted, built, etc.

Robert: That’s definitely part of it. But also for our internal developers, I mean, they’re used to this kind of hierarchy of being able to walk down the hall and talk to somebody else. They’re used to being able to call a meeting and go in and start scribbling some stuff on a whiteboard.

With an open source development community that doesn’t really work anymore. You need to use the mailing lists a lot more. You need to use things like IRC for communications. So, it was really a complete change in the way you communicate.

Scott: So, that’s one of the things that I wanted to drill into because you’re obviously steeped in closed source development methodology. You’ve been in open source long enough to where you’re now immersed in that.
What do you see as the differences, kind of taking them side-by-side when you think about the aspects of the SDL, right? …how’re ideas envisioned, how they’re kind of tasked for somebody to code them, how they’re put through QA, how a product is released… If you don’t mind, I wouldn’t mind just kind of walking through how each of those is sort of fundamentally different in a closed source versus open source project.

Robert: Sure.

Scott: So, I guess starting with analysis and design and that sort of thing, how do you see analysis and design and requirements differing in open and closed source?

Robert: Sure. The requirements from a closed source perspective, I mean, we have project managers who basically go out and talk to our users. They’ll do focus groups. They’ll do surveys; all kinds of things like that just to basically pin down to a set of requirements for release of commercial software.

For open source software, you have a community of users who propose some ideas or a developer who proposes an idea. It gets kicked around on a mailing list for awhile. Eventually somebody either just picks it up and implements it, or the other option is they just walk in with something they’ve already written and say, “Hey, how about this? Isn’t this great?” So, the difference is really to me that it’s a little bit more organic, if you will, in the open source community. Whereas analysis and requirements and product definition is a much more rigid process in the closed source world, in the open source world it kind of happens organically.

Scott: So, where do you see those potentially? I’m guessing that both sides have their advantages. What do you see as the advantages of focus groups, meeting with customers, and project planning and what do you see as being the advantages of kind of more of the organic method of sort of showing up with a feature and saying, “What about this?”

Robert: If you show up with a feature and it’s already got a prototype or an implementation done – it’s a lot easier to talk about because you can actually see it and you can play with it. Whereas in the closed source side, you’re talking more in the abstract, if you will. So, the advantage in the open source side is really that you typically get a lot more users providing some feedback on it, at least if they’re really interested. Eventually you get to experience it before it actually goes in, which are both useful things.

On the closed source side, I mean, the advantage is that the release potentially comes out with a set of features that are much more…I won’t say well-defined because I think that’s the wrong term, but a release will typically come out that is much more cohesive as a whole. Instead of having a bunch of new things sprinkled about like a typical open source project has that’re mostly unrelated, a typical commercial product release will have the sort of features that are more or less a cohesive unit, if you know what I mean. Does that make sense?

Scott: Yeah, yeah. When I was a developer one of the problems with me kind of deciding what should be in the product was that I could have a project manager come along and say, “The product should really do X.” I would try to convince them that it shouldn’t, not because I had a better knowledge of the customer than them, but because I knew what a mess that would be to try to code given the existing architecture.

Do you feel that that kind of thing happens on an open source project because a lot of the people working on it kind of know what it would take to code it? Like, they might shy towards or away from certain things?

Robert: Yeah, you don’t see that in the open source world in my experience.

Scott: OK. So, ideas get proposed on the mailing list, things like that. You guys have internal developers who are still working on it. Is it that everybody sort of volunteers to work on different things or do your own internal developers get tasked with certain features? Obviously in the community it’s sort of all volunteers, so somebody external has to decide to pick something up and run with it. I guess, talk a little about how that works in terms of actually getting the work done.

Robert: So, it doesn’t work all that differently for us because we do have an enterprise version of the product and we do have a product manager who still does all those typical closed source things. What typically happens is that the features that we foresee as important to get into the product will basically go through a similar process as a typical closed source world, but then we transition them over to the open source world.

So, we’ll basically write up, you know… if we have an idea we’ll either code it first, or more likely we’ll write what we call a “request for comment” document that basically describes the change we want to make then actually put it out there to the open source community to vet. Again, because we’re sponsoring it, the things that have somebody behind them who’s actually going to write the code will typically get approved by the PSC.

But, the benefit that I’ve seen in that is that we get a lot of really good feedback, positive influence, and changes to those RFCs before they actually get coded because we have a pretty broad user community out there in open source basically providing us with some really solid feedback.

Scott: Then in terms of things like QA and documentation, how much of that kind of gets done by the community? How much of that do you guys still have to take on?

Robert: For the open source project we do no QA. The open source community is responsible for all QA. So, between the software development team itself and the open source community, that’s where all testing happens for the open source releases. We do not have any type of internal QA or formalized QA process on the open source project.
We’ll decide that we want to cut a release, and the project steering committee will release a beta. We’ll get some users testing it, providing us with bugs. We’ll fix those bugs.

Autodesk has traditionally done betas, but they’re fairly late in the game. I would like to see us actually release our betas a lot earlier than we currently do to get more of that community testing done earlier in the cycle. Then, we have more of a chance to fix bugs, and make changes before we do our final release. We’ll go through basically one or two betas and a couple of release candidates. We’ve done up to two. We may do a third on 1.2, and then we’ll release the code.

So, with open source there’s no formal QA process other than the unit test has to run successfully on every build. But basically, it’s users hammering it, whereas for the closed source project we have our rigid, internal QA process. We have our internal QA team that does a fairly formal quality assurance testing that you would expect from commercial software.

Scott: Do you feel that one way or the other is more effective at kind of wringing the bugs out of the product?

Robert: What my gut tells me right now is that a blend of the two is actually best. The nice thing about the rigid QA testing is that when you release the product, you know exactly which code paths have been tested and which ones have not. What you don’t get out of that typically is the kind of real world testing that a user will put it through. So, my current thinking is that a blend of the two approaches would be better.

Sean: Just a piece of context – how late is late for you guys? If the project has a 36-month development cycle, or 24 months, or whatever, from the initial release definition or spec process down to when you actually have people buying the product, how far along on that trajectory are you guys typically releasing betas?

Robert: In commercial software typically our first beta release is about four months before release, and that for us is late. Part of that, though, is because Autodesk has adopted an annual release cycle, so that has dictated shortened cycles for everything. But, if we could find a way to release those new features earlier for beta testers to test, I think that would be a huge improvement and a way to blend the best of the open source world with the closed source commercial QA testing.

Scott: One of the challenges people sometimes run into when they open source an existing project is that on day one when you open source it, all the jobs on the project are already filled. You already have a team of 10 or 100 or 500 or whatever number of people who are engineers employed working on the projects, and they’re already sort of doing everything. So, there isn’t necessarily a feature that’s just waiting for somebody in the community to write.

So, from this perspective that kind of caused a lot of consternation internally about what was that going to mean for the people whose job was to work on the project. It kind of created issues with the community on day one about OK, well great, what do you really want us to do? I mean, what’s really open for us to work on that you’re not already working on?
Did you guys run into anything like that on this? I’m interested in kind of comparing the experiences. You obviously have a different product, and I’m interested if you ran into similar things or not really.

Robert: I would say we ran into the exact same thing. In fact, community contribution I think has been slow. It was really slow in the first year for two reasons: one, it’s a fairly large and complex codebase. It’s not the easiest thing to jump into. Two, for that exact reason. We had developers that were doing everything, so why do I need to do anything?

That has changed, however, I think in the last…I think this last six months or so we’re starting to see much more significant community contribution. Again, MapGuide is more of an application development platform for web mapping than anything else. It’s not really a product you install and start using. You install it. You build an application. You display it to a bunch of users. You refine it. A typical web product, right? So, what we’ve seen is that a number of people have seen limitations with respect to what they can do particularly around the client end of the product. They want more flexibility in the AJAX client. So, they started with some simple submissions – really just adding some little functionality.

But, of late we’ve actually had a solutions organization that got heavily interested in MapGuide early; DM Solutions in Ottawa. Basically, what they did is they actually wrote a new client front end; a new AJAX front end for MapGuide. They are actually now in the process of donating that back to the project. So, that’s a very significant piece of code that grew out of a need. Once they started developing some applications, they thought, “Hey, I’ve got some limitations here. We could probably do something about this with our experience.” So, they started writing some code. Sooner or later they had deployed it a couple of times for a couple of their customers. Eventually they came back to the project and said, “We think this will be useful for everyone, so we’d like to donate this.”

So, I think again it’s a matter of time on those commercial projects that choose to open source. I think eventually the time will come when developers will come onboard. But it takes a little bit of time for that exact reason. Initially, what do we need to do? Don’t know.

Scott: Well, it sounds like you guys are doing it the right way. I mean, the other reason sometimes why companies open source something is just because they want to sunset it to some degree. They’re done with it. They’re tired of it. They don’t want to put people on maintaining it, so they just say, “Here, we open sourced it. Community – go take it from here.” Those ones usually just kind of die pretty much on the spot.

Robert: Well, for us there’s some interesting history here you guys should probably know that I maybe didn’t introduce early enough for you. MapGuide was first released as an Autodesk product in 1996-1997. About three years ago we took on a complete ground-up rewrite. We were about halfway through that effort that we decided to open source.

So, we basically have a legacy product there in MapGuide. We started a complete rewrite project. We actually completed that complete rewrite project. That’s what we open sourced. But, halfway through it we decided the right thing for the future of this product was to actually open source it and we intend to continue maintaining and being a huge part of the development community on that project, but at the same time getting some open source involvement.

Scott: Yeah, that seems to be key. It seems to be key when the company that’s open sourcing it is going to continue to invest in it – that really seems to be a key to success. You know, companies seem to open source for a lot of different reasons. Sometimes it’s because they feel like maybe they’re not in the best position to do a certain piece of work – like if it requires porting to a different platform or architecture – or they feel like the community will have great ideas that they won’t generate internally.

Sometimes companies do it as a market disruptor. They see that the market is saturated with closed source, proprietary solutions, and by being the first ones to open source they can radically change the market. For you guys, what was your motivation in going open source? Why did that make sense for this particular product?

Robert: Well, I’d be lying if I said market disruption wasn’t a huge factor, because it certainly was. What we recognized halfway through this project… I mean, again, it’s basically a piece of web mapping technology. It’s basically a web based GIS platform. So, what we realized about halfway through… We lifted our heads up and looked around. We saw that web-based GIS was growing in popularity by leaps and bounds and ESRI is pretty entrenched. They are a commercial competitor. All of our other competitors have products in this space as well.

Then you’ve got Microsoft. You’ve got Google and Google Maps. You have Yahoo! Maps. There’s a bit of a commoditization around web mapping that we discovered. What we really realized is that the revenue opportunities for us are not in the platform per se, but more in the solutions that we can deliver on the platform. Does that make sense?

Scott: Yeah. So, by open sourcing the platform and being still heavily invested in it, you’re in a great position to provide solutions on top of that platform that might be hard for your competitors to match.

Robert: Right, because we can influence the platform’s direction as well as direction of our solution providers. Again, we’re a company founded in establishing developer partnerships. It’s really partners that develop solutions on our product.

All of our partners now can influence the direction of the open source project by either: a) actively becoming involved in development on the platform; or, b) hiring companies that specialize in open source development to go enhance the platform in some ways that they never could when it was a closed source project.

Scott: So, how does it work? Has open sourcing it disrupted the market and provided a lot of the benefits that you guys were initially hoping it would?

Robert: It’s definitely been disruptive and that’s beneficial. The larger benefit for me, though, is the fact that our solution partners now have an opportunity to influence the new platform in ways they never have before. They get to see things coming early. Typically we release the open source version two or three times a year, so they are getting features faster and they can start to work with them.

On top of that, they can influence things. The idea of this whole AJAX front end is a great example of that. It’s where a solutions development partner for us basically came in and developed a whole new client framework that’s going back into the platform.

Scott: Have you guys at this point had to contend with anybody who has wanted to do a really strong fork of the source? How have you prevented that or dealt with that?

Robert: I don’t see anybody wanting to do a fork. It’s LGPL licensed. That helps a lot in that respect.
My crystal opinion on that, by the way, is that if you have a project that has a good community behind it, that is a cohesive community, I don’t think you’ll see a fork.

Scott: Yeah, it seems like there’s definitely a high bar. Forking is the nuclear option. For somebody to fork and have it get traction, they have to be making at least the same amount of investment in it as you guys are. That would be a pretty high risk thing to take on.

Robert: Right.

Scott: It seems like as long as the company is really meeting the needs of the community and supporting the community in the right way, it doesn’t really happen. But Netscape is an example of where it did and the outcome of that was Firefox. People really felt like the main sponsor was really not taking it in the right direction and they had a better idea.

Robert: Exactly. Let’s be honest – that’s one of the reasons that the project steering committee, who really do have control, is actually four community people and only three employees. We can always veto something because it only takes one vote to veto. But, in my experience so far, working with this committee in the last 18 months, I don’t see us ever actually using a veto.

Scott: Great.

Robert: In communities you can suddenly get a rift with the company that wants to maintain control, or sometimes you get a rift in the development team itself. Drupal is a good example of that. They had a rift in the middle of the development team, and that resulted in the fork of Drupal. I don’t think that’s a common thing.

Scott: Right.

Robert: If you have a good, well-oiled community, I don’t think that’s likely to happen in a company that’s open.

Scott: Great.

Scott: This was fantastic. This was really informative from our perspective. It’s nice to be able to talk to somebody who’s got so much experience in both realms and can provide some objective insight about their experiences on both sides. So, this was really great for me.

Sean: Yes, same here. And Bob, we’ve extended this courtesy to everybody. We’re just curious if there’s anything else you want to get on the record or another question you’d like to propose, or things along those lines before we wrap up?

Robert: No, I think the only thing we really didn’t get to was one of your questions earlier about QA and documentation; I didn’t touch on the documentation piece. I would say that’s one of our biggest weaknesses with open source right now was that we really don’t have a solution around documentation in the community right now.
That’s something that I think will still take time to formulate. Everybody is busy and nobody has time to write documentation. That’s actually my experience as one of the issues with open source projects.

Scott: Unless you’ve got a sponsor who staffs that, it’s just not the most fun stuff to work on.

Robert: Right. So for now, most of the open source documentation has been staffed by Autodesk. But that’s getting harder and harder for us to continue. So, that’s one of the places that I am, if anything, a little concerned about for the future; the documentation and trying to get a way to continue to staff it.

Scott: I don’t think that’s unique. Part of the problem is the people who really, really work on it, don’t really need the documentation. So, they don’t necessarily feel the need as much as the people who aren’t as active in the community itself.

Robert: Right. But the problem is that it’s hard to get new users on board when you don’t have good documentation. That’s really the challenge.

Scott: Right.

Sean: Well, Bob, I really want to thank you for taking the time. I couldn’t agree more that it was a great interview. It was really, really nice perspective we got.

Bookmark this:

Interview with Matt Gibbs – Development Manager – UI Framework and Services – Microsoft

campsean — Tue, 11 Sep 2007 21:40:20 +0000

Interviewers: Scott Swigart , Richard Bowler

Interviewee: Matt Gibbs

In this interview with Matt, we asked him about:

Scott Swigart: Hi, Matt. Before we start, go ahead and introduce yourself.

Matt:I’m Matt Gibbs, and I’m the Development Manager in the UI Framework and Services at Microsoft. My team is responsible for ASP.NET and AJAX as well as parts of Silverlight.

Scott Swigart: Thanks! We’ve been commissioned to look into the premise that open source and closed source SDLs [Software Development Lifecycles] are fundamentally different, and that the differences manifest themselves in the final products in tangible ways.

That’s not to say that one is better than the other – certainly both models have their strengths and weaknesses – but it’s really designed to get information out there for two audiences: one is software vendors who are looking at building products. One of their first choices is whether they are going to go with an open source or a closed source model, and we want to look at some of the ramifications of that choice.

The second audience is people bringing products into their organizations, and we want to uncover some of the basic expectations that they could have around open and closed source, or some of the characteristics of each model that might impact their choices.

Richard Bowler: You’ve been the Development Manager for ASP.NET for about a year, right?

Matt: Yes, about a year.

Richard: What would you say are the major development challenges with that product?

Matt: I think for us, the biggest challenge is that we have a large customer base, so as we innovate, we have to be very sensitive to backwards compatibility. We have a fairly complex platform, so innovation is challenging when you’re trying to make sure that existing applications people have built with it continue working.

Richard: It’s a pretty complex project, and it’s built on another complex project, which is the CLR; to co-develop with the CLR team, do you just take what they give you and work from there, or how does that work?

Matt: We work really closely with them. In fact, we meet regularly to go through code changes we’re making with members of the CLR team, so that we’re all in sync.

Richard: It sounds like you’ve been able to direct feedback to them, saying, “We really need this kind of capability, let’s fold it into the next release,” or something like that.

Matt: Yes. And they’re pretty sensitive to how their changes impact us. They run compatibility tests and stress tests against our scenarios, as well.

Richard: It seems like a huge challenge, when you consider the number of software products that Microsoft has on the market that interoperate. It seems like a huge challenge to keep them all working together.

Matt: For the framework we’ve actually held a pretty strict line. I am part of a committee where we review things that were classified as “potentially breaking a developer’s code,” and the default was that those weren’t allowed – that was kind of the highest priority. Things like “security fixes” would be allowed. Then there always needs to be a way to get back to an earlier version’s behavior, through a configuration parameter, for example.

Richard: Speaking about security, we talked with Michael Howard, and also with James Whittaker about the SDL. It looks like it’s really paying off for you guys. Were you involved in the development internally when the SDL was being instituted at Microsoft?

Matt: Yes, actually I worked with Michael Howard on IIS where we ran into a few security issues that helped highlight the need for a more structured approach.

Richard: I know what the benefits are, because we discussed it with Michael and James. But what were the main difficulties in shoehorning those new processes over the top of existing SDL processes?

Matt: I don’t think it really came down to process difficulties, as much as it came down to just helping developers recognize that the mindset they had been using needed to be front-loaded with security all along.
In our first round, everybody recognized that we needed to pay even more attention to security, and make it a higher priority. But then the realization was that it can’t be one phase in the development process. It really needs to be something that we pay attention to all along.

Richard: To put it another way, then, you had to get a wrench on the culture. You had to make security important in everyone’s mind.

Matt: Right, and it really had to be the case that everybody came to the same conclusion. You couldn’t just tell people it was important – they had to see what kind of impact it had to customers, and recognize it themselves.

Richard: Did you get much internal resistance from individual developers, or was there pretty much across-the-board buy-in?

Matt: It was pretty wholesale – everyone recognized it. I think particularly since I’ve been involved in web platforms, and we saw a couple of places where we’ve made mistakes, everybody recognized, “OK, we have to do something different, and be more practiced in this,” because otherwise we’re not going to get the right outcome.

Richard: Where was ASP.NET in the process of instituting the SDL? Was Version 1 out before the SDL really came to the fore?

Matt: No, when the process was launched we were already in development. It was during that cycle where we decided to take a break, and we went back and said, “We need to invest time in this process.” Everybody did some extra security training, and then we essentially stopped development, and did a pass for just security. We looked at every feature, all the code, looking for specific issues, just to make sure we had done all the right things.

Scott: One of the things that I found looking at the open source side of things, is just the sheer number of distributions that they have to ship for a product. MySQL for example, I think has 16 distributions, because it has to run on all of these different flavors of Linux, none of which the people working on MySQL really have a lot of control over. Those versions of Linux are all rev’ing independently of each other, so the underlying operating system that it’s dependent on is changing all the time. You’ve got a whole ton of applications that are built on top of MySQL, and are therefore dependent on it.

I can’t even really imagine exactly how to test that, and I know that in the open source world there are issues about, “This version of MySQL isn’t compatible with this version of WordPress, which isn’t compatible with this version of Linux.” So, they are always fighting these compatibility issues as things just rev independently of each other all the time.
Microsoft has some rather similar challenges, in the sense that IIS revs with the server. ASP.NET revs on its own timeline, which isn’t tied to the base operating system, and Atlas is also on its own timeline to some degree.

Talk a little bit about the kind of test matrices that Microsoft builds, and the amount of resources in terms of hardware and testers that come to bear when you guys are getting ready to release something, or while you’re developing something to ensure compatibility with all the versions of all the things you ship.

Matt: It’s probably more expensive than you might initially think, because in the same way that there are a bunch of different Linux distributions, we also still test against a lot of different supported versions of the operating system.

We’ll test against XP and Windows 2000 and Win2k3, with different levels of the service packs and with components installed; with Visual Studio installed, without Visual Studio installed, and then different flavors of IIS. We’re already testing with IIS 7 on Vista where we’ve got a lot of integration work going on. When we go to do compatibility checking and functional testing, we have a fairly large lab. And the run time, unfortunately, is on the order of weeks to really ensure compatibility.

Scott: What I’m envisioning is that somebody pushes a big button, and a hundred machines get configured with tests. It all comes up and runs, and results get cranked out — test, pass, or fail. Is it something kind of like that?

Matt: Yes, and they categorize tests into different levels of priorities so they can say, “We’re going to do a quick sanity check on something,” and that can take a couple of days. Or, they can say, “We want to run the gauntlet, and run everything we’ve got,” like before a service pack or something, and that will take much longer.
It covers everything from different language packs, to different service packs installed. One time-consuming thing is that errors that are false positives come up, and they need to be investigated. These might be just test issues, and that takes time as well, because you don’t want to ship with something that might be a problem, so everything has to be investigated.

Scott: I know that Microsoft’s got certain terminology around this. I’ve heard of something called the BVT, which I think stands for Build Verification Test. There are things that have to happen before something gets released as a CTP, Community Tech Preview. Can you talk a little bit about that, in terms of what the different levels of testing are, and what kind of things have to get verified on those different levels?

Matt: There’s a scaled approach. On a developer’s machine, we’ll have what we typically call Developer Tests.
The developer cranks out a lot of small standalone tests against the feature he’s building, against his APIs. He’ll run those every time he goes to check in code. We’ll aggregate those for our dev team, so that you’re running a fairly lengthy set of them. But, they still run within just a few minutes, so that you can continue making fast development progress. Then we’ll run a superset of those before we check in.

Scott: So, that’s to make sure that one developer’s change didn’t break some other developer’s code.

Matt: Yeah, that the basic functionality is intact. Then, the test team runs what they call a set of nightlies. They’ll run those on every daily build that comes out, they’ll kick off an automated set of nigthlies. They’re small enough in scope that they can run overnight, and any failures can be investigated fairly quickly. They are what they call “Pri0” test cases; and that would be mainstream use-cases.
Then, the superset of that is something that’s maybe ten times the number of test cases, and that’ll run for days to verify the full functionality. That’s literally thousands upon thousands of test cases.

Richard: So, your nightlies maintain a level of sanity about the ongoing nature of the code base – yesterday it was still good, today it’s still good, tomorrow somebody’s checking in something bad. Let’s back up and fix this so you never get too far out of whack.

Matt: Then, you may find something when you go through a full test pass. When the tester originally made the test plan, they may have said it’s a fringe scenario, so it was classified as a priority two test case. You might only run it before a release, and then something breaks.

Scott: Microsoft has a lot of people who blog, so there are obviously a lot of people on the product teams who are fairly transparent about where they are and what they’re doing. I think Scott Guthrie put up a big blog post where he said, “You know, there’s a point in time where we don’t necessarily fix every bug because we have to weigh the risk of impacting the stability of the product.”

Every time we touch a line of code, there’s a risk that we’re going to hurt the stability of the product, so there’s a point where we switch from telling a certain team what we’re fixing to actually having to get permission to fix certain things because of where we are at in the stabilization process. Am I characterizing that right?

Matt: Yeah, we’ve spent a lot of time mining our bug tracking database, and you look at data points like, “For X number of bugs, how many fixes actually cause some other problem?” The closer you get to shipping, the more likely you are to look at something and say, “This is really not a very important bug, and people are just not really going to see this. It’s good that we did the due diligence and found it, but this late in the game, you run the risk of breaking something more important.” You might just leave a bug in there and fix it when you have more time to shake out the problems.

Richard: I guess if you didn’t do that, you’d never ship.

Matt: Exactly. With new engineers, you’ll regularly see them hit this kind of frustrating point when they have a bug, they want to fix it, but it’s time to ship and they get told no. Their point of view is that it’s a problem that should be fixed, and they have to be told that they can fix it, but not until the next version

Scott: I have to admit that it’s odd to think that leaving bugs alone raises quality.

Richard: In my own process over time it seems like when we get down to the mode when we’re primarily testing an implemented product and engineers are working on bug fixes and not new features and behaviors. It seems like as you get closer and closer to when you need to ship it gets harder and harder to jump through the hoop of where a bug ought to be fixed. At some point it’s got to be an absolute showstopper to stop the shipment; it’s got to be a crash or something like that, otherwise you’re just not going to fix it until the next release.

Matt: Yeah, we get to the very end where we call it recall mode, where the question is whether you would actually take the product off the shelf in order to fix this issue. At this final point in the process, unless we’d recall the product, we don’t fix it.

Scott: Another thing that’s pretty interesting to me is the whole way that Microsoft does milestones. Again, my impression of it from the outside is that somebody will say, “OK, these are the features that we’re targeting for milestone one, for M1,” and so developers will go off and they’ll be working on those features. There will be a graph of the bug count that’s just climbing, climbing, climbing; the number of test cases written are just climbing, climbing, climbing.

At some point, it seems to switch over to where new features aren’t being written, it’s focusing on stabilization for the M1 milestone. So, there’s a trend down in the bug count. It seems like there are a lot of different variables that are being looked at in those stages; what’s happening to the bug count, what’s happening to the perf numbers…
Can you talk about how all that stuff comes together? Am I characterizing these milestones correctly? Is that sort of how it works?

Matt: In the past, that is how I would have characterized it – a sort of waterfall approach where we crank out a bunch of features and accumulate bugs, and the test team cranks out test cases, and then we play catch-up and try to satisfy all those test cases and bring the bug count down isn’t the most efficient. But we’ve actually shifted in our group to a little different approach that we think is working better.

Instead, we’ve gone to what we’re calling a feature crew model where the dev, test, and PM work together from the start on defining the design, the set of test cases, and the unit tests, and they drive for more quality up front before the feature even gets checked into the product. So, they’ll work in an isolated source code branch getting a feature to a higher level of quality and then get it into the product.

Richard: It sounds like you’re moving, at least to some extent, toward a more Agile model.

Matt: Yeah, it’s ended up being a hybrid of Agile development with features of SCRUM, but you couldn’t really call it one or the other. It still falls into a sort of waterfall effect where we have a milestone of a bunch of features we’re going to try to get done by a certain time. There is still this mentality of feature crews driving for a deadline, so we picked the pieces of a bunch of different methodologies that seem to work well for us in delivering commercial software, and we are trying to make them work together.

Scott: Now, I’ve heard a little bit about these feature crews. Correct me if I’m wrong, but that seems to have been developed after the ship of Visual Studio 2005. It came out of … I think they called it like, MQ or Milestone Quality, or something like that?

Matt: Yeah, it was kind of built around the milestone for quality.

Scott: So, some of the ramifications of that are that if there’s a feature you’re dependent on to build your feature, it might take you longer to get that. But, when Microsoft releases things like Community Tech Previews, they’re going to be more stable by default, because things were only checked into the source base that passed a higher quality bar. Is that right?

Matt: That’s the goal, and we were able to benefit from the approach with the ASP.NET AJAX release. We embraced the feature crew model when it was still ramping up for the rest of the division, and it worked out pretty well for us. We learned a few things we could do better.
We also learned that some of the dependencies became a little bit difficult to manage. If you truly had a feature that needed something else to be completed, then you ended up with a staggered set of feature work waiting for future builds. You needed an equal set of things that could be done early, in order for everybody to keep being productive.

Richard: It sounds like what you’re doing is trying to make sure that your interim quality is higher by testing feature-by-feature, and by enforcing a quality standard on individual pieces instead of waiting for the entire conglomeration. Is that a correct characterization?

Matt: Yes, hold the line on quality all along. Don’t accumulate debt in order to get more features done earlier.

Richard: That seems really smart to me. There’s an old saying you can’t test quality into a product, and that’s really true. It seems really smart to me. Was that inspired by the SDL’s focus on early process involvement of quality assurance?

Matt: Yes. Before, we almost had two instances of lifecycles. You had development working on one lifecycle model, and you had the quality assurance teams running along six to eight weeks behind the dev team.
But, that wasn’t really working efficiently because all we were doing was having one team try to play catch-up with the other team. The dev team was cranking out features, and the test team was finding problems with it. Then, the dev team was trying to catch-up on the problems that the test team found.

Scott: With Visual Studio 2005 when it came out, there were a lot of issues around stability and quality, and things like that.
It was easily the most complicated development environment that Microsoft had ever built, maybe that anybody had ever built. It had amazing levels of cross-product dependencies; there were big products of their own that were put in there, like Team System.

Some of the impression, at least from the outside, was that it didn’t all come together and stabilize at the end the way that everybody had hoped. So, is that correct? Do you feel like maybe the initial version wasn’t as stable as everybody hoped, and so some of these things have been done as a response?

Matt: That may be the conclusion from my team’s perspective, focusing on the platform. I think we probably feel like we’re in pretty good shape because we have criteria and goals that were met.
But, I know in the MQ timeframes a lot of people focused on a lot of little things that weren’t quite right that they wanted to fix up for the next release. That was primarily targeted at the development tools environment, not the platform.

Scott: You work in the open source world in a sense with Atlas, right? Because there are a set of controls for Atlas that are effectively open source in that Microsoft puts the source out there. They’re maybe not open source in the sense that anybody in the world can just modify the source and submit changes to it, but please talk a little bit about this foray into open source with some of the Atlas controls.

Matt: The AJAX Toolkit Project is a separate team around the corner from us, and it is shared source. They take code submissions from the public. They have released the really snappy UI controls that everybody uses on top of the Atlas part of the platform.
It’s been well received, and people are able to grab the source, and build it, and contribute. It’s a little different for us in comparison to what we’ve done before. We’ll see how much the community itself drives lots of new controls, or if it’s something where they like the fact that the controls are out there, but they don’t really actively contribute a lot.

Scott: Certainly a key benefit of open source is that the source is out there for you to learn from. I think that even taking the Windows Forms controls and the ASP.NET controls, certainly a lot of best practices might have been gleaned from being able to look at how Microsoft did what it did.

Matt: Well, we’ve done that with the Atlas release now, too. That’s a first for us, and I’m really proud of it. When we released ASP.NET AJAX, we released the source symbols for it so you can attach a debugger, and step right through our source code.
It’s not the same as taking source code in from the community, but we look at it as being very open and transparent. It’s one of those things where in the open source community, everybody says, “I have the source, so I can do what I want with it.” From my perspective, though, people tend to say that but not do it.

Scott: I think they call it the Berkeley Conundrum. It’s a variant on “if a tree falls in the woods, but nobody hears it, it doesn’t really make a sound.” If there’s so much open source code out there, and not really enough eyeballs to actually look at it, does it really matter that it’s open source?

One of the things that seems to me like a huge challenge is that inside of Microsoft you can control exactly what people get to do to. You can control the training that they have to have around security, and performance, and best practices. You can control the unit tests they write, and you can control the process that the code goes through.
When you take a look at open source, you just get code delivered to you and you don’t have any control, or any view into the process that was used to create that code.

From Microsoft’s perspective, you’re taking submissions from the community; how do you ensure things like security, performance, and reliability, when anybody could be submitting the code, and you don’t have control over how they got there?

Matt: I’m not intimately familiar with it, but my understanding is that the team has essentially taken on the burden, so far, of ensuring some of the security themselves by essentially laying our security process on top of the code before it goes out to the public. I also think there’s an element of caveat emptor.

But, because it’s coming from Microsoft, there has been a certain expectation of quality. At this point, because we have a dedicated team working on it, they’re in the code and it’s still going through quite a bit of the Microsoft process.
Also, at this point, you can’t just submit and have it go right out to CodePlex. You submit as a team, and your submission is then under review for inclusion on CodePlex.

Richard: What do you think are the advantages of closed source over open source?

Matt: In my mind, it comes down to probably two things. One, the customer knows they have somebody to go back to, that there is a company that they’re buying software from that is backing it up. Two, there is a full-time developer that is working on advancing the software.
Open source has done fantastic things with having a community effort drive things to do good software. But, the fundamental difference is that with proprietary software you have an entity that’s responsible and people employed to push things forward.

Richard: What about the development of that process itself — the mechanics of designing, implementing, and shipping a piece of software. Do you see advantages to the closed source model, and if so, what are they?

Matt: I think you may have a little bit more of a culture of discipline but I’m not sure that it’s really that much different. I think there’s so much passion from people in the open source community in what they’re building that it may be the same kind of thing. The same kind of respect somebody has for their full-time job.

Scott: There are two myths that I frequently encounter, one related to open source, and one related to proprietary software. The open source myth that I hear again and again is that many eyeballs looking at the code ensures security.

Michael Howard really threw down the gauntlet on that and said, “Show me the data, because there’s just no data to support the idea that you can make sure code is secure just by open sourcing it and letting people look at it.”
On the closed source side, one of the things that I often hear as being an advantage is this notion that there’s a company to go back to, there’s a company that stands behind it. My impression is, I don’t have any guarantee that a bug that I find will get fixed in the product. I don’t have any guarantee that a problem that I’m running into will get solved by the company that produced it. I can file a bug, I can call product support; maybe I’ll get a resolution, maybe I won’t.
Do you think that Microsoft or any other company really offers that level of guarantee?

Matt: In my experience having worked as a quick-fix engineer where we investigated customer reported problems from PSS, it was very serious. If you couldn’t find some way to workaround it, it needed to be a product fix. Now with that being said, if there was a reasonable workaround, then you’re back to the issue of whether it’s worth the risk to go patch product code.

There’s not a case I can think of where a customer had a serious problem and we didn’t either find them a workaround or patch the product in order to get them back on track. The open source community can’t guarantee every requested change will be made either.

Scott: I guess the only caveat to that might be that obviously, it’s not going to be instantaneous, in either the closed source or the open source world. I remember a problem I found, and the answer was it’ll be fixed in Service Pack 2, but Service Pack 2 didn’t come out for another six months.

Matt: Very few fixes in proprietary or open source development are instantaneous. More complex issues may take longer, and again there is the concern about compatibility. When something like Service Pack 2 ships, it’s had many hours of testing devoted to it to ensure that even small fixes are getting good coverage.

Scott: Clearly, from your perspective, that guarantee of supportability is real, but there are just certain kinds of physical constraints on how fast the resolution might be available. But in your experience if there is a critical problem, either the customer gets a workaround for it or Microsoft starts working on a fix?

Matt: That’s been my experience in ten years here.

Richard: We talked to one of the guys that works on the Linux kernel. He basically said what happens is that people turn code in to the mailing list, and then everybody takes a look at it. If they’re interested and they snipe at it, then the developer makes some changes. Then, ultimately it either gets adopted by the maintainer – the person who owns that section of the kernel – or it doesn’t.

So, at least in some sense there’s no process to enforce quality during implementation beyond peer review. It seems to me intuitively that that’s a disadvantage. But, I just wanted to get your take on it.

Matt: Of course we’re not working on the kernel, but I know that before a piece of code I write is going to get anywhere that it’s going to get exercised quite a bit. It seems like inherently that’s an advantage.

Scott: Matt, talk a little bit about motivation inside of Microsoft. How much is it sort of computer-generated, and how much is it, “Developer Number 13 had this much code checked in, this few defects, found this many bugs…?” How does it work in the “big house?”

Matt: For us it’s all about our developer customers. It’s about the level of enthusiasm and people’s feedback, and it all feeds off of that. We get our customer’s excitement, what they want to see, what they’re going to be able to build. They say to us, “If only you added this feature you could save me time, and here’s what I want to see next.”
The people here in Microsoft are completely driven by what our customers want to see next. We may be in a bit of a unique position because our customer is a developer, and developers get really excited about the platform.

Scott: I’ve heard from people who’ve interviewed at Microsoft come back and say, “There really is a sense that by working at Microsoft you have an opportunity to change the world.” I would guess that many of Microsoft’s products … if you take a look at Office, Visual Studio, Silverlight, and how you hope those products progress over time, most of these are big bets. In most of these cases, you are part of something that you hope is going to be really big and industry-changing.

Matt: Yes, it’s the kind of thing where I’ve had people come up to me at ASP.NET conferences and say, “You changed how I do my work. It’s been an incredible experience and I can’t wait to see what you guys do next.” It makes an impact on their life. It makes it fun.

It’s different than what Scott was driving at; the “how many defects” and “how many lines of code” kind of thing. We really run off of developer-driven enthusiasm. There’s an enthusiastic, sharp, hardworking group of people that are really focused on the customer and how to make better developer experiences.

Richard: Well, thanks very much for talking with us. I appreciate it.

Matt: Sure.

Bookmark this:

Interview with Rod Johnson – CEO – Interface21

campsean — Mon, 10 Sep 2007 21:41:40 +0000

Interviewers: Scott Swigart

Interviewee: Rod Johnson

In this interview with Rod, CEO of Interface 21 and founder of the Spring framework, we asked him about:

Scott Swigart: Rod, thanks for taking the time to chat. If you don’t mind, please take a moment to introduce yourself.

Rod Johnson: I am the founder of the Spring Framework, which has become the effective standard for enterprise Java development. We also have a spring.net edition. I am CEO of Interface 21, which is the company behind Spring. I’m also the author of several books on J2EE development.

Scott: Great! Talk to me a little bit about the development methodology of the Spring framework. How does that work? Every open-source project seems to have its flavor. Tell me how Spring is built. How do ideas go from conception to features to in production or in development to release?

Rod: OK. I think when people try to understand what open source is, they instantly think about code and that the software is created by a disparate, far-flung group of people who work in their spare time. In fact, to understand the open-source model you need to step back from code and look at the totality of things, which includes documentation, includes support for users, and includes the discussion of ideas.

What we see in the core of our software is that the vast majority of the code is written by a small group of exceptionally good people. I don’t believe there is any other way to produce exceptionally good software. We have known for many years about what Fred Brooks calls the The Mythical Man-Month.

Scott: Right.

Rod: Open source doesn’t change that. We know that if you throw more people at a problem you don’t get a better solution. And it doesn’t matter what approach to distributing the source of the software you use. You’re not going to change that fundamental fact. If you want exceptionally good software, you want a cohesive team of exceptionally good people and you really want that team to be as small as possible.

Scott: How is that team motivated to work on the project?

Rod: The core people who connect to Spring daily are employees. They are being paid to do this. That model works very, very well if you look at open source in terms of the enterprise Java space and also obviously of the Linux space. You see that the model is evolving towards paying the contributors who write the code. For example, I think 21 out of the top 25 committers on Linux are employees of large corporations. The majority of the committers on Spring are Interface 21 employees. Another good example of this is JBoss, the Java application server. The majority of the code there comes from employees of JBoss, who are now employees of Red Hat.

Scott: I want to get your insight on the Linux kernel. My understanding of the Linux kernel is that unless Linus likes a certain patch, it’s not going to make it into the kernel. And yet, the bulk of the development is done by people who work for IBM, people who work for Red Hat, Intel, AMD, whoever. Right? Corporations are doing the bulk of the work. And yet what makes it in or out of the kernel isn’t really under the control of a company. Do you have any insight into how that works?

Rod: I am not an expert on Linux kernel governance. From what I see, one thing that is very, very important to developing high quality software and maintaining that software in high quality for the long-term is consequences: consequences of success, consequences of screwing up.

In the case of Linux, what we have is not a single entity, but we have a number of companies (and in Linus’ case partly also an individual, obviously) and a number of parties that are really baked into the system because there are large consequences if they screw up.

I don’t think one entity needs to be responsible for a piece of software. I think you can end up with a kind of partnership, but there have to be consequences if something is screwed up. That is one thing that scares me about the rather naive volunteerism view of open source because what are the consequences to a volunteer? Sure, in many cases they may be professionals. They may feel as a sense of personal ownership that they cannot be satisfied unless something is as good as it possibly can be. But there ultimately are problems because they don’t necessarily suffer the same kind of consequences if something isn’t taken forward.

Scott: What people have told me is that there is sort of a developer Darwinism that happens. In other words, to get your code in you have to submit it to a mailing list. You have to submit it to what one person described as a “full contact code review” and in some mailing lists you’ll just simply flat out get ridiculed and won’t be taken seriously if your code isn’t to a certain level. So, it seems reputation based, I guess. You can’t really get code in until it’s passed code review and up the chain of maintainers. So, on one hand there isn’t a lot of consequence to writing bad code, except maybe it was a complete waste of time. Your inability to do it well was pointed out for the whole world to see. So, it seems there is a certain amount of pressure to sit up straight and code well, especially if you’re going to try to get stuff into very high profile, very core open-source products like the Apache, or Linux, or some things like that.

Rod: Yeah.

Scott: One of the things that you pointed out is that people who are going to use open source shouldn’t consider it a free ride. It isn’t just something that you should just leech off if you want the stuff that you’re dependent on to continue to thrive, grow, and be there for you, and continue to be enhanced down the road.

But, at the same time, most of the core contributors to open-source products are highly skilled people who are employed by a company, and this is what they do all day long. So, if I’m going to be a user of open source, how could I best contribute?

Rod: The first thing is that I would hate to give the impression that openness means nothing. There is one interesting characteristic, for example, about a piece of software like Spring, which is overwhelmingly written by one company, compared to closed source.

You have the ability to work up to being a committer of the software. Not many people will take that up. And we may choose to approach you to see if we can hire you if you do it. And that is a way of expanding and renewing the developer base.

So, it is open in the sense that there is opportunity for people to come to be committers on the software through the community based on merit. That is interesting because if I want to be a contributor on the Java-based IBM’s WebSphere, I’m sure I could add some value based on my background to WebSphere, but I can’t put my code in there. There is no way I can do it without becoming an IBM employee.

Scott: Right.

Rod: So that is a genuine difference. There is a true meritocracy. So, coming back to your question, which was “How do you pay? How do you contribute back?”

Scott: Yeah, how do you contribute?

Rod: Well, I think the first thing is to remember that although it’s very important that there is that meritocracy based on code there are several reasons that the majority of users are never going to contribute to that code. The majority of users are not going to modify and compile it. The majority of users are going to use it essentially as if it’s a shrink-wrap product.

I make all of our consultants just use the binaries. If they’re working with our product with customers, we don’t like them to be working with the source code and making changes to the source code of our product.

The interesting thing is that it simulates more accurately how end users use it. So, often we’ll go into customers where we’ll see that they just realized that you can mount the source code and if you want to step through the same work source code…

Scott: Right.

Rod: …so, it just doesn’t make sense for most end users to even consider modifying the software. Consider that Spring is 1.2 million lines of code. Some of that code does things like transaction management, the fundamental configuration of your application. Some of it executes in virtually every runtime call set. It doesn’t matter whether its open source or closed source, you really don’t mess with code that does that kind of thing unless you know what you’re doing.

Scott: Right.

Rod: The process of knowing what you’re doing…For example, in our transaction infrastructure I, myself, as the original co-designer, made a change about a year ago. I spent almost a day conceptualizing the current state of the code (which one of our other guys had modified) and figuring out exactly how I could do these [changes] in a safe way.

I am a strong enterprise Java developer with 10 years experience. I originally co-designed this module so I know how it is meant to work, yet it still took me almost a day to feel that I can safely make a change without risking breakage of a core part of the software.

Imagine if I were someone coming from outside who didn’t have 10 years of experience, hadn’t co-designed the module. What is the probability that I’m going to make that change in a robust way unless I spend maybe five days fully getting inside it?

Now, if I am an end user and I am writing software, for example for a bank, like many of our users, they don’t want to pay me for five days to figure out how the Spring framework works internally. That is irrational. There is little economic benefit for end users. There is a high probability of being economically irrational in modifying the open source software.

Another good example is Linux. I did a lot of C-programming early in my career. I can code in C. Am I ever going to look inside the kernel and change any of the kernel? This makes no sense, whatsoever. So, code contributions and changes are not likely. However, there are many other things I can do if I wish to give something back.

One thing I can do is to help other users of the software. Most open source projects have forums and mailing lists. Every time a developer answers a question for a user that developer is distracted and can’t write code to improve the product. Both things are important but no one can do both simultaneously.

One thing that has worked brilliantly with Spring is, especially early on, we made a big effort in the core development team to answer users’ questions and be really, really helpful. That set up a virtual circle, where users saw the Spring development team was really helpful when they were first starting up. Gradually they used the software more and became experienced. Then, they see other people asking questions and they think, “Hey I know the answer to that question. Maybe Ron doesn’t have to spend time answering that question. I’ll get in there and I’ll help the other user.” That’s a valuable way of giving back.

Another thing is promoting the software because open source software, like any other form of software, can only be successful if it is popular. For example, you like a particular piece of software. You wonder whether there is a user group for that software in your city. Maybe you have time to organize the user group. Maybe you can persuade your manager to provide coffee and a room if someone else is willing to organize it.

Another way that you can contribute is through issue tracking. If you see something that you think can be better, go and report it. An issue tracker is the enhancement request. That works very, very well with open source because that is one area where access to the code really matters. We see people who make enhancement suggestions with pretty deep knowledge of the code. We might get an enhancement suggestion saying, “By the way, if this class extended some other class it would allow the possibility to do X, Y and Z. Have you considered that?”

What happens is that one of our developers looks at it and says, “Oh God, I would never extend that class. Oh, but I see what he’s getting at. Oh, yeah! He’s right. It would allow X Y and Z. So, I wouldn’t do it that way but I can see that there is a lot of value in what he suggested.” The ability to look at the software is quite interesting from that point of view. It’s not necessarily so interesting in terms of modifying the software but it’s quite interesting in terms of the ability to provide suggestions with as little or as much context as you want.

Finally, and this is the way in which I think most financial organizations will end up giving back, go find someone who is contributing to or sustaining that software and buy a support contract. Not only is that going to sustain the software very effectively but also open source software needs the ability to have maintenance, especially as open source moves into the enterprise. What happens if a system that does trading goes down at 2 AM?

There’s not much point if one of your guys gets paged. He didn’t ultimately write the software that is involved in it and the fact that it is open source doesn’t really make any difference. The need for a similar level of SLA and mission-critical support is exactly the same.

Scott: So, that’s one of the things that I want to drill into a little—two things actually. One is support and the other is the software development lifecycle.

It looks like if you take a look at something like Spring, there is nothing magic about the way it is built perhaps. You probably have internal meetings. You come up with plans. You come up with specs. Engineers are assigned. Software is written. It goes to quality assurance.

Open source gives you much tighter coupling to the user base and the community at large. They provide better issues because they look at the source code and even recommend a certain implementation and talk about the merits of a particular implementation. They file bugs directly into the same bug database that your internal people use. So, open source gives you that tighter coupling with your users. But—and I don’t think this is unique to Spring—I am finding this a lot; it is still built the way software has always been built.

Good people. Good process. Good QA. You have people assigned to do documentation. You have people assigned to do QA. Even though those are not the things that people would volunteer for, you’re not dealing with a bunch of volunteers. You’re dealing with paid employees.

Am I accurately summing it up?

Rod: I think you’ve got it. You raised one really interesting question there. You said that the community uses the same issue-tracker as we use. That is obviously true. We don’t have anything around the software in terms of management of bugs that isn’t publicly visible.

A big part of open source, to me, is free use of information. I know who wrote a piece of code, I know when it was last changed; I know what issue that person was trying to fix when they wrote that code. I can see all that information.
It also means that there is a great deal of security in adopting software that was produced by small companies. For example, at Interface 21, there are slightly under 50 people. We’re growing pretty rapidly, but that’s still a pretty small software company.

So, let’s suppose that our product was closed sourced. We would, in each deal, be dealing with escrow agreements, that kind of thing. Frankly, other people would not think of using our software. They wouldn’t necessarily know whether the internals of our software were any good. They could see it from the outside, but they wouldn’t know, for example, what’s our philosophy of bug fixes, what is the quality of the documentation internal to the code, or what is the quality of design.

I do believe that there is a very strong benefit in open source. I believe that all that information is freely available. You could look at the software, and I would strongly encourage [that for] customers making significant adoption decisions.
They’re probably never going to modify the code, but they should send some of their best guys to spend a day looking at the code. See what they think of it. See whether or not you think this code is maintainable, whether or not the language idioms reflect best practice. That’s a very interesting benefit.

Scott: One of the other things you talked about was support. That comes up a lot with open source, because the way to get support isn’t always as obvious as with closed source. If you buy something from Adobe or you buy from Microsoft or you buy from Symantec, it’s obvious where you go to get support.

In the open source world, it seems there are two models. One is, “OK, you pay MySQL, you buy their enterprise software and you get support.” It’s the same thing with RedHat and the same thing with Spring, I’m guessing – can you purchase a support agreement from Interface21?

The other option seems to be that you go to someone like Open Logic, who supports everything, and you just pay them for support across a lot of products. Now, you’re probably a little biased in your answer, but what do you think about the two different ways of getting support on a product?

Rod: There is nothing wrong with an aggregation model if you’re looking at, effectively, as a broker who is providing the ability to reach the bloke who wrote the software.

Where the aggregation model, I believe, is flawed, is in cases where essentially it’s not providing the ability to reach the people who wrote the software, and also where it is not providing the ability to pay for improvements to the software. So, what am I buying? So, let’s suppose we’ve agreed that we need to give something back. If we’re adopting free software for potentially the next 10 or 15 years, we need to give something back.

We care not only about how we get bugs fixed or how we get our production systems working, we also care that this software is still going to be mainstream in five or ten years or at least hasance of being mainstream because if it isn’t, we’re saddled with an expensive legacy. We also care, clearly, about the software evolving to accommodate new features that are coming forward in the industry over time. So that means: First, we need the ability to make critical fixes if necessary, to ensure that there’s no risk that our production systems are affected.

Second, we need to be concerned about where that investment is going to come from in terms of sustaining the product and moving it forward. So, we’re not just paying for an insurance policy today, we’re paying for the insurance company to be there tomorrow, and for the premium we’re paying to the insurance company tomorrow to be meaningful.

This is where some of the aggregation models, particularly what Open Logic calls it’s “Expert Community,” totally fall down. Because with the Open Logic model, not a cent of what you’re paying is going into writing software. You are not paying a cent towards writing software. That can never work in the long term. What you’re doing is paying, presumably- I haven’t looked at their price point, but I’m assuming their price point must be pretty low – a low price point for something that is not comparable with true enterprise support.

There are two problems with the model. One problem with the model is that it does not do anything for the health of the goose that lays the golden eggs. It says, “Well, I’m not going to feed the goose. I don’t care about feeding the goose; I’m just going to scrape up the golden eggs.” Obviously, the goose isn’t going to be there tomorrow, if that becomes the dominant approach.

The second problem is you just cannot get good quality support out there. Let’s suppose that I’ve got a piece of software that may have a lethal, critical problem in production. Now even though I think, for example, that the Spring Framework is a fantastic piece of software, the best pieces of software in the world have bugs in them. It’s impossible to write perfect software. It doesn’t matter whether it’s open source or closed source.

So, you’ve got a critical problem in production, and it’s really hard to resolve. You will see this happening with both commercial companies and closed source companies and open source companies. They all need to assign resources to work on a resolution of that problem until it’s fixed. When the developer in San Francisco clocks off work, he hands it over to the guy in Australia, who continues working on it until, probably, the guy in Romania starts work at the end of his day. And you need to be able to pay for that, and you cannot support that quality of support with essentially paying pocket money to volunteers.

Obviously, I have a vested interest. My company represents a different model. I hate it when people sell open source short. I hate the view that, “I only use this open source thing because it is going to be cheaper”. I feel so strongly about that because I used to be on the other side. I used to be the person making purchasing recommendations. In the financial industry companies where I worked, I would look at TCO. It made no sense to me to say, “Hey, I can save $100,000 here on a support contract.”

What I needed to look at was where, for example, one of the systems that I was involved in designing handles $5 trillion annually. If anything goes wrong with that system and we can’t fix it for a day, the cost to that company could be in the tens of millions of dollars. In fact, if it was down for two days in that example, the consequences to the UK economy could be quite catastrophic. So, I’m not going to be making the decision on, “Well, you know we can choose open source here and we can save $100,000 on a support contract.” I am more likely to be thinking, “OK, well if open source software seems to fit our requirements really, really well, what do I do if I have a mission critical outage? Who do I pick up the phone to call? Who can organize a team to work on this until it’s fixed?”

Scott: One follow-up: It seems that we are talking about the extremely large enterprise accounts that are handling enormous amounts of financial transactions or things like that. They have enormous vested interest in the quality and the future of the software that they are running. It seems like smaller organizations, whether they are choosing closed source or open source, kind of get to draft off of that.

Talk about the decision for the smaller organizations. For a 50-person company, how is the decision different than it is for an enormous financial exchange?

Rod: That’s an interesting point. For the smaller companies it depends on what you’re doing. For smaller companies who are not doing something that is as mission critical, it may be that the software effectively is free to them, and they don’t have the same requirements around mission-critical support.

Yet frankly, from my point of view as a chief executive of an open source company, if people who don’t need that kind of mission critical support don’t buy it because, for example, their company simply can’t afford it, good luck to them. I would love to see the 50-person company grow into a 500-person company with a website that has a huge amount of traffic for which they need to buy a support contract.

So, I think that open source expands the market and it brings the software to a new audience of people who don’t necessarily have the need or capacity to pay. Essentially, good luck to them! No one should give open source money out of the goodness of their hearts. They need to look at what they need as an organization and who can satisfy that.
So, for example, for every job where I have personally been involved, I would have been getting support contracts for the open source I was using because of the nature of those businesses.

However, let’s suppose I was working for a start-up, a dotcom that is trying to build up some kind of community site and is trying to get traffic. It doesn’t really matter if it goes down some of the time. The software that my company is developing is rapidly evolving. At that point I wouldn’t have a support contract. Let’s think about Twitter. Twitter is a great example here. Twitter clearly went through that. Then it became successful. Now they are encountering technology limitations, which are causing frustrating outage. Now you are getting into a different set of business needs. Now they are reaching the point where it is going to harm the growth of their business unless they improve the uptime of their infrastructure.

Scott: So, in other words, your goal as a start-up is to grow to the point where you need a support contract.

Rod: I have never put it in those words before but I think it’s an interesting way of thinking about it. In a way, you have a greater ability to pay the price that is appropriate to your business needs.

Scott: Right. At some point if you are a start-up and you’re not successful, then the open source that you are using is probably not as valuable to your business as if you are a start-up and you become very successful and you grow. You become an ongoing concern. Well, now your infrastructure has value because your business has value and you need to treat it as such.

Rod: Yeah. I mean what I would suggest is that in the early days people should look at purchasing things like consulting and training around open source.

Scott: Right.

Rod: I think that makes a whole lot of sense because it is simply going to reduce their total costs by making their developers more efficient.

Scott: Right.

Rod: So, the needs profile of the end-user is significantly different.

Scott: Well, I really appreciate you taking the time to chat.

Rod: No, I think that was an interesting discussion. Thank you for following up.

Scott: Thanks very much.

Rod: Thank you, Scott.

Bookmark this:

Interview with John McCreesh – Marketing Program Lead – OpenOffice

campsean — Sun, 05 Aug 2007 21:03:49 +0000

Interviewers: Scott Swigart, and Sean Campbell

Interviewee: John McCreesh – Open Office

John McCreesh

In this interview with John who is the Marketing Program Lead for Open Office we asked him about:

Scott Swigart: John, thanks for taking the time to chat. Could you take a minute and tell us a little about yourself?

John McCreesh: I have been doing work on a voluntary basis for OpenOffice.org for five or six years now. For the last year I have been leading the worldwide Marketing Project.
My day job is in big commercial IT shops. I got into open source during the .com era – I suddenly found this wonderful world outside! As I’ve got a technical background, I started hacking for projects and then after a time decided I could probably do less damage by marketing than hacking. So that’s where I am now.

Scott: OpenOffice.org is interesting to us because a lot of the open source projects that we look at are things like the Linux kernel or Apache. Those are almost by developers for developers, to some degree. The feature set is driven by the developers who work on it. OpenOffice.org is different in that it doesn’t target IT professionals so much. It targets end users, right? The users are people who are producing documents, essentially.

So I’m curious, with OpenOffice.org, what’s the mechanism for features to get proposed, to be slated to be worked on? What’s the process for people deciding they are going to work on a particular feature? How does that work? Because it seems that it would be different from something like Apache.

John: That’s right. I think the traditional open source model is very much about scratching the itch. If you are a developer and you are not happy with the editor that you are using then you go off and write your own. That model does not apply in OpenOffice.org where as you say, it is much more of an end-user tool.
I suspect a lot more of our developers would sit down and write a document with Emacs than they probably would with OpenOffice.org.
The thing is though, lots of people get a big kick out of developing for a project that their mom and dad use, or the kids use, or they can take around and hand out at the school and say, “Look, you can use it for free and I am one of the people who helped develop it.”
Another thing is that this is a huge piece of code. So, for developers that poses challenges, compared to a lot of traditional open source projects which are comparatively small in terms of code and have a small number of developers.
The development methodology of breaking things down into small modules makes it easy for more people to work in open-source. But, OpenOffice.org has grown up over 15 years, so there is some very old code there and there is some very new code there.
Getting developers in to find their way around and to make contributions can take a bit of time. But equally, if you’re technically inclined, to get some code in there, to build OpenOffice.org, and see your work coming up the other end, again, that’s a big achievement.

Scott: Right.

Sean: When we talked to Stormy Peters, one of the things we talked about is how products take in end user requests, But So with that in mind what is your the overall process with OpenOffice.org for taking in a user’s request likesuch as, “I would like the ability to handle text editing just a little differently here because my boss wants me to.” But they have no ability to write it. In other words: What’s the process for getting those things integrated?

John: Yeah, we sometimes get interesting conversations between developers who say, “Well if people want this, why don’t they just write it, or find someone that can write it?” I think one of the challenges for us is to bring together people who’ve got the ability to respond to the user’s requests, and to get the users to put the request together.

We have the standard open source toolkits. Anyone can raise an issue or raise a request for an enhancement on our website. It will get checked to make sure it is valid. Other people can go in and support it, add notes to it and so on. So there is that traditional open-source process: people vote for things. It is an entirely open process and you can see whether what you’re looking for is commanding support in the community.

But equally we have a fair number of developers who are paid to work on the project. So, Sun Microsystems’ developers take the OpenOffice.org code and release it as StarOffice. Like any commercial software house, they have big customers who say to them, “Hey, you know this particular feature of StarOffice is poor. Can you do something about it?” If they’ve got a support contract with Sun, then Sun will put some developers onto it and they will get into the OpenOffice.org code base that way.
Similarly, when Novell started looking at the code base and decided to take OpenOffice.org and make it a significant part of their SUSE offering, they started using it internally themselves. There were things that they thought, “Yeah, we need to do this. We need to add this feature. This is important to the kind of market we are going to sell in.” So again, they took some of their developers and put them to work on it.
So OpenOffice.org is a combination of the traditional software model where you have a software company who listens to its users, plus input from users around the world.

Sean: What percentage of the new features that go into OpenOffice.org do you think are driven by a developer request mostly from soup to nuts or vs. what is onethose that is are driven predominantly by an end-user request but then just implemented by a developer?

John: That’s a tricky one.

Sean: Do you think it is equal or is the product still driven by developer input more than end-user input?

John: Well, it’s hard to say. One of the things that people said to us at the launch when we released the Version 2 product was, “Yeah, you’ve got all the functionality that we want but can you make it run faster?”

Now, from a developer point of view, a lot of the hacker community was really motivated by this. The thing about making more efficient code and making it smaller and leaner and all the rest of it really rang bells with a lot of people. That was something that had tremendous developer appeal, but it also has real marketplace end-user appeal. So I don’t know whether you would say that was driven by end-users or driven by developers. It was a happy area where the two interests coincided.

Sean: OK.

Scott: So on an open source project there is usually a chief maintainer right? If you take a look at the Linux kernel, it’s Linus Torvald who ultimately gets to decide what’s in or not. Is it Sun Microsystems that’s the final arbitrator of what gets checked into the main source tree and what doesn’t?

John: We have an Engineering Steering Committee, or ESC. It’s the ultimate arbiter if there are disputes about what goes into the code or what doesn’t, and the general technical direction of products is decided there.
On a day-to-day basis, the project is too big for any one person to sit there and say, “That goes in and that goes out.”

Scott: Right.

John: So OpenOffice.org is divided into a number of different projects, where the project leads have the ultimate say as to what goes in or what doesn’t go into a particular build.

Scott: OK. But I’m guessing that because Novell and Sun are so involved in it that some of their people would be owners of these different subsystems?

John: Yes. And I think that’s perhaps going to continue more and more as more commercial companies sponsor developers.

Scott: And that’s not uncommon. The one thing that I have heard across the board with open source is that company involvement is not a bad thing. Open source owes a lot of its success to the fact that IBM, Sun, Novell, Intel, AMV; other corporations are paying developers full time to work on it. So that’s an integral part of the process.

John: Yeah, I saw some analysis years ago about the Linux kernel. At that stage most of the contributions were coming from people who were paid by corporations.

Scott: Yeah, I would guess OpenOffice.org isn’t any different. I mean, when you take a look at a product that’s this large, that is this complex, there is a decent barrier to entry—I would guess—in really being able to get in there and really understand the code base, really develop experience with it, really understand the architecture and just get to the point to where you can make good, clean contributions that are going to add features or fix bugs and do it the right way.
For a lot of these projects, they have grown to the point where there is a high barrier to get to the skill level you need to get that to be able to do that. So it makes sense that the people who are being paid to work on it are naturally going to have an advantage in just really be able to produce the quality of code that is going to ultimately make it in.

John: One of the changes we have tried to make with the architecture in the past couple of releases is to open up the product more for extensions.

Scott: Yeah.

John: People who aren’t as technically skilled, and will find it a real challenge to go through pages and pages of code (some of which might be commented in German) and try and make sense of it, are quite capable of providing functionality in the form of a plug-in.

Scott: That also seems to be a key factor in the success of a given open source product—that it has a good extensibility story. It has a good modularity story. Apache is a great example of that. To work on the Apache core there is a high bar. But to write modules—anybody can do it, essentially.

John: Firefox is another classic example of that. It’s used a lot. Again, the developer gets a real kick out of seeing their piece of code looking as if it is part of the core product. Once you have incorporated the extension into the menus and in the help system, etc. then as far as anyone looking at the product is concerned, you have written a piece of Firefox or a piece of OpenOffice.org code.

Scott: Right, right.

Sean: I have a question about the new features. How is the QA process handled? A lot of our conversations have taken us in interesting directions as we talk to people who submit stuff, not in any pejorative sense for an open source but it just is interesting how who are contributors as the process of checking in code, andgoing through the process of validating it differs from project to project in terms of the rigor or the processes that go about it. So how does that work for OpenOffice?

John: There are two schools of thought. In OpenOffice.org there is the “Community OpenOffice.org”. So if you go onto our website and download OpenOffice.org, that is what you get. There’s also a hacker’s version of OpenOffice.org, which doesn’t go through the Community QA process. This version feeds into some of the Linux distros, who will take this code and will do whatever QA they feel is appropriate around it.

So if you want an absolute leading edge of OpenOffice.org, you go to the hacker’s version. If you want something that has been through quite a structured QA process, you go for the community version, which is what we do.

Scott: The community version, is that…? Who is ultimately responsible for doing that QA? Are there QA teams within some of the corporate sponsors of OpenOffice? Is it more of the responsibility of the distros to do that QA as part of putting their distro together?

John: QA is one of the OpenOffice.org community projects, which runs one of the most widely geographically distributed QA processes I have ever seen, because it’s not just a matter of getting the American English master version correct.

Scott: Right.

John: The native language teams in all the various countries will go ahead and do a QA process on their own build. Ultimately that’s one of the things that controls how quickly we can release. We used to go for long periods of time between releases. A couple of years back we looked at that and decided it was putting developers off. If you have to wait a year before you can see your code emerging into the marketplace, you’re not really very interested. We’re now down to about four releases for the year.
We would like to do more but the QA process is a limiting factor.

Scott: Sure.

John: Similarly for the hacker’s version of the code: if Red Hat or Ubuntu decide to use that as their source, they have to put it through whatever QA process they feel is appropriate for their marketplace. We all have tradeoffs between features and a stable product.

Scott: Yeah, yeah. And things like documentation also—is there a documentation team as part of the project as well? I would guess that would run into the same localization challenges because the product is such a global product.

John: That’s right. We also have a Documentation Project that does user guides and manuals and how-tos and all that good stuff. But there must be about a dozen other independent sites on the web where people have set up help forums or wikis or whatever.

Scott: Gotcha.

Sean: How much of the documentation creation is deflected or is tuned based on gaps you find that users are asking for? How much documentation is written because someone feels a personal need to write that documentation, or because they know people have struggled with it—similar to a developer adding a feature just because they need it?.

And how much of it is driven top-down from OpenOffice, such as, “We see X amount of requests for this, we need to build a doc set out to cover that need?”

John: It’s a mixture of the two. The way that I got into the OpenOffice.org project in the first place was I wrote a piece of documentation – a ‘how-to’. I’d been playing with the database stuff, thinking “Hey, this is really cool.” But it wasn’t not obvious how you got into it or how you found it in the menus.

So, I asked a few questions on the lists, and wrote the how-to. That got a fair amount of circulation, so I realized there was obviously more to OpenOffice.org than meets the eye. And that was how I got into the project.

So, an awful lot of that goes on. You can never have too much documentations and how-tos for a product as big as this, because different people have different learning styles and different needs.

Sean: One last follow-up on that. Are there types of documentation or areas of the product where you find that maybe there’s a pocket missing content-wise because it’s more community-driven versus top-down?

And are there any defined areas where you feel there is a gap and you have to motivate the community to contribute in a particular area?

John: I think the problem we have is the old Google phenomenon, that there is now such a mass of stuff out there on the web, it’s tricky finding the stuff that may be current, knowing what versions it applies to, and then finding the information in the form that you require.

We usually do one big master user guide, which we publish on the web. It’s also available from one of the publish-on-demand organizations, so you can order paper copies of it from there.

But apart from that we don’t try and have a monopoly on documentation because there are just so many other people out there who want to do it.

Sean: But you would say that if there is a pain point of sorts, it is that, “OK, I’ve Googled how to do mail mergex, but I don’t know if it’s the appropriate steps for this version of the product.” You know what I mean?

Or if it is, it might be missing an extra step that could accelerate ithe process of getting the job done from a user perspective. It’s like you said, you could solve a lot of developer problems by Googling them. That’s almost like Step 1. But on the other hand, you’ve got to make sure it maps to your version of the API, your version of the development tools, those kinds of things.

So, that’s a fairly accurate reflection of the main pain point.

John: I think so. We can produce a master reference manual, but it is in reference manual style, and some people find that off-putting and they’d like it more conversational style.

Sean: So, one question that I have is in OpenOffice, one of the key features is its cross-platform nature. It runs on Windows, it runs on Mac, it runs on every Linux distribution. Do you have any sense of how much development effort goes into ensuring that it is supported and runs well on all of those different platforms compared to the effort that is put into building new features?

John: I think it’s back to the QA limitation, that the more that we officially support, then the more testing that you have to go through. It’s also a major technical challenge because, if you’re completely cross-platform, then in a sense you’re always slightly sub-optimum on any given platform.

Scott: Well, that’s another question. Does OpenOffice.org strive to have full fidelity across all platforms? Because I know that some open source products go the route of they’re really optimized for Linux, and they’ll run on Windows or Mac or things like that. But they’re not 100% the same experience. Do you know if the philosophy of OpenOffice.org is that the experience and the features be 100% and the same, even if it means that it’s not really optimized for any platform?

John: That’s effectively where we find ourselves.

Scott: OK.

John: This is why the Mac port has taken such a long time, because the Mac is such a specialized platform and Mac users want all their applications to look just exactly like a Mac application should do. So, it’s quite hard to do that and still maintain the cross-platform thing.

Scott: So, there’s a core that’s really generic, but then for some of the presentation layer stuff, there’s actually Windows-specific code, Mac-specific code, you know, Gnome- and KDE-specific code.

John: We provide for example a common file selector out of the box. Or you can switch it off and say, “Just use the native file selector for my platform.” But the more of this you offer, the more your documentation has to be platform-specific. The common look and feel that makes the documentation piece easy to do.

Scott: Right.

Sean: This is in a different direction, but there’s an argument that goes that certain products in the open source world tend to follow rather than innovate. Sometimes OpenOffice.org is held up by that, right?

It’s trying to get to some base level of parity with either the most recent or the current version of the obvious competitor, which is Microsoft Office. But adding features that leap ahead beyond the current evolution of the office product isn’t typical.

I mean, just this statement, what’s your general response to that, when you bump into that? Because I’m sure you must have, at least at one point or another.

John: The problem is that office productivity software is a mature market. It’s been around a long time and we’ve been through the WordPerfect and the SuperCalcs etc.. By now, people know what they want out of an office software product, and they also have an expectation about how it’s going to look and feel. You know, where they’ll find things on menus.

So, you need quite a strong driver to go out and break out and produce something radically different. Chances are that unless it fundamentally gives people a better way of working, then they’re not going to accept it. Why would they learn this whole new thing if they only want to write a memo? They already know how to do that.

Sean: I guess in a product like OpenOffice, what would be the path to innovation? If you’re saying, let’s imagine a world where both are free, right? And we’re trying to equate it based on our level of productivity benefit to the end user and those kinds of things alone.
Where do you think OpenOffice.org or another product that’s a productivity suite, can have a leverage point there, I guess?.

John: Let me tell you where I think this happened. When we launched Openoffice.Org 2, we closed all the functionality gaps between OpenOffice.org and Microsoft Office. So for the average user, what they do 90% of the time is absolutely no different on either package. We recognized that and Microsoft recognized that.

So, our response was, well, the one thing that people are telling us is that, “OK, maybe we’ll move to your software, but every time we move we’ve got all these data conversion errors, or data conversion problems”. Or “I spend all my time writing these documents, but unless I’ve got your brand of software, I still can’t access my documents.”

So, we were hearing a demand from users that they wanted to own their intellectual property, the documents and spreadsheets that they’d created, independent of any software supplier. This took us off down this road that is ultimately the OpenDocument Format that you’re well aware of.

Sean: Yeah.

John: So users got an ISO standard for how office data should be stored. Once you’ve got that, then no software vendor can ever lock you into a product again. And it makes it easier to get your data and use it in corporate systems and all that good stuff.

So, that was our response to “how do we differentiate ourselves in the marketplace from Microsoft?” We knew that Microsoft would have grave philosophical difficulties going down that route.

At the same time, Microsoft looked at the same problem from their side and went, “What can we do to put a gap between ourselves and OpenOffice.org?” And their response was not actually to add new features or change the core of the product, but to change its look and feel.

This is where we got the Ribbon and all this other stuff. Sure, it looks different. Their response was pretty much a consumer-driven thing. “Let’s make it look sexier in the marketplace. Let’s get it looking so sexy so that people see it at home and then they go back to work and say, ‘Hey, we’ve got to have this Ribbon thing at work.’” It doesn’t actually add anything to their productivity, but it looks cool.

So, Microsoft has gone off in one way, which is a consumer-sexy-look-and-feel thing, and we’ve gone down the more technical route. What people are telling us on the domestic side is they want to know that their grandchildren will be able to pull up Granddad’s diary from 50 years ago and still be able to access it in whatever office software they’ll be using then. On the commercial side, public administrations worry that 20 years from now, someone’s going to come slap on with a Freedom of Information Act requirement to dig out some documents they’ve filed today. Are they going to have the word processor that wrote that document 20 years ago? Absolutely no chance!

So, the ODF development was our response to that demand for freeing the data from the application. So, was that innovative? I think that was hugely innovative. I think it was far more innovative than Microsoft’s response – . they got a few bells and whistles and their menus look a bit different.

Scott: Something like the Ribbon, that’s an interesting point, because how does OpenOffice.org make a decision about whether or not OpenOffice.org will adopt that look and feel or not?

Because I think that’s been a debate. OK, Microsoft significantly changed their look and feel, should OpenOffice.org do the same or not? What’s the thinking on that? And, I guess, how do you make decisions like that?

Sean: And –before you answer that—one of the things that I’m pretty cognizant of is that part of the change for the Ribbon was obviously just change itself. And then the other part of the change was driven by some studies that said, “Well, let’s put the first thing a user wants to use first in the Ribbon, etc.”

There are a lot of arguments about the decisions that were made, right? One person’s choice might not be another’s. But that was the driving influence too. But And to Scott’s point, how would something like that come to fruition in the OpenOffice.org environment? If it was deemed necessary, obviously…

John: OK. The Ribbon thing is one of the best things that happened for us for a long, long time, because over the years, we were sick of hearing arguments that said, “OpenOffice.org menus are slightly different from Microsoft Office’s, therefore, if we’re going to have to migrate people, it’ll cost us billions of dollars to retrain everybody to know that option isn’t here, it’s somewhere else.”

Scott: And now they’ve given you even a bigger change on their own end, right? So, yeah, there you go.

John: It’s a much simpler migration path from an end user perspective to go from current Microsoft Office to OpenOffice.org than it is to go to completely new Ribbon-style interface.
Will it catch on in the marketplace? I don’t know. I think it’s a big gamble on Microsoft’s part. They’ve tried a couple of times in the past to push the market in ways the market has eventually revolted against. They have had marketing failures in the past.

And it also takes an awful long time to get people off legacy versions of Office. There’s never very much more than low double-digit numbers of people using the latest version. So when latest version is such a big step change, it’s a big gamble for them.

Would we ever go the same route? Well, if five years from now, if everybody in the world has decided that Ribbon is the way to go and that’s what they want in a product, I suspect we will have to do something similar or come up with something better and convince the world that’s there’s a better way to do it. But at the moment the jury is very much out.

Scott: So in other words something like that, the prudent thing to do is to take a wait and see approach, in other words.

John: Exactly, just as Microsoft is doing around ODF. It’s doing its usual thing of trying to say, “Well, OK, if you want a standard, we’ll invent the Microsoft standard, and try to sell that to the world.” So that was a fairly predictable response.

Scott: Now, I thought I did read something saying that Microsoft was going to allow something like a “Save As ODF.” I thought recently there’d been a change in their thinking on that. I mean that’s outside of the scope of this conversation. But it seems like I bumped into that, but maybe I, maybe not, I don’t know.

John: What they have done is they’ve offered some support to an open source project to set up converters – ODF plug-ins for Office. Part of their recent agreement with Novell—and I think there’s something with Linspire as well—had some clause in it about working to get more file compatibility.
So, they’re not stupid. They’re keeping in touch with the technology, finding out how it works, so if they do find the market is forcing them down that way, you know it’ll be a comparatively easy thing for them to quietly drop into the next release of their product.

Scott: OK, I get it.

Sean: Well, one question on that too. So would you say that one of the strengths of open source is that you’re not going to push for a change as significant as the ribbon unless the user base asks for it? In other words, you won’t push a change like that top-down and if the users aren’t asking for it then a wait-and-see approach makes the most sense because nobody’s asking.

John: That’s right. It’s hard to think of anything where we’ve gone to change for change’s sake. But, equally, if one of our developers came up with a really cool new feature that no one had ever seen in an office suite before and, and we looked at it and thought, “Hey, that’s, that’s wonderful,” we’d do everything in our power to get it out into the product.

Scott: Well and I guess too you guys have the option of, like you said, there’s the hacker’s build, there’s the experimental build, and it seems to me like a lot of things could be tried out there, and if they caught on and were popular, it might make sense to move forward into the stable tested one.
Is that how it works? Is there a Darwinian effect with features where, somebody codes it up and it makes it into the hacker’s build, and then it either lives or dies there? It either catches on and it makes it into the, the community one or it doesn’t and it doesn’t.

John: Yep, that’s been a route in the past and we think the extensions route will be a much faster way in the future. So if someone brings out an extension, and we see that everybody and his dog is downloading it and blogging about it saying how wonderful it is, and if it makes sense to put that into the core product, then that would be a very good way of moving the product forward.
But, equally well, if it’s working well as an extension then why would you want the overhead of putting it in the core product, unless from a maintainability or efficiency or some other reason?

Sean: Well, what, what, one question I want to ask goes back to the original genesis of what we’re trying to accomplish in our investigation, “if the question was posed to you, “OpenOffice.org is predominantly an open source project. Microsoft Office is obviously a closed source project. From a development methodology standpoint, what would you say are the important characteristics that are advantageous on the OpenOffice.org side compared to Microsoft Office?”

Scott: What Sean is going for is what do you see as some of the biggest advantages of having OpenOffice.org developed as an open source product? What comes out of the open source process that is very advantageous?

John: OK, from a developer’s point of view clearly it means you know anyone can contribute to the project without being a Microsoft employee. From a general marketplace perspective, the appeal of open-source is it’s a transparent process. Anyone can request features, record bugs, whatever. If you’ve ever tried doing that with Microsoft you’ll know it’s not a transparent or efficient process. Anyone can come along to the OpenOffice.org conference and talk to developers and buy them a few beers and say, “Hey! Why don’t you come work on what I’m looking for?”

Then there’s the whole issue around openness of what’s in the code and what isn’t in the code. There’s been umpteen conspiracy theories over the years about Microsoft back doors etc. OK, most of those are just Internet conspiracy theories, but for a lot of governments in the world who are suspicious of big corporations, the fact that they’ve looked inside the code, seen what was there, get their own people looking at it, is very important.

And finally, software companies come and go. Even Microsoft will go someday. If you own the code or if you can get a copy of the code you’re guaranteed that you can run it just as long as you can compile it.

Scott: John, thanks for taking the time to chat.

John: Thank you.

Bookmark this:

Video Interview with James Reinders – Chief Evangelist – Intel’s Software Products Division

campsean — Thu, 26 Jul 2007 22:46:43 +0000

Interviewers: Scott Swigart, and Sean Campbell

Interviewee: James Reinders – Intel

In this video interview at OSCON 2007 we talked to James Reinders – The Chief Evangelist for Intel’s Software Products Division.

We’ll talked to him about the following:

Bookmark this:

Code Review Processes in Open Source Projects…

campsean — Tue, 17 Jul 2007 00:31:25 +0000

Just walking through reading it but here is a interesting study by the University of Victoria on code review processes in Open Source Projects.

http://opensource.mit.edu/papers/Rigby2006TR.pdf

Some of the research questions are:

What types of review does a project use?

Why are patches rejected?

Who performs the review?

When are reviews performed?

How long do reviews take to perform?

Etc.

There is a ton of interesting data in the study from a breakdown of the Linux “Pyramid of Trust” method, two reviewer model that Mozilla uses to the voting model used by Apache.

Well worth the read.

Bookmark this:

Interview with Marc Miller, Open Source Software Evangelist in the AMD Developer Outreach Organization

scottswigart — Tue, 10 Jul 2007 23:29:04 +0000

Interviewers: Scott Swigart, Richard Bowler, and Sean Campbell

Interviewee: Marc Miller

In this interview, we spoke with Marc Miller about his views on the current state of open source software. Marc works for Advanced Micro Devices (AMD), and in January, Marc took on a role as the open source software evangelist in the AMD Developer Outreach organization enabling Linux kernel and application developers to develop optimized code using both AMD and 3rd party tools and resources. In his role as a software Alliance Manager for AMD 2001-2006, Mr. Miller played a significant role in developing a Linux marketing strategy with a focus on integration of AMD technology with software tools developed by the open source community and industry partners. Throughout his career at AMD, Marc has been a key contact for open source developers wishing to work with AMD, and has been an open source ambassador for AMD, helping to coordinate outbound and inbound communication between AMD and Linux developers.

In this interview Marc talks about:

Richard: First off, I have to ask what you think about the idea of Microsoft funding a blog that looks into the differences in delivery model between open source and closed source. Do you think it’s possible that that blog would be unbiased? I mean, we’re doing our best to make it that way, but I’m wondering if you think we can do it.

Marc: There’s a lot of suspicion right now in the open source community. In the aftermath of the Novell-Microsoft pact, it appears that nobody quite understands what Microsoft is up to and what their game plan is. Certainly the community is extremely skeptical about anything that Novell or Microsoft does at this point based on past experiences, for example with the Samba and Mono communities.

That said, while a blog, focused on open source and sponsored by Microsoft would be treated with skepticism by certain segments of the open source community, there would also be people interested in reading it. Everybody wants to understand Microsoft’s angle on Linux.

Richard: I figured that. I figured there’s going to be healthy skepticism.

Sean: When you speak, Marc, about what the angle is, are you asking what Microsoft’s angle is with Novell?

Marc: Yeah, and the way you look at it will depend on who you are. If you’re a large company like AMD that values both their Microsoft and Novell relationships, we’d be looking at that blog for opportunities where our customers can benefit from the marriage of the two.

One of the key things about the Novell and Microsoft pact was Steve Ballmer getting up on a box and saying that this is to protect Novell’s customers from being sued for use of Microsoft patented IP. That has a lot of people very frightened.

For example, people will be looking in those blogs for what the caution areas are. What products might I get burned on if a court decides Microsoft’s claims on software IP are legally defendable?

Richard: That’s interesting. So, with that in mind, I have a few basic open source questions that I hope you’re willing to address and give us your opinion. One is, what are the challenges an open source company faces when competing with closed source companies?

Marc: Very often the open source companies are trying to profit from something that’s not the source code itself.

Take a look at someone like Canonical. All of their software development efforts have been to get Ubuntu into the hands of as many users as possible. They make Ubuntu Linux available at no charge, under open source licensing mechanisms. Their profit model is currently based on support services. For someone like Canonical it’s a question of does the open source community develop products where the user might be willing to pay something for a service like support?

Richard: So, it seems to me that the biggest asset a software company has is the source code. What traditionally has been called trade secrets, intellectual property, or secret sauce.

Marc: In the closed source case, yes.

Richard: It’s an interesting idea to make money by giving your main product away.

Marc: The GPL completely allows for an open source program to be sold for profit. But it also allows multiple copies of that to be made and redistributed without royalties. So, it’s a real interesting area. Since it has been difficult to turn a profit from just selling a GPL application, traditionally most companies that back open source projects use the software itself as an enabler for whatever service or subscription that they plan to sell. In some cases these companies also sell a closed source product that compliments the open source projects.

Richard: So, what do you think are the most compelling advantages to open source over closed source? What is open source able to do much better than closed source can do, in your opinion?

Marc: Open source allows for freedom of innovation in the absence of a driving business case in a lot of cases. Take the Debian community, which has emphatically refused to adopt the same way of doing AMD64 architecture support the way that other Linux distributions have. To give you the ten-second synopsis, other Linux distributions support running 32-bit and 64-bit code in the same environment. Debian felt the way the directory structure was handled in other AMD64-supporting distributions was the wrong approach, and they felt they needed to adopt a structure that would prepare Debian Linux for future innovations. So they elected to support an approach that was different from what other Linux distributions were taking and refused the quick fix implemented for business reasons. I use this example to illustrate that, in the open source community, everybody has the freedom to innovate according to their own interest and ethics, business justification or not. I respect the Debian community for holding to their values. In the closed source space it’s a much more restricted audience; the freedom to do what a particular developer feels is the right thing just isn’t there. People are still confined by ethics, but those can be overridden by corporate objectives and customer requirements.

For example, products sometimes are shipped before they’re truly ready just to stay on schedule. Debian is famous, if not notorious, for delaying a release until the software is really ready and no known critical bugs exist. If a bug is missed prior to release, open source methods allow opportunities for the community to fix the bug after the fact.

In open source software, anyone can disagree with a corporate direction and then redistribute their own version which has things written the way that they think it ought to be. This promotes a freedom to innovate that goes beyond what you can often find in the closed source model.

Richard: How does open source keep from having endless slightly different distros? Is it Darwinian? The things that really work and get picked up live on, and those that don’t just sort of peter out?

Marc: The biggest strength and the biggest weakness of open source is the freedom of choice. Michael Dell has been quoted in the press prior to the May 1st announcement that they were going to start shipping Ubuntu. Each time he’s quoted about responding to when are you going to start shipping Linux, when are you going start supporting the community, he says, "I’m fine with supporting the community. Which one? Which one do I pick?"

Every Linux distribution has a different philosophy. Look at Red Hat; they have had a very long history of preserving backward compatibility and ensuring stability of the products such that when their enterprise customer upgrades, they’re not going to have to recompile their kernel and reinstall a bunch of apps.

Look at the SUSE case; they try to take a much more balanced approach to include innovation in cases where it might be a minor inconvenience, but they believe that the customer will appreciate the value that they get out of the product for that minor inconvenience.

Now look at Debian; I once suggested that Debian try to encourage enterprise ISVs like Oracle to integrate their software with Debian Linux as a way to expand their visibility and market share. A Debian developer responded, "Who cares about market share? What we care about is open source." Everyone on that discussion list thread agreed there’s no reason to make Debian more compatible with Oracle databases because Oracle isn’t open source. Debian’s mission is an open source revolution across all software segments.

So each Linux distribution, besides having a different code base, has a different philosophy behind it. And that creates a lot of choices for the user and that’s a big strength, but that choice complicates the decision too. There are two main product lines in the Windows release – a desktop/workstation line, and a server line of products, and one can even argue that those two product lines are more similar than dissimilar as they leverage many of the same operating system components. There are 359 open source operating system distributions according to Distrowatch, most of which are Linux.

Richard: It seems to me in doing research on this subject, I run into an awful lot of open source advocates who seem to be open source advocates more because of the anti-capitalism sort of flavor to it, at least in some corners of open source.

That seems to be a common voice, although it doesn’t seem to be a common motivation of all of the open source projects I’ve looked at. It looks like there’s an awful lot of people writing open source trying to figure out a way to make money on it. They believe in open source for maybe collaborative reasons, maybe for development cost reasons or whatever, but they want to make a profit.

But this anti-capitalist message seems to be sort of pervasive in one from or another through almost everything I’ve read. Do you think that that hurts the open source community in the eyes of people who are agnostic in terms of development paradigm? Do you think some people are turned off to open source because they see it as sort of anti-business, and they aren’t?

Marc: The attitudes of the open source community have changed over the years. If you look at the Linux community of the late ’90s, you’ll find that’s very much the case, to the point when we launched x86-64.org I asked that all of the corporate logos be removed from the website. Since then we’ve brought them back because the open source developers realized that without a corporate motivation and corporate sponsors that there’s a lot that they’re prevented from doing. Many open source applications are hosted and mirrored on either boxes in a corporation’s data center or using corporate sponsorship money. Linux Symposium wouldn’t happen without corporate sponsorship. Linux-supporting companies look after the interests of the open source communities in standards bodies and industry forums that developers don’t usually have access to.

Also, a lot of corporate sponsors pay developers to do what they would do anyway. And those developers have become grateful to those corporations in a way that has lessened the anti-capitalism trend.

Sean: Let me ask you a follow-up to that, and I know exactly why Richard called it an anti-capitalism bent, but do you see that as the most appropriate phrasing for it? I mean that’s kind of the quick draw phrase that everybody uses, but is it anti-capitalism or is it anti something else?

Marc: I’d say it started off as anti-capitalism because the feeling was that most corporations A) produced strictly closed source software and B) didn’t understand the open source model well enough to be any good to what people like Richard Stallman were trying to promote. So, it was an anti-capitalism setup.

But as I say, modern-day, most of these well known developers are sponsored by a business. So, a lot of that trend has died down.

Sean: So what do you think it’s kind of evolved into? Is there still kind of an anti- bent to the whole community? I mean is it kind of anti-control, or anti-throttling of innovation, or what would you stick there?

Marc: The Linux community is a lot more pragmatic about software IP than they were 15 years ago but long term, open source enthusiasts still are driving toward elimination of software as a proprietary product. It’s a freedom of knowledge movement, and much in the same way that you can go to Wikipedia and read about any topic that someone has contributed knowledge to. In the software case, that freedom of information is creating software instead of encyclopedia articles, and the content creates a new market. You can still make money off software creations, without making money from the software itself. Perhaps that’s services for the software, perhaps it’s a subscription to access premium content.

Open source enthusiasts feel that they are resisting controls over innovation and strive to obtain freedom to innovate in a way that’s beneficial to the customer, regardless of business case. It’s not anti-business but does challenge many existing business models.

Richard: In other soft product areas, like music, publishing, and stuff like that, you have a war between the copyright holders and those who think that stuff should be free. Do you think that closed source companies have a right to protect their source, or do you think somehow they’re subverting the greater good?

Marc: My opinion is that they have every right to do with their products whatever they need to do. If the software author also supports an open source movement, that’s a great thing but the open source model is not for everyone. In Eric Raymond’s "Cathedral and the Bazaar," there’s a whole section about reasons to open source, and that if your product does not meet all those criteria, then maybe you are better off keeping it closed. Often a solution stack is incomplete without certain pieces of software that happen to be proprietary.

For example, people buy a solution stack that includes Oracle and SAP because they derive value from the closed source applications, not because it’s built on top of an open source infrastructure. The open source operating systems (such as Linux) offer great features too, but today, most OS licensing and services revenue is driven by the application workload inherent in the solution stack the OS was bundled with, not necessarily the OS or benefits of open source.

Richard: So, you don’t think open source is in opposition to closed source, but rather just a different paradigm?

Marc: Yes.

Richard: OK. That’s cool; that’s interesting to know. I was trying to run down project-development leads on OpenOffice.org, and I think a little bit over half of them are employed by Sun. I’m curious, speaking non-altruistically, what does a company gain by sponsoring open source?

Marc: Besides the points I’ve already covered, they gain favoritism as an open-source-friendly company, that developers will even develop for their closed products because they believe they are contributing to the greater-good effort.

Richard: So, it’s sort of developer PR at some level? I’m sure not all of it, but there’s some, "We’re good guys, look, we’re sponsoring."

Marc: From a business perspective, that’s right – that’s what many of them see. The ability to leverage innovations from the community is also attractive, but the contributions aren’t guaranteed in the same way that the developer PR is.

Richard: That’s interesting.

Sean: For about seven years I’ve been involved in deep, technical efforts around evangelism, and I’ve certainly seen closed source companies realize and economic gain from sponsoring and supporting a community.

For a company that’s got a mainly closed source model, and they embrace open source model on the side, do you have any kind of anecdotes or examples of places where the company looked back and saw gain?

Marc: Probably the closest thing that I can think of is Adobe’s decision to allow projects like OpenOffice.org to include PDF generating tools, and their decision to make PDF a completely open specification that they encourage other people to put into their products. This has benefited Adobe’s PDF software products, even though Adobe’s products are closed. So, by creating an open standard, they’ve enlarged the market for themselves, and by supporting open source efforts, they’ve gotten a lot of mileage in the market that they sell into.

Richard: Last week Adobe open-sourced Flex, which was a free product. What, do you suppose, they gain by open sourcing it? Do they gain lower development costs, because they’re going to get free collaboration?

Marc: Well, I’m not very familiar with Flex, but suppose that they did the same thing with Acrobat Reader, which is another free product that they give away. They have already developed the market to the point that PDF is ubiquitous. If you want to make a document that is guaranteed to be readable on just about any platform out there, whether it’s x86, Apple, or a smart phone, you generate it in PDF.

If they decided to make an open source Acrobat Reader, that allows a lot of projects to tap into it and borrow code from it; it promotes the whole integration story of plugins and addons that go beyond the original plugin interface for the application. By giving their developer community the freedom to innovate, the development direction would be managed by the community.

You asked about lower development costs. The cost only goes down measurably when the community contributes something that the “copyleft” holder was going to pay for. Open source software lets a lot more innovators in the door to help with software development, but sometimes additional resources have to be found to do the work that no one contributes for free. The point that the open source community is trying to make is that open source innovators are shared among many open source projects. It would be expensive to recruit each of those developers under contract, and even more expensive to maintain that software over a long period of time.

Indeed, many projects have benefited from the open source model. Open source is not a guarantee of success any more than having great technology. There has to be demand for it, and from the right audience or no one will show up to do the work.

Richard: I see. A couple weeks ago, I ran into the CSI company that was started by Stuart Cohen. It’s sort of open source. The paradigm is that CSI hires developers, and managers as part of a geographically centered development teams. Companies who want a particular piece of software pay for CSI to manage that effort.

The idea is you might have half a dozen companies who want to have some software package. They pay CSI to develop it, and they also may make some of the developers on their staff available to the management of CSI to contribute to that project.

However, the thing that interested me the most is that was that once the software package is developed, it may not be re-licensed as open source. Rather, it’s going to be the decision of the contributing companies how to re-license, or even whether to re-license it. The presumption, I guess, is that they’ll go ahead and let it be available through some sort of open source model, where it can be modified and redistributed without cost, but they don’t have to.

It looks like a blending of open source and closed source. You’ve got this shared cost and collaboration of open source, but you don’t necessarily have the other main underpinning of open source, which is that the source code is available to everybody.

Do you think that that’s where we’re going, into different sorts of paradigms that blend proprietary and open source methodologies, and do you think that’s good or bad?

Marc: Though they would prefer everything to be open source, open source developers are beginning to accept that there are certain things that are difficult for a company to expose without getting them into potential legal trouble. They don’t like it, but they’re understanding. Overall, I think it’s a good thing.

Richard: So, it sounds like open source is drifting away from highly idealistic to more pragmatic; still holding onto their ideals, but being more pragmatic about attainability in certain situations.

Marc: There are some exceptions, but for the bulk of the open source community, yes, they’re becoming a lot more pragmatic about things. They still prefer to hold onto certain ideals, and continue to direct everything in that direction, but most developers are fairly pragmatic about it.

Richard: So, I had a couple of interviews with some guys at Microsoft focused on ensuring security for Microsoft products: Michael Howard and James Whittaker. Both of them, separately, said that the biggest advantage that closed source has, in almost every area—architecture, implementation, quality assurance, and documentation—is the ability to mandate a process. Microsoft has the ability to say, "This is part of the job, and you have to do it," whether it’s unit testing, adhering to coding standards, security reviews, or whatever. The implication was that would be hard to put together with open source because all you have is the code, you aren’t certain of the steps the developer went through to produce that code.

Is there good adherence to process in open source projects? Is it something that’s improving? Can you comment on that area, in general?

Marc: The open source community has code control procedures and models of their own. For example, in the LinuxBIOS/OpenBIOS project, you can submit a patch to the mailing list only once you’ve gotten one of the key maintainers to review and approve the code. So, there are still very much the rules for developing that code and processes to be followed. Some of these methods probably emerged from source code control models originally used in closed source settings, but in the spirit of open source, why change or reinvent what’s already working well?

The rules for submission seem to vary a lot for each project. You can have quite a range of different quality control mechanisms. But, usually, there is a quality control mechanism there.

Richard: Yes, there has to be. You can’t release software unless you have some kind of quality control. It seems like you have to have some sort of gatekeeper over the code base, or it’s going to deteriorate in a hurry.

Marc: Without someone moderating the process, you end up with wars of someone changing a bit of code, and another person changing it back. Endless back and forth. Code control also guides the direction of development so that community goals can be met.

Richard: Yes. So, what to you see on the horizon? What do you think is in the near future, generally, in open source?

Marc: A lot more of, as you put it, the blending areas particularly as Linux becomes more widely used in different server and desktop environments. For example, in multi media areas we’re seeing a lot of companies that create set top boxes that use a combination of closed source and open source software in embedded devices. It’s an area where businesses are realizing there’s money to be made with the combination of open source software and closed source IP.

Virtualization is a big area of expansion. A lot of companies want to make money off of virtualization, sometimes for security, and sometimes it’s to increase the range of compatibility across operating system environments. There are both closed and open source options, such as KVM, VMWare, and Xen. Each innovates using different philosophies. So, this is an area that I’m watching with interest.

Device drivers are also an area of much attention. The Linux kernel community has been very emphatic that they will not accept any closed source code into the Linux kernel. And yet the way that you develop drivers for Linux is to include it in the kernel. For example, the FCC has given 802.11 wireless networking devices a certain range of frequencies that may be used. Many of those devices are essentially programmable radios. The hardware vendors don’t want to open source that because that allows someone to then misuse the product and gain access to frequencies that the FCC wouldn’t approve of. Sometimes in order to reveal specifications, licensed IP used in the hardware design has to be revealed.

The drivers I’ve just described can never be part of the Linux kernel, and yet Linux users want to be able to use all these devices. In recent months, just really January of this year, certain key people in the Linux community are starting to realize that this prevents a lot of hardware companies from participating in the way that they would like to, and that though hardware vendors have nothing against open source models, for one reason or another they simply cannot release their code. I’ve been watching a number of efforts to make it easier for hardware developers to produce Linux drivers. I hope this topic is discussed during the next USENIX Kernel Summit and that the maintainers find a way to prevent Linux from being stifled further on the basis of compatibility problems with new technologies.

Richard: Do you see Linux making inroads into the desktop market? Do you think that they can gain a significant part of that?

Marc: Client computing models are changing from traditional desktop and laptops, and with these changes there are new opportunities for Linux. Different types of desktop applications or use models also offer an opportunity for Linux.

I think in time we will see Linux emerge in the same way – having strengths in certain types of client devices. I don’t know what that strength is going to be yet for Linux but many people in the community are exploring what it should be. Today, most ISVs only want to deal with one desktop operating system, and that’s Windows.

Today there are a number of ways, including virtualization, that you can run Windows apps in a Linux environment. As that improves, I think we’ll see a lot of ISVs soften to that idea. And, as they look at some of the things that we’re developing for Linux, they may decide that it’s better to develop a Linux app that will also run under Windows than it is the other way around.

As the Novell-Microsoft Interoperability Lab continues to grow their efforts I can definitely envision Microsoft having to run Linux apps inside their environments much the same way that we have those today in Linux.

Richard: It seems to me that products like OpenOffice.org and other open source products that are becoming strong, mature applications in the traditional desktop applications space have to work to mitigate Microsoft’s grip on that market. In the past, when Windows first started to take over the desktop, it was because that’s where all the applications were. You could put Linux or something else on there, OS2 or whatever, but all the mature applications were Microsoft applications, and they had to run on Windows. And I think that’s sort of how they got that grip on the desktop market. It seems like the first step toward making that a more competitive market is through the applications phase.

Marc: Yeah, I agree with that.

Scott: It looks like the TCO battle has been fought to a stalemate. Microsoft has Microsoft sponsored reports saying they have lower TCO. On the Linux side, there are reports sponsored by Linux supporters saying that Linux has lower TCO. Do the communities find any of these reports credible? Have you come across any truly independent TCO reports that you’ve thought were especially good? Is it even reasonable to expect that the finding of some TCO report will be borne out in the specifics of your organization?

Marc: In the independent studies I’ve read, per machine, the costs are pretty much the same for a supported machine if you compare Windows to a truly equivalent Linux implementation, but this fails to take into account the workload processing power costs. In the open source model, I have visibility into what code is slowing down my e.g. mail server, exclude the things I don’t need to have running, and make the machine more efficient. Without that optimization, more machines would be required for the same workload regardless of the operating system in use, which adds to the cost.

Richard: Marc. Thanks for taking the time to chat with us.

Marc: Thank you.

Bookmark this: