Interviewee: Alexandre Julliard.

In this interview we talk with Alexandre. In specific, we talk about:

• Developing an open source project in cadence with Windows
• Philosophies about how Linux should reach out to Windows users
• The leadership model used in the Wine project and deciding what goes in
• Deciding what applications to support and what to leave out
• The difficulties of working with the closed Windows API and Wine’s relationship with Microsoft

Sean Campbell: To start, please tell us a little bit about your background as it relates to Wine, your role as CTO with Crossover, and that sort of thing.

Alexandre Julliard: I’ve been working on Wine since the beginning of the project, 15 years ago, when I was still a student. For a number of years, I worked on Wine in my spare time, and then in 1999, I joined CodeWeavers to do Wine development full time. I’ve since become CTO of CodeWeavers where, essentially, all we do is build around Wine.

Sean: With a project of that longevity, what interesting evolutions have you seen, in areas like a community evolution or a usability evolution?

At the same time, as the Windows platform and the types of applications on it evolves, does that create a race condition of sorts, where you’re trying to catch up to everything that’s changing, and at the same time, adding new features to Wine?

Alexandre: Yes, mostly the evolution is driven by Microsoft and how Windows is evolving. And clearly, in that time frame, Windows has become much more complex. One of the major transitions we had to make was from 16 bit to 32 bit, which was very hard for Microsoft, and very hard for us, too, as we had, of course, the same problems.

The sheer size of the Windows API is, I would say, the major factor in the evolution of Wine. The first versions were small, self-contained binary code that just provided all the APIs. Then we had to make the code much more modular, to split things up, to define interfaces between the different areas of the code. We also had to start being very serious about regression tests. At the beginning, we were just hacking things until it worked. But, it became clear that it wasn’t scaling, and that we needed regression tests to be able to nail down the behaviors of the APIs, and make sure that when we add more stuff, we don’t break the old APIs.

That was a major factor in the evolution of Wine, and the fact that it’s really becoming stable is because of the huge test suite that we now have.

Sean: Given that you have to deal with the potential of really any Windows application running on Wine, you’ve got a massive set of applications, and I’m sure, lots of legacy applications that your users hope will still run, in cases where maybe the legacy application isn’t even really fully supported by Windows anymore.

But before I ask about that, I’m curious about what you are doing around 64 bit, given that you had a big evolution from 16 to 32. You take somebody who puts 64 bit Vista on their machine, and inevitably, iTunes will only kind of sort of work until they get it working, or there will be some kind of issue with their TiVo software package

Of course, those examples are just consumer grade stuff, but, there are also issues with drivers, with smart cards, and on and on. Have you started to see those issues creep up on the roadmap as something you’ll have to address? Or, is that just far enough out from your perspective that it’s still predominantly a 32 bit world?

Alexandre: Well, it’s clearly on the roadmap, but it’s not something we’ve had a problem with yet. There are simply no 64 bit applications that anybody wants to run that don’t have a 32 bit version. We don’t run drivers, so we don’t have the problem of the kernel being 64 bit. That’s not an issue for us; it’s just the applications.

Sean: That makes sense. Let’s talk a little bit about the QA process, then. Can you give us a sense of how many people are on the QA team? What’s the method of doing QA? How much user involvement versus core members from Crossover are involved in supporting Wine? Your relationship to Crossover and Wine seems a little bit analogous to the way Canonical is to Ubuntu, but maybe that’s a mischaracterization.

Alexandre: The relationship of CodeWeavers to Wine is maybe more like Red Hat to Linux, in the sense that we’re one of the contributors, but we don’t direct anything in the project. Obviously, I’m personally directing things in the project, but CodeWeavers doesn’t have a word in my choices with Wine maintenance.

Sean: That’s a good clarification.

Alexandre: As far as QA, we have several approaches. At the start, in Wine itself, we have a regression test suite, which basically performs unit tests of very specific Windows APIs. We test them one by one, call them with all sorts of parameters and test the return values and all the parts of the behavior of a single API. It is purely automated, so it is run after each change.

We also have higher level application level tests, and we do some of that at CodeWeavers. We have a team of three or four people who test the applications. We also have automated mechanisms, based on a tool called CXTest, which has been written by CodeWeavers specifically for Wine testing. For instance, it lets us run application installations in a fully automated way.

And then, we also rely simply on bug reports from the community. We have an application database on the Wine website where people can report what works, what doesn’t, and a public bug tracking system. We try to leverage the power of open source where all of these people are willing to help and everybody tests a different application and reports problems.

Scott Swigart: Linux is obviously becoming more popular, specifically on the desktop, which certainly must be good for you guys because of things like the ASUS netbook. It seems like there’s more demand for Linux desktops than ever. Certainly the vast majority of people are running Windows, but Linux is starting to penetrate into mainstream thinking.

I encounter two predominate thoughts. One is that we should make Linux as much like Windows as we can, because that’ll make it very easy for people to move from Windows over to Linux, and that’s the ultimate goal.

It seems like you guys are aligned with that school of thought. Wine and CodeWeavers exist because they recognize that some people are just going to have to run Windows apps as part of what they do, either at work or at home. Office is probably the most the common one that people want to run.

But there’s another camp that says, “The problem is Windows. We’re not going to solve it by making Linux more like Windows, and we shouldn’t we even go down that path.”

Do you ever encounter that point of view? What is your response to the idea that making Linux like Windows is a problem, rather than a solution?

Alexandre: It’s certainly something we hear a lot. Many people don’t like Wine because of that, because they think that Linux should get its own applications rather than using the Windows ones. I think that, with the number of Windows applications out there, it’s simply not realistic to hope that all of them could be brought to Linux right away.

The motivation for Wine has always been to make a transition possible. And once enough people get on the Linux desktop, then you create the demand for real Linux applications. I certainly agree that the goal shouldn’t be to make Linux just like Windows; there’s no point. Linux is different and should remain different, but there needs to be a way to enable people to switch, and that’s where we come in.

Sean: In a different area, you’ve been leading the Wine project for a long time–what has that position of leadership been like? In a lot of open projects, you see a certain amount of churn at the leadership level, or you see the project evolve into something else, which causes the leadership team to change. But you’ve had a very steady focus for a long time leading this community.

What kind of leadership style have you taken with the Wine community? Is it the benevolent dictator? Is it a very community-oriented decision making process? Give us a sense of what decision making looks like in the Wine community. Is it top down or bottom up driven? Because every community we’ve talked to about open search projects seems to be just a little bit different.

Alexandre: Well, it is clearly the dictator model.

Sean: [laughs]

Alexandre: I’m really making the decisions, and I think part of the reason people accept my decisions is because I’ve been around for so long. And I’m the only one who’s been around since the beginning, so I have an overview of the project that nobody else has.

I’m a big fan of the dictator model, not just because I’m the dictator in this case, but because I think it’s really the most efficient. It cuts down on discussions, on politics, on all that stuff that can cause a project to lose a lot of time and cause trouble between developers. I think it’s good to have a very clear way of making decisions and not to lose time over decisions, even if they may not always be correct.

Sean: That totally resonates with my entrepreneurial experience. Sometimes it’s better to make the decision that’s 85% right if you can evolve it later. You’ll at least have made a decision as opposed to just getting deadlocked and not making anything at all, especially if you can do 85% goodness while you’re waiting to fix the 15%.

On a different topic, what was the rationale behind the versioning scheme for Wine? Obviously, the project was in use for an extended period of time before it hit “version 1.0.” But yet, after some years of development, you felt it was time to go to 1.0.

Alexandre: The idea was to provide something that people can really use. Given the size of the Windows API, it just doesn’t make sense to release something like this only half implemented, because nothing works. Most applications really require large percentages of APIs to actually be working, so that’s why we didn’t want to create false hopes and release something that wouldn’t support anything beyond Solitaire.

Sean: So what led you to finally decide you’d reached the 1.0 milestone? Was is just that you felt you had covered enough of the Windows API, or was there some other instigator?

Alexandre: One part of it was support for enough applications that people find interesting today, which are not the same that people found interesting five years ago. There were also some questions of internal design and a number of things we wanted to clean up to make sure that we can grow from here without needing to re-architect things and break compatibility and things like that. So, we wanted to make sure that 1.0 was a good foundation for the future.

Sean: Tell me a little bit about the process of getting an application “supported,” although I realize that’s kind of a gray issue because certain things will work and certain things won’t? What’s the process around filtering out, “Well, that’s somebody’s pet application, but we don’t really feel a lot of people use it?”

Is that a fairly subjective judgment on your part? There are things like Office that are fairly easy to establish that everybody uses. But, there’s probably that second tier or third tier of applications that I imagine it’s a little hard to decide about, especially if it’s a vocal sub group that wants it supported.

Where do you draw the line between the core things you’re going to support and the things that you push off to the community?

Alexandre: Clearly, we need to go where the market goes and try to figure out what applications people want. But in general, it’s very hard to say, “I have this application and I want to make it work.” What we usually end up doing is to fix bugs all over the place, and then we look at 10 applications and we find out that three of them work and we decide that we aren’t going to support the others for now.

It’s really more reasoned by how well an application is already working than by a conscious decision of concentrating on one application instead of another one.

Sean: It sounds like sort of a mixture where there are certain things that you obviously want to work on, as well as other things where the decision is made by the amount of friction it takes to get it working and the amount of work you have to go through.

Alexandre: Yes. And the process of getting an application to work is fairly slow, so we depend a lot on the community and on people spending the time to make their favorite application work. Once they do that, it becomes an application that we support. And it is sometimes the case that one developer is the only one in the world who cares about this application, but that’s really enough to make it work and support it.

Scott: Talk a little bit about what some of the most important applications are. From my perspective, on the outside, it seems like Photoshop is something that comes up a lot, and both the older version and the current version of Office seem to be really high priorities for people. What do you see as the top five applications that everybody wants to run?

Alexandre: Office is clearly one of the major ones, though it has changed a bit. A few years ago, there was no discussion, everybody wanted Microsoft Office, and today, it’s a bit less so. People are happy with OpenOffice, and there is less pressure on Wine to support Microsoft Office. Photoshop is in high demand, as is Outlook (more so than Office as a whole), and games are fairly big too.

Scott: I noticed that there is a Crossover product specifically for games.

Alexandre: Yes, that’s fairly recent actually. We noticed that running their favorite games was something that people really wanted to do with Wine, and it turns out that most games are fairly easy to support, compared to something like Office. The range of APIs that games use is much smaller.

Scott: Outside of the 16 to 32 bit change, what was one of the hardest changes you had to go through?

Have you had the experience of waking up one morning to realize that Microsoft has released some update to “X” product, or a Service Pack, or a particular OS change that was particularly more painful? Can you describe for us one of the more challenging engineering things that you had to deal with as Windows was evolving?

Alexandre: The truth is that Microsoft cannot change things easily, because they are a slave to compatibility just as we are. We typically don’t suddenly find out that we need to support something that we hadn’t planned for.

The most problematic thing for us recently is that applications are starting to depend on features that are only present in recent Windows versions. For a long time, you couldn’t release a Windows application if it didn’t run on Windows 95. So we had Windows 95 compatibility and everything was fine. Now there are more and more applications that don’t bother with 95 compatibility, so we find that a lot of the architecture that was good enough for 95 support is not good enough for XP.

Sean: What do you think about the VMware Fusion and Parallels coherence and unity features, since this is starting to blur that line of what OS you’re running? People are starting to see the option, for example, of running XP on a virtual machine under Linux, and that’s where you load your Windows only apps.

What’s your opinion about actually running the app on Windows through something like Wine or Crossover versus using a virtual machine where your Windows apps go, even though you’re living in your Linux partition most of the time?

Alexandre: There are different aspects to that scenario. Clearly, no matter how much effort VMware and the others make, they cannot get the level of integration that we can achieve with Wine, where the applications can integrate with the Linux applications in all sorts of ways.

The other thing is that with Wine, you don’t need a Windows license, and you can be completely independent of Microsoft if that’s what you want. That’s something you will never get with VMware.

Scott: At the technical level, as you’re working to support an application, I’m guessing that you guys have implemented a large set of Win 32 APIs, so you start running the applications and see what does and doesn’t work, right?

How do you trace that, as an application’s running? What are some of the tools or diagnostics or whatever you use to figure out where it broke? Maybe it’s because the application hit an API that you haven’t implemented, or it hit an API that you’ve implemented, but it’s not 100% compatible. Talk a little bit about how that works.

Alexandre: The main difficulty we face when we run an application that doesn’t work is to find out why. We have some mechanisms, we have a lot of traces in the code. In particular, we have a way of tracing every single call that the application makes to the Windows APIs. We have an integrated Wine debugger in case of a crash and things like that.

Really, what it boils down to is, you look at the traces and you try to spot in the trace something that doesn’t look right. That’s pretty hard. You need a lot of experience to be able to find those needles in the haystack.

Once you have found something that doesn’t behave quite right, usually it’s trivial to fix. You just write a test case, see how it’s supposed to behave under Windows, and change the implementation. That’s really the easy part.

Scott: I guess that since you’ve always got the Windows API as a reference, it’s no mystery.

Alexandre: Actually, the Windows API is a mystery and that’s why you have to run all of these tests.

Scott: [laughs] Right.

Alexandre: And it’s the same way with documentation, it’s very incomplete, so the tests are the only way of finding out how it’s supposed to behave.

Scott: What kinds of interesting or surprising things have you learned about the Windows API? In some ways, you looked closer at the Windows APIs than just about any programmer who builds applications for Windows. As you have looked at it from your unique perspective, what’s been interesting or surprising about the Windows APIs themselves?

Alexandre: Well, I don’t know if it’s surprising, but the bad quality of the API is clearly the thing we take from that. Most APIs are buggy or don’t behave as recommended or just have some have corner cases that get triggered by some applications that nobody knows about, probably not even Microsoft.

People just write code and try it on Windows, and it works, and they don’t know why. Likewise, nobody knows why it doesn’t work on Wine, so we have to figure out why the code works on Windows. We actually find a lot of bugs in applications. Wine is a great debugging tool for applications.

Scott: [laughs]

Alexandre: The thing is that the Windows API actually goes to a lot of effort to hide the bugs. Whenever the application does something incorrectly, the Windows API tries to hide it instead of crashing or showing an error.

It’s good in a way that it doesn’t crash, of course, but it also prevents people from debugging their applications. So, we find a lot of bugs that if the Windows API were a bit more straightforward in what it accepts, the bugs wouldn’t have made it into the applications.

Sean: Have you seen any kind of evolutions over time, whether positive or negative? Have you seen evolutions in the API in terms of quality, or do you feel like you could sometimes get a sense of which way the wind is blowing with the Microsoft development process internally just by where you see them put emphasis on the API from release to release or service pack to service pack?

Alexandre: Yes. In the recent API, they have clearly made an effort to try to avoid security issues like buffer overflow and things like that. That’s clearly something they are now adding to the new API that they never cared about before.

Lately it seems they are more trying to spot bugs early and to crash when something is incorrect, instead of just hiding the bug. That’s definitely a positive evolution.

Scott: Have you seen Microsoft’s relationship to your project change at all over the years? I don’t think it’s any secret that Microsoft views Linux as a competitor, so they’re not really out to make stuff run better on Linux. What’s been your relationship, if any, to Microsoft, either as a company or individually–has anybody been at all supportive of the work that you’re doing?

Alexandre: Actually, we’ve not had any contact with Microsoft at all. We know that they are very aware of Wine, and they watch what we’re doing very closely, but we’ve never had any kind of official contact, either support or threats or anything. They just don’t talk to us.

Scott: They have just completely left you alone? They never sent the lawyers, but they never sent tech people to help you out either?

Alexandre: No.

Sean: We’ve reached the end of our time, but I wanted to thank you for taking the time to talk with us today.

Alexandre: OK. Thanks.

Bookmark this:

Interviewee: Leon Shklar

In this interview we talk with Leon Shklar – EVP of Media Technology for Reuters. In specific, we talk about:

• Moving Reuters.com to an open source stack
• Choosing and supporting open source software
• Making code changes to open source packages
• Licenses and governance in open source projects
• The perception and reality of open source companies

Sean Campbell: Leon, could you introduce yourself?

Leon Shklar: Sure. I am EVP of Technology for the Media division of Reuters (now Thomson-Reuters Markets).

I hold a PhD in Computer Science from Rutgers, and I have been doing some part time teaching there as well. That came in handy in team building – the team includes some of the former Rutgers students and other people working together through a couple of startups and other jobs.

I have been at Reuters for a little over two years. Prior to that, I was responsible for the technology side of the Wall Street Journal Online. I am pretty proud of what our team accomplished both at WSJ.com and here at Reuters so far.

To wrap up on the history, I worked for Belcore from 1990 to 1996. Before that, I worked with a small software company, called Computer Corporation of America, which made DBMSs for IBM mainframes.

In 1996, I took the plunge into the world of startups. We started a small company in ‘96, and then it got acquired in ‘99. Then everything went up in 2000 and came crashing down in 2001. When things started going south, I joined Dow Jones, and was responsible for the Wall Street Journal Online, from summer 2002 to spring 2006, which is when I went to Reuters.

On a related subject–with a friend who worked with me together in a couple of jobs, we wrote a book on Web Applications Architecture that came out in its first edition in 2003, by Wiley. The second edition is supposed to come out later this year.

Sean: It looks like it has some really good reviews. Scott and I have actually been lead authors on three technical books too. The joke my CTO gave me once was that “All authors like the act of having written. The act of writing, though, they are not so sure about.” I’d have to say I agree with that.

Tell me a little bit about the technology stack that Reuters sits on, in terms of how you guys use open source, what are the major products, and how that underpins the organization.

Leon: The responsibility of our team includes both consumer web sites and business-to-business media products. The former include the family of Reuters.com web sites–eighteen of them to be precise, for different countries and regions. The latter include text, pictures, and video products distributed over the Internet and via satellite. Overall, quite a mix of current and legacy systems, and not all of them are compatible. Our team is on a mission to bring all of media systems to a common platform.

Over the last two years, we migrated Reuters.com sites from Microsoft .NET technologies to the open source stack. There is some similarity in what we have done back at Wall Street Journal Online and here. When I started at Wall Street Journal Online, there were a bunch of disjointed pieces. The site itself was based on Vignette, there were some scripts, and there was a pretty big buy-in on the company side into Solaris and Sun and commercial tools. The company already had licenses for WebSphere and Oracle, which of course are pretty expensive.

I don’t think the open source components were as robust in 2002 as they are today, and the migration there was away from Vignette, heterogeneous scripts, and all kinds of obsolete CGIs and stuff like that, to a cohesive platform based on the Oracle tier, WebSphere, and a bunch of applications built on top of that.

The OS platform was Solaris. It wasn’t a cheap environment to run, because Solaris systems weren’t cheap. The Oracle licenses and WebSphere licenses are not cheap either, in terms of maintenance and upgrades. But we ultimately got the system to be pretty solid, by the time I left in 2006.

When I interviewed for the job at Reuters, the main question that I was asked, well, can you fix the site? It is not standing up.

Sean: Tell us a little bit about that. Was the issue page load times, or that users felt the site wasn’t responsive; what was the real issue?

Leon: It was a combination of problems. Response time was not very good, and the site wasn’t able to sustain the load at peak times. The other problem was that the implementation of changes was very slow.

The site was implemented as one big blob. To make even small changes, you pretty much needed to deploy the whole code-base.

That was a very difficult thing indeed. I am not religious about open source versus Microsoft, but one thing that’s true is that the bar for doing good work with open source is much higher. You need much better people. To build simple things with .NET is pretty straightforward. So you may have lower average technical levels on the team, when the requirement is just to write to code according to a functional spec. My personal feeling is that .NET by its nature is too permissive to bad coding practices.

Sean: Where do you feel .NET is too permissive in terms of coding practices, by comparison to open source?

Leon: Ultimately, all the door-opening in .NET is done through tools. There is not reasoning from first principles. You write functional code in tools and it gets compiled and then it all theoretically comes together. .NET is designed to do things for you, and that is a perfectly good thing when you’re building simple web sites.

It becomes not necessarily such a good thing when you’re building complex and scalable applications. By the way, that’s part of the reason why .NET is so successful–80% of sites are rather simple, and they don’t have an incredible requirement in terms of performance flow and even the frequency of changes. So for those, it’s perfectly good.

It’s not a very high bar that you need to pass in order to be able to write code for .NET. And it’s much easier to hire people, which means the team is not as expensive.

All that’s fine–it’s just when you get to something like 10-20% of the general population of sites, that you start running into real challenges in using .NET. I’m convinced that you can write good software in anything, and I expect that the implementation of MSN.com, for example, is done well (though you never know). I would guess that the people who are doing it have good expertise and knowledge of that technology.

Unfortunately, that is more of an exception rather than the rule. On average, the expertise level of .NET web development teams I’ve seen is considerably lower than the expertise of open source teams. With open source, you have a big challenge in building a really strong team. The advantage is that you have full control over the platform that you’re building, and ultimately, when you have a problem, you can reason from first principles.

I have seen the best .NET sites with very annoying intermittent errors in them, and nobody knows how to find them. The reason for that is, if the tools do help, there is not much you can do. It’s very difficult to look at elements of the code; it’s very difficult to trace HTTP messages through and to figure out, “Well, it’s this missing header that’s really causing this strange behavior.”

Scott Swigart: Talk a little bit about the infrastructure that you guys are built on now. Are you on sort of a LAMP stack?

Leon: It’s LAMJ.

Scott: So it’s Java instead of PHP or Perl.

Leon: Right. We have Linux boxes with Apache fronting Tomcat. The backend is MySQL with master-slave replication.

We have a number of application components that we built on top Struts. Of course, we built our own platform with our own application components, but we tried our best not to reinvent the wheel.

The content management system is something we had a lot of pain and discussion over two years ago, but we were not able to find the one that everyone liked. I’ve seen a number of commercial products, and generally, the problem with them is that the amount of effort to customize and support them is close to that of building your own.

Drupal was not there yet, and we ended up building a content management system, which is specialized to support our custom applications. Because it is specialized, it doesn’t pretend to be generic, and users are quite happy with it.

Scott: Mark Frons expressed the same kind of thing–he sees that as an area that’s ripe for some open source project to coalesce around, because right now everybody seems to be building their own CMS to some degree.

Leon: That’s true, and there are not very good commercial products either, that I know of. Vignette was trying to build a content management system; I don’t think they succeeded. There are a bunch of other companies like Documentum and Interwoven that have content management systems. They’re very generic, and they don’t appear to be very robust in what we need.

So what ends up happening is that if you use one of those, the effort involved in installing, supporting, and customizing it is as high (or higher) as the effort for building a very custom system of your own.

Scott: In terms of the stack that you’re built on top of–MySQL and Linux and things like that–do you rely on your own expertise and the community and things like that for support and figuring out problems? Do you get support from a vendor?

Leon: I would say that 95% of the time we rely on our own expertise and we rely on the information that’s available in the community. When we were first deploying MySQL, we didn’t have enough expertise, so we had to get help from MySQL. All the tuning they did for us was very, very useful.

We have a support agreement through UNISYS that covers MySQL, Apache, and Tomcat. We almost never contact them with questions regarding Tomcat, Apache, or MySQL, but when we need it, we have this backup.

Scott: One aspect of the value from their acquisition by Sun that MySQL talked about was that they had been looking at doing an IPO in a fairly turbulent market. And with Sun, they felt that Fortune 500s would feel safer about using MySQL if it is backed by another Fortune 500. I would be interested to get your take on that, as a MySQL customer with long-term involvement with open source products.

Leon: I am not necessarily excited about it. This is conjecture, but when I heard about Sun acquiring MySQL, it didn’t make sense at first. The only context in which it does makes sense is that Sun starts distributing MySQL with Solaris, because Sun is trying to gain acceptance for Solaris, as an alternative to Linux.

My concern with that, and one of the reasons we went with Linux to begin with, is that the open source tools are built for Linux first, and only then ported to Solaris.

Sean: It’s the LAMP stack, not the SAMP stack.

Leon: Exactly. So if MySQL is going to become part of the SAMP stack, sooner or later they are going to start doing development on Solaris and porting it to Linux.

Sean: Some people say, “Hey, the great thing about Linux, and Apache, and a lot of these kinds of things are that they are open source, so if I really run into a problem, I can get into the code and fix it.”

But other people say, “Well, I have no interest in doing that. I don’t want to run my own private fork of Linux or Apache. I use those as a product. There is value in the fact that they were built using an open source methodology, but they are not something that I am going to really go in and muck with.”

What’s your take on all of that?

Leon: It is not a good thing to create your own branch of an open source product. That said, if we are in a critical situation, it’s not completely out of the question to go into the code and put a fix in, and then submit it, and make it part of the community so that it goes into the next release. Ultimately, when the next release becomes available, you replace this little emergency fix that you made.

But going into the code is the last resort, and you would only do that if there is no other way. We have people that know (or can analyze) the code for open source products pretty well. At the same time, we do have support agreements with Unisys and MySQL, and we would rather they did that, if deemed necessary.

If need be, we can work with them to figure out what the problem is, and try to come up with some temporary fix or workaround. It is clearly not a sustainable scenario to constantly make changes to open source components. To be specific, making changes to Apache, Tomcat or MySQL code is not a good idea, except as a short-term measure.

Sean: It sounds like, in a practical sense, those situations just don’t really come up very often. The products that you are building on are mature, and you are not doing such exotic stuff that you are taking them into a use case where they have never been used before.

Leon: We try not to. These are fairly sophisticated products. When we first deployed our system, we ran into the kinds of bugs that required people to go into the code. Not to make a change, but to understand the code and to find a workaround.

Most of the time, if you have an exotic error, you don’t have to fix the code. Once you understand the implications, you probably can come up with a workaround.

Sean: So you are talking about having a really high bar for your staff, if people have to be able to trace into the code of the web server, and things like that. But on the other hand–like you said–the scale of the stuff that you guys are doing is way outside of that 80% kind of case for web sites.

Leon: Yes, it is. And let me put it this way. I don’t remember a scenario where we actually had to change Tomcat code. We never changed the base Apache code either, but then there are plug-ins that are distributed with Apache. We did change those, even the ones that are considered part of the standard distribution.

Sean: I suppose that the plug-ins are a little more special purpose than the core web server, and they are not necessarily used by everyone the same way.

Leon: I would also say that they are not as mature, because not so many people are using them. The number of people using Apache is enormous; the number of people using a particular plug-in can be two orders of magnitude lower.

Sean: Many times, people are using a lot of open source stuff around the LAMP stack, like components, unit testing tools, automated build systems, and libraries.

Leon: We use that as well–we use Subversion as the versioning system, we used Maven, and we used to use Ant. That’s pretty much the primary environment.

Sean: How do you evaluate which open source projects to bet on? If you just go to SourceForge, there is a lot of stuff there, some of which is kind of abandon-ware, whereas some is very actively maintained and being worked on.

Leon: You try not to use the libraries that don’t have a lot of activity around them. You also need to look very carefully at their license; you don’t want to use a viral license.

There are a couple of different licenses, as I am sure you know, for open source. One type lets you do whatever you want. Another type lets you do whatever you want, but you can’t resell it. Then there are a number of really bad licenses, where you use the code as you wish, but all the code that you write becomes open source.

And those, we don’t want to use.

Sean: We had an interesting conversation with the president of the Apache Foundation, and he talked about some of their tenets. One of them is that any Apache project has to have maintainers from at least three different companies, so that no one company can force the direction of the project a certain way.

Which is a little different from how MySQL works, or a lot of the Sun stuff works, where it’s very much corporate governance versus community governance.

The other thing that he talked about was that they strive for a very commercial-friendly license, where you can embed stuff with the Apache license in commercial software and you’re not required to open source your derivative works.

Leon: That’s critical.

Scott: On one hand, you’ve got open source, and then a sub-community of that is the Free Software Foundation, which is very much ideologically oriented around the idea that “the world would be a better place if everybody released their source code for everything.”

Leon: I don’t buy into that. I’m very much in favor of the commercial model, and I guess my conviction is very close to that of MySQL, because they do derive commercial benefit from open sourcing their code. They made a conscious decision; and they’ve got many people in the community contributing to the evolution of their code. At the same time, they obviously invested a lot of money.

In the end, their revenue is not coming from licensing–it’s coming from services, and they’re the primary experts on the database. Consequently, they have the ability to charge premium money for those services. I’m totally in favor of this sort of approach to open source where software is a commodity, and you get interested parties to contribute to its evolution.

Scott: On the closed source side, if you’re using Microsoft stuff or VMWare stuff or stuff from Adobe or whoever, do you feel like you feel like you have to look at the EULAs to the same degree?

Leon: With open source, you do have to pay more attention to the licensing. With commercial software, generally I haven’t found it to be a problem.

As a side issue, open source and closed source are not born equal. I wouldn’t use an open source component in a Microsoft environment. If I were forced to use a Microsoft environment, I would probably get a commercial database. If I had to build a system on the Microsoft OS, I would use SQL Server. I would not use one of open source projects, because I don’t feel they’re robust enough. There are not enough acceptances. There is not enough power and attention from the community. It’s very, very limited.

I would rather not run MySQL on Windows in production. I would most definitely not run one of those open source databases built for Windows, which are not MySQL. I don’t want to name them, but there are a number of them that were built for Windows and are considered open source projects, and they’re not so good.

Scott: For open source that only runs on Windows, the community is probably a magnitude smaller than open source that primarily targets Linux.

Leon: That’s one consideration, and the second consideration is that Microsoft is not the primary environment that attracts the open source community. I don’t want to sound like a bigot, but I do believe that the average expertise level of developers that work on Linux is higher than the average expertise level of developers that are working in Windows. And it’s not because Windows is not good and Linux is good. Windows is an excellent operating system that went through a couple of evolutions.

It’s just a completely different scenario. It’s relatively closed, and the conscious decision that Microsoft made at different times was to build something that targeted 80% of the development community, and it’s done well for them.

Sean: One of the reasons that I like my Macbook is that I can run any OS on it that I want at this point.

Yet they’re a closed source company, by and large. And they’ve got this massive innovation that they’re driving. I just saw the iPhone sales numbers this morning. Yet you take another example of a big company like Google, who gets a lot of credit for being open source, but they’re not really in a sense.

They’ve built their business on open source and they have some projects, but it’s not like they turn around and say, “Well, here are all the algorithms we use. Just go deal with them.” And then you’ve got companies like MySQL, that are purely open source, and they’re out to make a profit.

So I’m curious, as you look out across these different vectors, how do you think innovation plays into that? Because from my perspective, when I look at open source products, the more technical the product, and the more the product is by developers for developers, the more innovation seems to stream out.

But when they get into more commercial products like Open Office, to be frank, does it really have the same innovation level? That’s not to say that I’m in love with Outlook. I’ve got my problems with that too.

From the standpoint of acquiring technology in your role, when you’re looking for stuff that’s innovative, do you see a similar sort of breakdown, where the more technical the open source project is, the more likely it is to be innovative?

Leon: First, there is something very interesting that you said that I was kind of itching to respond to immediately. Think of Google, think of Apple. They’re not open source. But there is a perception of open source.

I’ll give you an anecdotal example from real life. We started using the Google appliance for search, and we were planning to deploy it and they had a bug that was preventing us from going live. And they actually guard their code and algorithms quite closely.

So we said, “All right guys, when are you going to fix the bug.” And they said, “Oh, the next release is in three months.” But that doesn’t work for us. We went back and forth and in the end they said, “You know, guys, we understand. We can’t release it sooner than that, so here is the code.”

I put a lot of value in that. It’s a different philosophy. Yeah, it’s closed, but Microsoft would never in their life do that.

Sean: Microsoft actually does make the Windows source available to certain customers, governments, etc., but I don’t think that’s widely known.

About the second part of the question, is there an innovation continuum in open source, where certain projects have innovation that is arguably better than in a closed source model, whereas in others maybe it struggles to really hit the mark and bring things forward?

Leon: You definitely have a strong point, and you gave an example before. I don’t think Open Office is necessarily innovative. I don’t use it. It has kinks still, and it’s too much trouble.

Sean: That’s an easy one to throw a stone at in terms of its innovation, but I want to get your sense as someone who has to acquire and find the right technology, where you think the soft spot is.

For example, the Mac is amazing for UI innovation and fidelity of user experience. On the other hand, go try and find a really good database product for the Mac.

On the flip side, I don’t’ think people have seen Vista as the paragon of great UI innovation, you know what I mean?

Leon: Just to elaborate a little bit on this example of Open Office, I don’t believe it’s quite fully an open source project. It’s heavily sponsored by Sun, and if it was strictly an open source project, the community would not spend this much effort in building an office product.

So there is a difference in an open source project that is supported by the community and a company deciding to open source something. When Sun open sourced Solaris, it didn’t suddenly become an open source operating system in the full definition of the word.

The fact that something is not closed source doesn’t make it open source. There is also a community element.

Scott: Right–who’s allowed to contribute and who controls the direction of it.

Leon: I believe that open source projects with strong community interest in them and strong community participation in them are all pretty innovative. Not all projects that are based on public code are innovative open source projects.

The other part of that is, taking Apple as an example, it is proprietary, but at the same time I can take an Apple laptop and open a command window and I can type my Linux command.

As far as business decisions go, catering to 80% of your audience is good, but it has a price. In the case of Windows, you don’t get a very strong following on the side of cutting edge developers. Because developers don’t love Windows machines. I don’t know a single one.

Scott: Oh, I know some extremely passionate .NET developers.

Leon: Ok, maybe it’s my bias coming through…

[laughter]

Sean: As usual, we have a lot more questions we wish we could have asked, but I want to be sensitive to the time, as we’d scheduled an hour, but thanks for taking the time to chat. This was a great conversation.

Leon: Thank you.

Bookmark this:

Interviewee: Britten Martin

In this interview with Britten Martin we talked to him about:

• The role of support inside Microsoft.
• A walkthrough of how a support request is handled.
• The ways in which the outside community can interact with the internal support processes.
• How a product is supported differently in beta than when it’s finally released.
• How a bug eventually become a fix in the product.

Scott:  Britten if you could just introduce yourself for the record. That is usually a good place to start, and then I’ve got an initial question after that.

Britten:  Cool. So, my name is Britten Martin, I am a group manager within CSS which is Customer Service and Support. I’ve been with Microsoft since 1999 where I came in as a support engineer supporting Windows 2000. So I have been a support engineer, I have done some work with our outsource partners in a partner technical lead role, then I moved into frontline management and did that for about three years. In my current role I am a manager of managers, I have an organization of about 75 people and I am the global sponsor for System Center Support, which is your SMS and MOM products.

Scott:  So talk a little bit about what falls under the scope of support inside of Microsoft. I mean, engineering rather has its domain, marketing has its domain: what is support defined as and what are kind of the responsibilities roll up under that inside Microsoft?

Britten:  That is a good question. It really depends on which segment of support. It is a great clarifying question, Scott, because I am within Commercial Support which is the enterprise-based support, your back‑end, your servers, many of your big customers that buy dedicated support contracts, the government, many of the major companies around the world. We also have certainly have a consumer end which is your home user running Vista, Office, also Xbox, Zune. Any product that you can think of that is released by Microsoft has support. So really what it is, the thing that separates, I think, Microsoft from other companies in that, is that no matter what’s out there, you can pick up the phone, call 1‑800‑MICROSOFT, tell them what your product is, and you’ll get through to someone to support your issue. Although it certainly depends on the customer segment that you are in, who you are routed to, where you go and how much it costs.

Scott:  Sure.

Britten:  In Commerical Support, we support customers who have technical how-to questions about the products and certainly those who are getting error messages.† Also, if you need help setting it up, if you need more consultative type services, we do that as well.

Scott:  OK. Then one other thing I was somewhat curious about too is ‑‑ one thing I have always been curious about I guess – is how a problem works its way through the cycle.

Britten:  OK.

:  So, you start out when the phone’s ringing and you have one customer, two customers, and three customers: they seem to be having a similar issue. How does it work its way through the process from the technician helping them on the phone, to maybe later on deciding it needs a KB on it, to maybe a hot fix, to maybe being rolled up in a Service Pack?

Britten:  That is a good question too. OK. I will speak in today’s terms because major things have shifted over the last couple of years. There used to be a lot of process behind just what you were saying, which is, you know: "When do we do a KB? How many issues before we make a hotfix?" Nowadays with the advent of blogs, which is kind of, what you are doing, you know, what this interview is about, in support we have become a lot more proactive with getting immediate information out to our customers. I will give you an example from my group, within System Center: last week we had an issue with SCCM, the new version of SMS. Just like you described: one call, two calls, three calls. All of a sudden we are like, "Wait a minute, there’s something going on here!"

So we got some information together, dumped it out into our blog so it was out there. On the back‑end our support engineers, who are your front‑line resources, are working with the escalation team, our escalation engineers who do the debugging, who look at source code, who work with the product group. They are debugging the issue, and they get it down to, "OK, this is clearly a bug." At that point, that escalation engineer has a hotline per se to the product group, they pass along all of their research, they are reproducing the steps, reproduce the issue, send it on to them, all the documentation.

Then it is kind of in the product group’s hands to decide, "Is this something that we need to fix now. Do we need a hotfix, is it just something that we can put documentation out." You know kind of that form of decision making process. In this case it was a major issue, it was a deployment blocker, huge customer pain, so they released a hotfix. Then once again, depending on the urgency, is it a publically available hotfix or is it something that you have to call support to get a hold of? In this case, we knew this was a major issue so we made it available out on Microsoft.com, put a KB article out there, and linked the hotfix to it. Therefore, customers could immediately just go and download that hotfix.

Down the road certainly that’s going to be in the Service Pack 1 or similar, so you can pretty much figure for all of our major products, any bugs that we fix, or any hotfixes we release up until, there’s a cut‑off date, will make it into the Service Pack.

Scott:  And then talk to me a little bit about the timeframe, right, between the time where maybe the first few calls came in, until the time that hotfix and that KB article are out there.

Britten:  Well, once again it is very, very dependent upon the criticality and the widespread nature of it.

In this case from first call to KB/hotfix was about five days. In addition, some of those delays ‑‑ they had identified the code, they had written a hotfix within about two days, but there were some delays, you know, going through legal on official KB review, and getting everything up there and replicated. So from beginning to end, about five days.

Scott:  Yeah, that is fast. I do not think most people would ever think of Microsoft turning something around that quickly.

Britten:  Some times, I think Microsoft is seen as very slow to respond, but in reality, itís often not the case.† †If it is truly a deployment blocking issue or something that causes widespread customer pain you will see a quick turnaround on it and you will definitely see the documentation out there in a timely manner. I have been here eight years, I have seen issues where it was tough to figure out what the fix was, what the resolution was. Where it was something that would hide itself into multiple components so it was a complicated fix to write, it takes a little longer. However, if we can identify it, if we do a good job on the support side with identifying it, providing reproduction steps to the product group, you know, they can turn it around quickly.

Scott:  So one question then too, would be, what does the triage process look like if after reproducing the bug you quickly realize that it has security implications versus something that’s a deployment blocker, which might be the next layer and then further down which are just kind of usability or general problems in using the software?

Britten:  Yeah, I think, once again when you are running into security type issues what you get with that is even more resources thrown at the problem. We have security aliases internally, they have product group members, and they have certainly a high-level support presence as well, that if something comes out we can fire it up and quickly. There †is a whole process that is enabled. Let us say there is a new Sasser or Blaster or any of those. We have processes in place that will quickly call together members of support, product group. We have open lines 24 hours a day. It is kind of the war room type mentality.

It has product group members on it. It has support members. So people on the field can call in and say, "Look, I have got an issue with this bug or with this security hole and just want to make sure I have the up to date information". So they can call into this grid, they can get the information.

On the back‑end, it is hard to say how long it is going to take, depending on the fix but you are going to have even more resources available. You will see a fix as quickly as we can get it out there. After the Sasser/Blaster/Nimda pain, we really turned the corner not only on product development but also on the support side to ensure we could have a quick turnaround.

:  What do you think the level of community involvement is in the support process because obviously that is a huge piece of the open source model. Right down from many eyes looking forsecurity bugs to a very well populated mailing list where there is a high discussion level, traffic, and things like that. However, at the same time with the closed source support process while the team is staffed potentially adequately with full time employees, there are general concerns out there with a closed source model in terms of how much the community can get involved.

So what are the avenues the community can actually affect the support
process at Microsoft and/or see what is happening under the covers of the support process?

Britten:  Yeah, I will tell you that it has really been something we have been working on for the last few years. The product groups and the support teams have been trying to find ways to tap into that community model because there are many folks out there that do our support day in and day out with customers that have a lot of knowledge that could be helping the broader community.

So, one of the things that we have is the MVP program, the Microsoft Most Valuable Professional. One of the things that these folks do is to actively take part in the public newsgroups for our products. I can point to a couple of products ‑ Small Business Server is the poster child for that. It has a very strong community. Customers can go up to the Small Business Server forums and post problems and the MVPs will quickly get involved to help resolve the issue.

There is a lot of energy around building a stronger MVP community and empowering the MVPs to help solve customer issues. Many of the product groups have CPE, Customer Partner Experience, teams, that are really focused on tapping into that partner community.

Another example from my world System Center Essentials, or SCE. We actually launched support for that on a complete forum based model, where we had engineers from CSS monitoring the forum, and we worked to build strong partner participation. If the partner could not answer the question, or did not answer the question within 24 hours, someone from support would jump in. We are trying to grow more community presence around support.

Globally, there is more partner presence out there as well. A good example is SoftGrid or Softriciy, a recent acquisition. In Europe, before Microsoft took over, there was not really a ton of dedicated support. It was all done through the community. Therefore, we have been looking and trying to learn from them on how they involve the partners, and how we can keep that type of interaction going. I know there is a log of energy around this and it is a
huge initiative.

Sean:  Well, one question which is really back to the second part of the original question which, is that a lot of the product groups in Microsoft are being affected by the open source model.† For example recently we talked to Shawn Burke who has been instrumental in getting the source code† for the framework out the door. How much of push to learn from the open source community is affecting the support team in general? For example could you see a day where the status on bug fixes is more open and less a black box until the actual fix arrives out on the web?

Britten:  So thinking more about ‑‑ I am partner, I am out there. I find a bug or a hole, I put it out there, and really being more transparent if that is going to be fixed. Is that kind of…?

Sean:  Yeah, exactly. I realize it’s not going to be a plate glass window ‑ it never is. But at the same time I think a lot of people will look at support right now and say, once I send in my support request I don’t really have a lot visibility of how they are stacking it, ranking it, prioritizing it, and where it goes, until maybe I get an email that says you have been deferred. Right?

So I am just curious, I know a lot of the product groups specifically are trying to expose a fair amount of their development processes, so I was curious if maybe that was making its way over to the support side.

Britten:  You know it is a fair point. Let’s say you are running a beta version of the product and you are involved in the one of the official beta programs, such as TAP ñ our Technology Adoption Program.† †The beta version is fully supported and you can actually go and directly file a bug. In addition, I know that they try to make a lot of that process transparent, so you know if the bug is a duplicate. Is it being fixed? Is it being deferred to the Service Pack 1…whatever. So actually, I know on some levels we do that.

On the support side, I think we make good use of our blog. There is a huge blogging effort, because there is a lot of knowledge within our support engineers and within our internal databases. For one reason or another that has never been made public. So I think we try to do some of that as well. But I can see the point. I mean maybe someday and maybe that some day is not too far away where you can log in and say, here is a bug that I submitted. Alternatively, even better for Windows, here is the bin of things that are going to be fixed in the next two weeks, months, whatever.

Sean:  Exactly

Scott:  Well, and so you know not every support request anymore comes in through a phone call, right? There are things like the Connect site. There are things like Windows Error Reporting services. I guess if you would not mind talk for a minute about some of the different channels that, a problem can kind of come in through and… Go ahead.

Britten:  No, no, it is fine. So today, and if I use Americas Commerical Support as a kind of a measuring stick, somewhere around 85% of our support issues come in via the phone ‑ very traditional means of operation. We are really trying to push more of our online type offerings. One is online submission where you can go up and create your issue; you can attach data, submit it to a support engineer. That support engineer reviews that data and calls you back, hopefully with an answer.

The Connect site is a lot more beta focused at this point. I am not an expert there, but that’s where you get into many of the cool things where you can actually submit bugs and all that. We also have the managed forums, which we talked a little bit about. The managed forums are a completely different avenue. Most of those are staffed by Microsoft support engineers as well as product group representation and the partner community.

So you have an issue with Windows, you can go up to a specific Windows forum, post a question. You might get an answer fromsomeone in the product group, a partner, or a support engineer. Therefore, you have those as well.

On the consumer side, they do really cool things with automated responses. Therefore, you submit an issue and it checks your data, checks your error log to see if it is a known issue. Before you would hear back from the support engineer, you might get an automated response with, you know, try these two things. If this does not work respond back and the engineer will contact you, then as you said, you had the whole Watson or error reporting. The error reporting stuff is very cool, because regardless of bugs filed or anything else that goes on, all the product groups take all of their Watson or their error reporting data and they make sure that as they look to roll out service packs, with new versions of products especially, they want to handle those top 15, top 20, top 50 Watson issues. They want to make sure that they are knocking those off.

Therefore, it is an interesting mix of many different avenues for customers to submit data, and in some cases, a Watson log, the error reporting, customers might not always think they are really contributing to the next version of the product, when in essence, they truly are.

:   I realize, too, that you are right, that is largely a mechanism for when a product is in Beta. I guess, I have two questions, but one is a follow up on this thread initially. Talk a little bit, I guess; about how a product is supported differently while it is in beta, versus once, it has been released?

Britten:  That is a good question. We have a few different types of beta opportunities for customers.† There are more formal relationships, such as TAP (Technology Adoption Program) and RDP (Rapid Deployment Program).† With these formal programs, customers agree to deploy this product into some type of production environment. In return, they get access to submit bugs, they get access to product group members, and they get
access to dedicated support engineers.

From within support, let us say we have a new version of Windows, we might have three or four support engineers in the US dedicated to this beta support. These engineers get a hold of the product early, they get access to product specs, they visit the product group, and they may even get to go out on site to customers who are deploying this product so they get some real world hands on experience.

Now, that is for the structured beta program. If you’re not a customer in one of these programs, you likely can still download beta copies of the software. Even in these situations, there is support out there, through the forums that we talked about before. Once again, you probably get answers from those same beta engineers and some of the product group folks. You just do not have that one-to-one interaction in a lot more, you know.

Does that make sense?

Scott:  Yeah. Yes, very much so. One of the other things that ‑ I will kind of circle back to something you said earlier – If it needs code to solve the problem, you know, if it is not just configuration file that was messed up or any one of the myriad of things that could cause a problem. You know, some registry key got corrupted or whatever. However, it actually needs the coding hot fix, you could hand it over to the engineering, send it over to the product team and they triage it, compared to other stuff.

I mean, have you ever run into situations it seems like this would inevitably happen where they’ve got their engineers dedicated to building the next version of the product, and meanwhile, there’s an endless list of bugs in any complex software that they could be tasking engineering resources with. In addition, it seems that they kind of have to make that call about pulling people off engineering new features to work on a hotfix or something like that.

Britten:  Yeah.

Scott:  And how does that process of triage work? Because support is kind of on the front line with the customers, it seems like sometimes you guys might have to go back to them and say, "Um, you know the time frame that you’re looking at fixing this in probably isn’t really going to work, so think about it again."

Britten:  Right. In addition, I will tell you, it is really dependent upon the product group. You look at something like Windows. Windows has a whole team, and you hear it from time to time called Win SE. ,Windows Sustained Engineering. They have a whole team dedicated to current version, and whatever supported versions are out there, on the hotfix front.

With some of the smaller products, you may run into situations where they have to pull developer resources off the next version to fix current version or legacy version problems. However, I tell you, in my time here, I have never seen that as being a blocker in getting something fixed. If there is truly an issue that needs to be resolved in the current version of the product, especially if its security related, you can be assured that they will find someone to handle it.

However, as a general rule of thumb, all big product groups have sustained engineering, folks that are dedicated to current version problems, or legacy version, so it is not necessarily that huge resource problem that you would think.

Scott:  OK. Therefore, many times, they are not pulling people off features. They have people who are dedicated just to the maintenance and hot fixes.

Britten:  Right. That is your sustained engineering folks, for the most part. As I said, I think you would run into some smaller product that might not be in that same boat, but the major ones have sustained engineering.

Scott:  It seems, too, like support, it is a little bit of a difficult situation, because, I guarantee, if I call with any bug, it is a critical show‑stopper. The world will end if it does not get fixed. So how do you kind of manage customers, who are generally going to feel the importance of their bug is high, but Microsoft does not necessarily determine it so in the grand scheme of things?

Britten:  Yeah. I tell you, our people are experts at conflict management and really setting customer expectations, and certainly the empathy piece as well. I mean, you have to be empathetic with the situation, but you also need to be very transparent and upfront.. It is saying, "Look, here’s where the bug is at this point. It is something that may or may not get fixed. Here’s the timeline."

With many of our customers in the Commercial space, these customers also have account managers that are Microsoft employees that really serve as the liaison between support and the customer. Therefore, we do a lot of work with the account managers to help us on the customer maintenance side of things, when maybe the messaging isn’t as good as we’d like it to be.

However, frankly, all major problems end up getting fixed. They might not get fixed in the period the customer would like to see it, but, between a hotfix, the next service pack, and the next version of the product, all issues do end up getting resolved.

Once again, it is really just dependent upon how many customers are having the same problem. Is it something that other customers could run into? In addition, really prioritizing, like you said before. I mean, the product group, while they have a sustained engineering team, has a limited number of resources, and they need to prioritize what are the big-ticket items that need to be worked on.

However, that skill of conflict management is one of the key pieces for successful support engineers, because, every day, when we answer the phone, when we respond to an email, that is generally that networking administratorís worst day of the year. The servers are down. They are catching heat from their management.† Something is going on, they are being yelled at. Our engineers deal with this on a day in and day out basis and as a result, they get very good at handling the heat.

:  So one question I am curious about is, how do you deal with it when the support engineer knows that it is really a usability issue and not a bug, sure there is a sequence of phone calls and bug reports, but it is pretty clear that this isn’t a bug, it is more of an issue of usability. And it might be in the land of the undiscovered feature right? We were told, if I remember, some person in the office team told me that something like 90 percent of the feature requests for office, are already in the product, but nobody knows the feature is in there. So how do you people educate the product teams and what does that flow look like? How do you finally classify that, "OK this isn’t really a bug, it is an issue of usability?"

Britten:  That is a good question.

Sean:  Because obviously the product team, like everybody else is blind to what they build, right? Of course, it is useable. Because I built it, I mean every developer has fallen to that problem and we all have seen some pretty horrible UIs over the years. So how does the support team educate the product teams on this point?

Britten:  Yeah. In the past, if you back up five years, 10 years, support was as you said an island. We were out here, taking all the heat and there was limited interaction with the product group. About five years ago, things drastically changed.† we started doing what is called supportability in our world. We review what are the top issues, how long does it take us to solve those issues and how much do they cost the company for us to support these issues?

Therefore, we came up with a list of the top 10 issues. We then present that back to the product group. Now the interesting thing about that is the product group actually funds us to do support for their product. So, when they see that a problem in their coding is †costing them $800,000. they to take some notice , and say wow OK, what does right look like, what are we missing here?

Sean:  Well, just a quick follow up to that. What are some examples of things where you feel like you touched the product, and you made a real delta that the customers ended up bettering from, but at the same time you folks probably would never personally got credit for it

Britten:  That is right. I donít have a list in front of me but DNS is one that consistently scored high for Windows 2000 and even some into Windows Server 2003.In addition, there was so much push and so much interaction that there were some major changes that happened later in 2003 and into 2008 that have drastically changed the way DNS works.

Another one I can give you that clearly had all the supportability people driving this was terminal server licensing. Terminal server licensing, when you go back to 2000 and once again, kind of into 2003 with a complete and utter nightmare for something that should be very simple. Consistently that silly process ended up top two, top three on our supportability list for Windows Server. And some very small tweaks based on advice provided by support completely changed this, dropped out of the top 10, out of the top 20 in support issues and saved the company a lot of money.

Those are two examples of technologies I know we beat on. However, on all the products out there, SQL, Exchange, Windows, SMS, MOM, they had the supportability groups that review the top 10 issues. In fact, I was just in Redmond last week, looking for the SMS side and they do cool things with it. I mean they look at top 10 issues for previous version of the product, look at the top 10 issues for current version, and review whether or not they fixed the top 10 call generateors? If not, what do we need to do to make sure it is in the next version that we do? Support is a huge data mine that we are just starting to realize the power of all the data that we have.† We are a great voice of the customer.

Sean:  One quick follow up to that and then Iíll turn it back to Scott. I am curious because you mentioned that five years ago there was some type of a change, right? In addition, all of a sudden, you folks were a lot more integrated. What drove that? In addition, I realize with all changes ongoing, there is probably stuff you can say and stuff you cannot, but I am sure somebody reading the interview is going to be like, "Wait, wait, and ask him that." So what drove it at a high level?

Britten:  I think that is fair. I wish I had some great and gory backroom drama to share, but if it happened, I just donít know the details.† In my opinion, you know, part of it was just the fact that, wow, especially on the Enterprise side, Windows Server 2000 started to really turn the corner and become a huge enterprise related product. People were out there buying it and implementing it and someone on the CSS side as well as, I can guarantee you, the Windows product group was like "We need to tap into this.î

Around this time, I was a support engineer and I suddenly realized that the product group was really listening.† In all businesses you started hearing about some monthly conference calls where we were there to provide feedback on top issues.

It was a time when we were moving from being just a consumer, home user company to being a legitimate player in the enterprise. In addition, there was a lot of pressure at that point to make sure these products were just better, and I think someone realized "Wow, CSS has a lot of knowledge out there; we need to tap into it."

Scott:  Well, how much now do you guys get involved in product development from the standpoint of, you guys are looking at the product, while it is in development and saying, "You know what, if that error happens, there is no way we will able to trace it back and figure that out. We need instrumentation here, we are going to need these things, spit it out to a log file, we are going to need these kind of support features built into the product to be able to kind of quickly
and officially troubleshoot things."

Britten:  That is a huge piece of our beta involvement. I know we talked about it before. For each beta, we dedicate engineers from our support teams to the pre-released product. Well, one of the way they make an impact is by providing that ongoing feedback to the product group. They are not only working with beta customers, they are not only answering newsgroups, but they are also messing with the product itself.

Therefore, they filed their own bugs. They have supportability war room meetings where they say, look there needs to be some logging here, or this error message does not make sense, or if this component is blowing up, I need something to trace it back. That is generally our involvement.

We do spec review. So many of our engineers have been on betas before, they are hooked in with the product group contacts. As the next version of the product is being planned, our engineers get involved to review specs, and just generally throughout the beta process from the beginning of our involvement through RTM, we are providing that ongoing supportability feedback.

So not only are we always supporting our customers, but we also look to help the product group on the supportability side. It is like you said, I mean, they are focused on here is a new component, here is a new thing that customers can use, maybe not always focused on how would I support that if it breaks?† Letís be honest Ö if we donít provide our feedback to the product group on what we need, it just causes more pain for us once the product is released.

Scott:  One other one question is, so a new product is coming out and how do you get support folks up to speed on it? In the OpenSource model the development team in some cases is the support team and there is a tight integration between that team and the open source community so there isn’t really a need to educate the "support team" in this case. So how do the support engineers come up to speed on a given product?

Britten:  This is another benefit of our involvement on the beta side.† †We get access to the source code. We get engineers playing with the product. We get that constant interaction with new builds of the product as it comes out. As a result, months, in some cases years, before the product actually ships, we have experts on this product.

Now, our experts certainly might be great technical writers, but we have a group called Global Technical Readiness. These folks are really program project managers who will say, "All right, we need Vista training for our support engineers." Therefore, they get their list of top issues. They get a list of new components, features, all of that. In addition, they start mining for data using these beta resources to come up with, "Here’s what engineers need to know. Here are the troubleshooting pieces; here are the top issues we have seen with our beta customers. Make sure engineers know this and that."

Therefore, over the course of this beta cycle, we are also in the background writing our own training. So these engineers write the training and as RTM approaches, depending on the training strategy, they might train all their engineers or a subset of the engineers. However, the knowledge is based off spec; it is based off beta engineer experiences, and beta engineer customer interaction.

In addition, the product group, in recent years, has really stepped up to help with training. They will review our training. They will provide insights. They will attend beta deliveries of the training and help the trainer get comfortable with the material and the product.

In some cases, the product group will also do a session of live meetings on, "Hey, here’s how we intended you to support the product. Here are the supportability features that we thought would be helpful." Therefore, you get a lot of interaction with the product group as well. From the beginning of the product to the end, not only are we helping customers on the beta side, but also our selfish win is to get the training that we need.

Scott:  So, let me ask the flip side of that, right? So, you mentioned Configuration Manager. You’ve got System Center Configuration Manager 2007 coming out. Everybody wants to learn it…

Britten:  Sure.

Scott:  …because everybody is motivated. Have do you rather incent people to keep wanting to support the previous version when there is something new, shiny, and interesting.

Britten:  At least on the Commercial stage, I do not know that I have ever really run into that problem.

Scott:  OK.

Britten:  Just simply because, in the enterprise customer segment, what you generally see is not the consumer model where a product releases, and everybody goes and gets the new version. However, it is a gradual uptake, usually. So, say Configuration Manager. Right now, and I can give you the specifics on this one, it is 10% of our total SMS volume. Therefore, engineers get a couple of SCCM cases, but they get the majority of SMS 2003.

In our minds, that is beautiful. Because what happens is, slowly over time they can build their SCCM knowledge while still providing that long held knowledge on SMS. They can help those customers resolve their issues. Therefore, it is a slow mix that generally happens. I do not know, it is ‑‑ they are tech junkies. They love the new stuff, but I guess maybe we all know that we have to pay our dues with the old stuff as well. [laughing] I do not know.

Scott:  OK. That makes sense.

Britten:  That’s a good question but it never really happens quite that way on the Enterprise side of support.

Sean:  What it sounds like for you folks, is that from an enterprise standpoint, you can point to pretty clear statistics on customers who are significant accounts. Alternatively, you could point to customers who, for lack of a better phrase, probably spend a fair amount of revenue, right. In addition, it is probably easy to also look at the segment of the market and say, "We’re not going to give up 15% of SMS just because SMS, the latest version is out" ‑‑or in this case, obviously the rebranded version.

Britten:  Right.

Sean:  Because you folks can point to real metrics in terms of your percentage of the number of people who are putting in support requests and things like that.

Britten:  Right.

Sean: Thanks for taking the time to talk with us Britten.

Britten: No problem.

Bookmark this:

Interviewers: Scott Swigart

Interviewee: Rod Johnson

In this interview with Rod, CEO of Interface 21 and founder of the Spring framework, we asked him about:

• Development methodology of Spring framework
• Contributors to Spring
• Selection of code and features for the Spring framework
• Ways in which developers support open source
• Open source development cycle
• Importance of technical support
• Code aggregation in open source development
• Choosing open or closed source within small businesses

Scott Swigart: Rod, thanks for taking the time to chat. If you don’t mind, please take a moment to introduce yourself.

Rod Johnson: I am the founder of the Spring Framework, which has become the effective standard for enterprise Java development. We also have a spring.net edition. I am CEO of Interface 21, which is the company behind Spring. I’m also the author of several books on J2EE development.

Scott: Great! Talk to me a little bit about the development methodology of the Spring framework. How does that work? Every open-source project seems to have its flavor. Tell me how Spring is built. How do ideas go from conception to features to in production or in development to release?

Rod: OK. I think when people try to understand what open source is, they instantly think about code and that the software is created by a disparate, far-flung group of people who work in their spare time. In fact, to understand the open-source model you need to step back from code and look at the totality of things, which includes documentation, includes support for users, and includes the discussion of ideas.

What we see in the core of our software is that the vast majority of the code is written by a small group of exceptionally good people. I don’t believe there is any other way to produce exceptionally good software. We have known for many years about what Fred Brooks calls the The Mythical Man-Month.

Scott: Right.

Rod: Open source doesn’t change that. We know that if you throw more people at a problem you don’t get a better solution. And it doesn’t matter what approach to distributing the source of the software you use. You’re not going to change that fundamental fact. If you want exceptionally good software, you want a cohesive team of exceptionally good people and you really want that team to be as small as possible.

Scott: How is that team motivated to work on the project?

Rod: The core people who connect to Spring daily are employees. They are being paid to do this. That model works very, very well if you look at open source in terms of the enterprise Java space and also obviously of the Linux space. You see that the model is evolving towards paying the contributors who write the code. For example, I think 21 out of the top 25 committers on Linux are employees of large corporations. The majority of the committers on Spring are Interface 21 employees. Another good example of this is JBoss, the Java application server. The majority of the code there comes from employees of JBoss, who are now employees of Red Hat.

Scott: I want to get your insight on the Linux kernel. My understanding of the Linux kernel is that unless Linus likes a certain patch, it’s not going to make it into the kernel. And yet, the bulk of the development is done by people who work for IBM, people who work for Red Hat, Intel, AMD, whoever. Right? Corporations are doing the bulk of the work. And yet what makes it in or out of the kernel isn’t really under the control of a company. Do you have any insight into how that works?

Rod: I am not an expert on Linux kernel governance. From what I see, one thing that is very, very important to developing high quality software and maintaining that software in high quality for the long-term is consequences: consequences of success, consequences of screwing up.

In the case of Linux, what we have is not a single entity, but we have a number of companies (and in Linus’ case partly also an individual, obviously) and a number of parties that are really baked into the system because there are large consequences if they screw up.

I don’t think one entity needs to be responsible for a piece of software. I think you can end up with a kind of partnership, but there have to be consequences if something is screwed up. That is one thing that scares me about the rather naive volunteerism view of open source because what are the consequences to a volunteer? Sure, in many cases they may be professionals. They may feel as a sense of personal ownership that they cannot be satisfied unless something is as good as it possibly can be. But there ultimately are problems because they don’t necessarily suffer the same kind of consequences if something isn’t taken forward.

Scott: What people have told me is that there is sort of a developer Darwinism that happens. In other words, to get your code in you have to submit it to a mailing list. You have to submit it to what one person described as a “full contact code review” and in some mailing lists you’ll just simply flat out get ridiculed and won’t be taken seriously if your code isn’t to a certain level. So, it seems reputation based, I guess. You can’t really get code in until it’s passed code review and up the chain of maintainers. So, on one hand there isn’t a lot of consequence to writing bad code, except maybe it was a complete waste of time. Your inability to do it well was pointed out for the whole world to see. So, it seems there is a certain amount of pressure to sit up straight and code well, especially if you’re going to try to get stuff into very high profile, very core open-source products like the Apache, or Linux, or some things like that.

Rod: Yeah.

Scott: One of the things that you pointed out is that people who are going to use open source shouldn’t consider it a free ride. It isn’t just something that you should just leech off if you want the stuff that you’re dependent on to continue to thrive, grow, and be there for you, and continue to be enhanced down the road.

But, at the same time, most of the core contributors to open-source products are highly skilled people who are employed by a company, and this is what they do all day long. So, if I’m going to be a user of open source, how could I best contribute?

Rod: The first thing is that I would hate to give the impression that openness means nothing. There is one interesting characteristic, for example, about a piece of software like Spring, which is overwhelmingly written by one company, compared to closed source.

You have the ability to work up to being a committer of the software. Not many people will take that up. And we may choose to approach you to see if we can hire you if you do it. And that is a way of expanding and renewing the developer base.

So, it is open in the sense that there is opportunity for people to come to be committers on the software through the community based on merit. That is interesting because if I want to be a contributor on the Java-based IBM’s WebSphere, I’m sure I could add some value based on my background to WebSphere, but I can’t put my code in there. There is no way I can do it without becoming an IBM employee.

Scott: Right.

Rod: So that is a genuine difference. There is a true meritocracy. So, coming back to your question, which was “How do you pay? How do you contribute back?”

Scott: Yeah, how do you contribute?

Rod: Well, I think the first thing is to remember that although it’s very important that there is that meritocracy based on code there are several reasons that the majority of users are never going to contribute to that code. The majority of users are not going to modify and compile it. The majority of users are going to use it essentially as if it’s a shrink-wrap product.

I make all of our consultants just use the binaries. If they’re working with our product with customers, we don’t like them to be working with the source code and making changes to the source code of our product.

The interesting thing is that it simulates more accurately how end users use it. So, often we’ll go into customers where we’ll see that they just realized that you can mount the source code and if you want to step through the same work source code…

Scott: Right.

Rod: …so, it just doesn’t make sense for most end users to even consider modifying the software. Consider that Spring is 1.2 million lines of code. Some of that code does things like transaction management, the fundamental configuration of your application. Some of it executes in virtually every runtime call set. It doesn’t matter whether its open source or closed source, you really don’t mess with code that does that kind of thing unless you know what you’re doing.

Scott: Right.

Rod: The process of knowing what you’re doing…For example, in our transaction infrastructure I, myself, as the original co-designer, made a change about a year ago. I spent almost a day conceptualizing the current state of the code (which one of our other guys had modified) and figuring out exactly how I could do these [changes] in a safe way.

I am a strong enterprise Java developer with 10 years experience. I originally co-designed this module so I know how it is meant to work, yet it still took me almost a day to feel that I can safely make a change without risking breakage of a core part of the software.

Imagine if I were someone coming from outside who didn’t have 10 years of experience, hadn’t co-designed the module. What is the probability that I’m going to make that change in a robust way unless I spend maybe five days fully getting inside it?

Now, if I am an end user and I am writing software, for example for a bank, like many of our users, they don’t want to pay me for five days to figure out how the Spring framework works internally. That is irrational. There is little economic benefit for end users. There is a high probability of being economically irrational in modifying the open source software.

Another good example is Linux. I did a lot of C-programming early in my career. I can code in C. Am I ever going to look inside the kernel and change any of the kernel? This makes no sense, whatsoever. So, code contributions and changes are not likely. However, there are many other things I can do if I wish to give something back.

One thing I can do is to help other users of the software. Most open source projects have forums and mailing lists. Every time a developer answers a question for a user that developer is distracted and can’t write code to improve the product. Both things are important but no one can do both simultaneously.

One thing that has worked brilliantly with Spring is, especially early on, we made a big effort in the core development team to answer users’ questions and be really, really helpful. That set up a virtual circle, where users saw the Spring development team was really helpful when they were first starting up. Gradually they used the software more and became experienced. Then, they see other people asking questions and they think, “Hey I know the answer to that question. Maybe Ron doesn’t have to spend time answering that question. I’ll get in there and I’ll help the other user.” That’s a valuable way of giving back.

Another thing is promoting the software because open source software, like any other form of software, can only be successful if it is popular. For example, you like a particular piece of software. You wonder whether there is a user group for that software in your city. Maybe you have time to organize the user group. Maybe you can persuade your manager to provide coffee and a room if someone else is willing to organize it.

Another way that you can contribute is through issue tracking. If you see something that you think can be better, go and report it. An issue tracker is the enhancement request. That works very, very well with open source because that is one area where access to the code really matters. We see people who make enhancement suggestions with pretty deep knowledge of the code. We might get an enhancement suggestion saying, “By the way, if this class extended some other class it would allow the possibility to do X, Y and Z. Have you considered that?”

What happens is that one of our developers looks at it and says, “Oh God, I would never extend that class. Oh, but I see what he’s getting at. Oh, yeah! He’s right. It would allow X Y and Z. So, I wouldn’t do it that way but I can see that there is a lot of value in what he suggested.” The ability to look at the software is quite interesting from that point of view. It’s not necessarily so interesting in terms of modifying the software but it’s quite interesting in terms of the ability to provide suggestions with as little or as much context as you want.

Finally, and this is the way in which I think most financial organizations will end up giving back, go find someone who is contributing to or sustaining that software and buy a support contract. Not only is that going to sustain the software very effectively but also open source software needs the ability to have maintenance, especially as open source moves into the enterprise. What happens if a system that does trading goes down at 2 AM?

There’s not much point if one of your guys gets paged. He didn’t ultimately write the software that is involved in it and the fact that it is open source doesn’t really make any difference. The need for a similar level of SLA and mission-critical support is exactly the same.

Scott: So, that’s one of the things that I want to drill into a little—two things actually. One is support and the other is the software development lifecycle.

It looks like if you take a look at something like Spring, there is nothing magic about the way it is built perhaps. You probably have internal meetings. You come up with plans. You come up with specs. Engineers are assigned. Software is written. It goes to quality assurance.

Open source gives you much tighter coupling to the user base and the community at large. They provide better issues because they look at the source code and even recommend a certain implementation and talk about the merits of a particular implementation. They file bugs directly into the same bug database that your internal people use. So, open source gives you that tighter coupling with your users. But—and I don’t think this is unique to Spring—I am finding this a lot; it is still built the way software has always been built.

Good people. Good process. Good QA. You have people assigned to do documentation. You have people assigned to do QA. Even though those are not the things that people would volunteer for, you’re not dealing with a bunch of volunteers. You’re dealing with paid employees.

Am I accurately summing it up?

Rod: I think you’ve got it. You raised one really interesting question there. You said that the community uses the same issue-tracker as we use. That is obviously true. We don’t have anything around the software in terms of management of bugs that isn’t publicly visible.

A big part of open source, to me, is free use of information. I know who wrote a piece of code, I know when it was last changed; I know what issue that person was trying to fix when they wrote that code. I can see all that information.
It also means that there is a great deal of security in adopting software that was produced by small companies. For example, at Interface 21, there are slightly under 50 people. We’re growing pretty rapidly, but that’s still a pretty small software company.

So, let’s suppose that our product was closed sourced. We would, in each deal, be dealing with escrow agreements, that kind of thing. Frankly, other people would not think of using our software. They wouldn’t necessarily know whether the internals of our software were any good. They could see it from the outside, but they wouldn’t know, for example, what’s our philosophy of bug fixes, what is the quality of the documentation internal to the code, or what is the quality of design.

I do believe that there is a very strong benefit in open source. I believe that all that information is freely available. You could look at the software, and I would strongly encourage [that for] customers making significant adoption decisions.
They’re probably never going to modify the code, but they should send some of their best guys to spend a day looking at the code. See what they think of it. See whether or not you think this code is maintainable, whether or not the language idioms reflect best practice. That’s a very interesting benefit.

Scott: One of the other things you talked about was support. That comes up a lot with open source, because the way to get support isn’t always as obvious as with closed source. If you buy something from Adobe or you buy from Microsoft or you buy from Symantec, it’s obvious where you go to get support.

In the open source world, it seems there are two models. One is, “OK, you pay MySQL, you buy their enterprise software and you get support.” It’s the same thing with RedHat and the same thing with Spring, I’m guessing – can you purchase a support agreement from Interface21?

The other option seems to be that you go to someone like Open Logic, who supports everything, and you just pay them for support across a lot of products. Now, you’re probably a little biased in your answer, but what do you think about the two different ways of getting support on a product?

Rod: There is nothing wrong with an aggregation model if you’re looking at, effectively, as a broker who is providing the ability to reach the bloke who wrote the software.

Where the aggregation model, I believe, is flawed, is in cases where essentially it’s not providing the ability to reach the people who wrote the software, and also where it is not providing the ability to pay for improvements to the software. So, what am I buying? So, let’s suppose we’ve agreed that we need to give something back. If we’re adopting free software for potentially the next 10 or 15 years, we need to give something back.

We care not only about how we get bugs fixed or how we get our production systems working, we also care that this software is still going to be mainstream in five or ten years or at least hasance of being mainstream because if it isn’t, we’re saddled with an expensive legacy. We also care, clearly, about the software evolving to accommodate new features that are coming forward in the industry over time. So that means: First, we need the ability to make critical fixes if necessary, to ensure that there’s no risk that our production systems are affected.

Second, we need to be concerned about where that investment is going to come from in terms of sustaining the product and moving it forward. So, we’re not just paying for an insurance policy today, we’re paying for the insurance company to be there tomorrow, and for the premium we’re paying to the insurance company tomorrow to be meaningful.

This is where some of the aggregation models, particularly what Open Logic calls it’s “Expert Community,” totally fall down. Because with the Open Logic model, not a cent of what you’re paying is going into writing software. You are not paying a cent towards writing software. That can never work in the long term. What you’re doing is paying, presumably- I haven’t looked at their price point, but I’m assuming their price point must be pretty low – a low price point for something that is not comparable with true enterprise support.

There are two problems with the model. One problem with the model is that it does not do anything for the health of the goose that lays the golden eggs. It says, “Well, I’m not going to feed the goose. I don’t care about feeding the goose; I’m just going to scrape up the golden eggs.” Obviously, the goose isn’t going to be there tomorrow, if that becomes the dominant approach.

The second problem is you just cannot get good quality support out there. Let’s suppose that I’ve got a piece of software that may have a lethal, critical problem in production. Now even though I think, for example, that the Spring Framework is a fantastic piece of software, the best pieces of software in the world have bugs in them. It’s impossible to write perfect software. It doesn’t matter whether it’s open source or closed source.

So, you’ve got a critical problem in production, and it’s really hard to resolve. You will see this happening with both commercial companies and closed source companies and open source companies. They all need to assign resources to work on a resolution of that problem until it’s fixed. When the developer in San Francisco clocks off work, he hands it over to the guy in Australia, who continues working on it until, probably, the guy in Romania starts work at the end of his day. And you need to be able to pay for that, and you cannot support that quality of support with essentially paying pocket money to volunteers.

Obviously, I have a vested interest. My company represents a different model. I hate it when people sell open source short. I hate the view that, “I only use this open source thing because it is going to be cheaper”. I feel so strongly about that because I used to be on the other side. I used to be the person making purchasing recommendations. In the financial industry companies where I worked, I would look at TCO. It made no sense to me to say, “Hey, I can save $100,000 here on a support contract.”

What I needed to look at was where, for example, one of the systems that I was involved in designing handles $5 trillion annually. If anything goes wrong with that system and we can’t fix it for a day, the cost to that company could be in the tens of millions of dollars. In fact, if it was down for two days in that example, the consequences to the UK economy could be quite catastrophic. So, I’m not going to be making the decision on, “Well, you know we can choose open source here and we can save $100,000 on a support contract.” I am more likely to be thinking, “OK, well if open source software seems to fit our requirements really, really well, what do I do if I have a mission critical outage? Who do I pick up the phone to call? Who can organize a team to work on this until it’s fixed?”

Scott: One follow-up: It seems that we are talking about the extremely large enterprise accounts that are handling enormous amounts of financial transactions or things like that. They have enormous vested interest in the quality and the future of the software that they are running. It seems like smaller organizations, whether they are choosing closed source or open source, kind of get to draft off of that.

Talk about the decision for the smaller organizations. For a 50-person company, how is the decision different than it is for an enormous financial exchange?

Rod: That’s an interesting point. For the smaller companies it depends on what you’re doing. For smaller companies who are not doing something that is as mission critical, it may be that the software effectively is free to them, and they don’t have the same requirements around mission-critical support.

Yet frankly, from my point of view as a chief executive of an open source company, if people who don’t need that kind of mission critical support don’t buy it because, for example, their company simply can’t afford it, good luck to them. I would love to see the 50-person company grow into a 500-person company with a website that has a huge amount of traffic for which they need to buy a support contract.

So, I think that open source expands the market and it brings the software to a new audience of people who don’t necessarily have the need or capacity to pay. Essentially, good luck to them! No one should give open source money out of the goodness of their hearts. They need to look at what they need as an organization and who can satisfy that.
So, for example, for every job where I have personally been involved, I would have been getting support contracts for the open source I was using because of the nature of those businesses.

However, let’s suppose I was working for a start-up, a dotcom that is trying to build up some kind of community site and is trying to get traffic. It doesn’t really matter if it goes down some of the time. The software that my company is developing is rapidly evolving. At that point I wouldn’t have a support contract. Let’s think about Twitter. Twitter is a great example here. Twitter clearly went through that. Then it became successful. Now they are encountering technology limitations, which are causing frustrating outage. Now you are getting into a different set of business needs. Now they are reaching the point where it is going to harm the growth of their business unless they improve the uptime of their infrastructure.

Scott: So, in other words, your goal as a start-up is to grow to the point where you need a support contract.

Rod: I have never put it in those words before but I think it’s an interesting way of thinking about it. In a way, you have a greater ability to pay the price that is appropriate to your business needs.

Scott: Right. At some point if you are a start-up and you’re not successful, then the open source that you are using is probably not as valuable to your business as if you are a start-up and you become very successful and you grow. You become an ongoing concern. Well, now your infrastructure has value because your business has value and you need to treat it as such.

Rod: Yeah. I mean what I would suggest is that in the early days people should look at purchasing things like consulting and training around open source.

Scott: Right.

Rod: I think that makes a whole lot of sense because it is simply going to reduce their total costs by making their developers more efficient.

Scott: Right.

Rod: So, the needs profile of the end-user is significantly different.

Scott: Well, I really appreciate you taking the time to chat.

Rod: No, I think that was an interesting discussion. Thank you for following up.

Scott: Thanks very much.

Rod: Thank you, Scott.

Bookmark this: