Interviewees:Shawn Burke.

In this interview with Shawn Burke of Microsoft we asked him about:

• Shawn’s motivation for helping to make the framework source code available.
• Whether there are any limits on who gets access to the source code.
• Whether there was any internal resistance initially to providing the framework source.
• Why Microsoft doesn’t just provide a zip for the source.
• Whether Microsoft’s individual developers write code differently knowing that their actual source might be available for the world to see someday.
• Whether or not Microsoft is planning on making available other development artifacts such as test scripts, threat models, etc.
• Whether a closed source company or an open source company is better positioned to support external developers who need access to the source for a given project.

Scott: All right. So if you take a second and just briefly introduce yourself, that is probably a good place to start. Then we can drill into questions specifically around the source for the.NET framework 3.5.

Shawn: OK. I am Shawn Burke; I am a director here at the.NET development platform at Microsoft. I run a group that we internally call the agility team. We run small high value projects to try to help deliver value to customers in agile ways. Not necessarily using agile technology, but just being agile about how we deliver.

Scott: :So in other words, it sounds like you are looking for sort of low hanging fruit where you can do things that are a little bit outside of the norm but would not necessarily be an enormous effort, but would deliver a lot of value to the customers quickly. Am I characterizing it correctly?

Shawn: That is right. Our team works a little bit outside the bounds of the typical projects and we are not constrained by the same rules they are. We are able to do things that are more customer focused and a little less process focused way.

Scott: Just as a point of clarification, people have said Microsoft is “open sourcing the.NET framework.”That is not what your Microsofts intention is and not really what you are doing. You are making your source code available for reference. You are not really open sourcing it the way that, you know…

Shawn: Nope, some people in the community used characterization, that is definitely not what were doing.Were making the source available as a tool that helps people be more successful with the platform in its current state.

With this effort, my goal has been from day one, a simple scenario, which is you are debugging against an API, you are getting an exception from the API, and you cannot figure out why you are getting it. It is often very difficult to figure that out just from documentation. However, your ability to step into the API often makes it really clear what you are doing wrong.

Scott: Right. In the example that Scott Guthrie put up on his site, this is something I’ve run into,? At the point where you DataBind something, all kinds of stuff happens like down in the plumbing.

Shawn: Yes.

Scott: And the exception that you get, you know, you do not really have a lot of visibility and sort of what you did wrong in your code that caused the DataBinding exception. And so that’s like…

Shawn: Yes.

Scott: That’s like the key scenario that you went after.

Shawn: Yes. That is like the thing that you are unableto do currently you know, because I work here I have the benefit of being able to do that a lot. Quite frequently, the amount of time it saved me , “Oh, Ill just grab the symbols” and you do and start to debug with them and immediately you’re like “Oh! Because I forgot to do this or that.” It becomes so clear that I really wanted to enable that sort of service for customers because its such a powerful thing

Scott: Yes. And just to back up another thing that you said, too. I never read it. I was not indicating that Microsoft was somehow trying to create the impression that they open sourcing this. I know when I looked at it, it was clear to me that Microsoft was releasing the source for reference purposes, you were not releasing it so that people could…

Shawn: Yes. I get it. I did not mean to sound defensive there!

Scott: [laughs]
I’m sure it comes up a lot.

Shawn: Yes. That was the first thing we saw on a large website that is often not terribly pro Microsoft

Scott: [laughs]

Shawn: Fortunately many people came on there and said, “No, no, no.” They never said that, that is not what they are doing. I think most people get that now.

Scott: So what you are enabling is a sort of step into debugging experience where normally, as soon as you call into the API, you could not continue to drill down and step in. Now you can.

One of the things I was not clear on is who gets this? Is this available to all versions of Visual Studio and all customers or is there a sort a subset? Who is this available to I guess?

Shawn: If you are running on Windows, it is available to anyone who can use it or to look at it.

As far as the Visual Studio integration, you need to have VS Standard or above. You cannot do it with Express versions. You know the express versions of Visual Studio?

Scott: Sure.

Shawn: Express does not get it for some kind of fundamental reasons.The main reason there is it just simply because Express does not expose the UI that you need to set this up or have some of the other facilities internally to take advantage of it

Scott: Sure. Sure.

Scott: Yes. The only reason I ask is that some of the people in the Microsoft ecosystem have made their source available kind of under premium plans with their customers. However, that is not the approach that Microsoft is taking with this. This is not sort of a premium thing, this is just generally available.

Shawn: Yes, that is Code Center Premium (CCP), this works similar to that CCP , but there you need to have an account and a Smart Card actually. And you have to go through much more security than in this system. This is a public reference release of the code, so there is really no credentials required. There is none of that.

Scott: So when you sort of floated this idea internally, it sounds like it was generally well received. People kind of jumped on board and wanted to help make it happen. Just behind the scenes, what were some of the things that it took to enable this? It is more than just dropping a zip of the source out there to make this happen.

Shawn: Yeah. Well, yeah. It is good that you brought that up. This whole thing started quite a few years ago. When I was on the Windows Forms team, which I was for seven years, I always felt like we should be releasing the source for many reasons. Others had done it. I felt like it was good for customers. When I was the Development Manager on Windows Forms in early 2005, I did a blog post about this, saying I wanted to make this happen and talking about some of the challenges to doing so. It was kind of justI was just kind of brainstorming aloud about different ways and what some of the challenges were. The response was enormous.Hundreds of comments.

A funny story, I did that blog post, and then I went down to Cabo San Lucas with my girlfriend a couple of days later. When I turned my phone on halfway through the trip, just to see if it would work in Cabo, I had a flood of text messages that said, “Please call work.”

Sean Campbell: [laughter] All right. So you learned a lesson of do not blog anything but “I’m going to Cabo” on the blog before you go to Cabo, basically.

Shawn: Yeah. I think if you would really pull it down to basics, that is it. Well, actually, I had always done tons of blog posts that no one’s…

Scott: Meanwhile, you created this firestorm of all these people running around going, “What have you done?”

Shawn: Yeah, exactly. I like to joke that it created, basically, an international incident because it got picked up by eWEEK, Mary Jo Foley, CNET, and all the rest. Quite a few people picked it up. It generated a lot of interest. That was a good impetus for me that there was a lot of excitement in the community about this.

If you fast forward a couple of years, I did some more ground work on this thing in 2005 and 2006, got busy with new role I’m in now, and dropped it for six or eight months, and then picked it back upI think it was late last year, late 2006and started looking at how we would get this done.And Ive been shepherding it along since then.Getting this out there really is the realization of a dream that I have been chipping away at for quite some time.I really hope Developers find it as useful as I think they will!

You guys mentioned pushing it out as a zip. One of the real problems was that I did not want to have a situation where every team that did this put a zip file or MSI out every four months, so the customer had to go download, install, and set up his little studio to use. What if you have a hotfix or different version? You know what I mean. I did not want to have a situation that was going to be cumbersome for us to ship and for customers to consume.That would kind of defeat the purpose.

What really got me rolling on this was I started talking with the guy who developed this technology we use internally. Internally, you can get the source and demos for almost every product we build here at Microsoft off this thing called “Source Server.” Source Server has versioning built in, making sure things do not collide so you can have side by side versioning, and it all works very smoothly. The server side piece for source server is just a web server. There is no magic there so its very simply to deploy. There really was not any barrier to making to using that technology externally.

So what we able to develop was a system where teams could publish source very, very efficiently. After a first source publish, they can continue to do source publishing at near zero cost to them, which is great. On the customer side, you have a scenario where a customer has a few simple set up steps. Then they kind of auto magically get source and symbols for what we have chosen to make available at that time for free. No searching around on MSDN or having to deal with where to put the stuff on the machine.

Scott: It sounds like, too, one of things that you are saying this accomplishes is it sort of knows exactly what the client is running and building against. If they have installed the service pack or have not installed the service pack, they get the right source. They get source that matches up with their environment.

Shawn: Yes, if it is on the servers they can get it. One of the decisions I made in getting this done was that, the cheater thing I did was I said, “You know what, I ‘m not going to define policy on this. I’m going to get a system developed where individual product groups and teams can make decisions about how often they push source.” Meaning, does it make sense to push source for every new release we do? Maybe, I dont know. Does it make sense to push it for every service pack? Maybe, I do not know. I think those might be different for different teams. I think we are going to do learning in working with the community about what really makes sense there. However, the system is set up to handle it, in any case.

Scott: Like you said, generally people were receptive to this. Were there internal objections that had to be overcome? Where there people who had concerns that, “Hey, this is crazy. Why would we do this?”

Shawn: Well, if you go back into 2005, I think I had many senior people that I knew, vice president level people that were generally supportive of this, but very cautiously supportive. For me a little bit, one of the reasons it’s been a multi year effort is I had to push carefully and let some perspective change as the industry has changed and the environment changed over the last couple of years.

It has gotten significantly easier. Something happened, I think, in the greater consciousness of the industry over the last 12 months or so. A lot ofI do not want to say a lotresistance is too strong of a word. A lot of the real cautiousness and kind of by-default-no talking-into-yes stuff dissipated and switched to actually a lot of enthusiasm. We need to be careful about protecting the IP that we spent billions of dollars developing. However, I think people really see the value of this now. It has been great for us and for our customers for a subset of products. Now I have seen teams are signing up to push their code out really quite enthusiastically.

Scott: With the.NET framework, it is in a little bit of a unique position because, on one hand, Microsoft has invested a lot in it, and generated a lot of IP. However, on the other hand, it was never really a secret what was in the framework anyways. You could use reflector and other tools to disassemble what was in the framework and sort of figure out how it was doing what it was doing. However, it did not enable this “step into it while you are debugging” experience. It did not make it super easy to learn from the framework. This might have been a little unique in that the IP was not all that protected to begin with.

Shawn: Well, yeah. Yeah, it is funny. The case that I built for this, with managed code in particular, definitely had that aspect to it, that our level of secrecy around how this stuff works, as well. One of the problems with reflector, using reflector, the EULA does not really allow for it.

Scott: Ah, interesting.

Shawn: The EULA does say that you are not supposed to reverse engineer. That said, I felt like it was a good idea to give customers a way to do this within the license, which this is.

Scott: Hah! Yes, I guess I never really realized that that might be in the EULA. I think, especially among the more advanced developers and in the blogs, and in all that kind of stuff, it is such a common practice.

So, now that you have done this, you have taken this first step and you’re going to make source code available for certain subsystems of the framework, and that’s on a feature by feature basis, right?

So, Windows Forms will decide when they push their source code in, ASP.NET will decide when they push theirs in, am I understanding that right?

Shawn: Well, we have pretty much what you would consider, if you go back to.NET 2.0 and think about what made up the Framework at that point. We’ve got pretty much all of the big pieces there. We’ve got BCL, Windows Forms, ASP.NET, Data and XML.

So those guys are already taken care of. Therefore, we’re going out with essentially what is the original.NET framework. The other stuff will fall in. It is really a matter of me getting with a given team and through the process. The code preparation process is actually fairly labor intensive. It takes time to schedule that and manage it.

I had teams that were able to do that. A lot of teams have been obviously really busy getting VS 2008 and.NET framework 3.5 finished up. We will continue to push out more code as time goes on. We wanted to get the big ones first.

We also managed to get WPF in there as well, so its not just the older stuff.Lots more is in the pipe to come online as time goes on.

The choice was either we wait a couple of months to do this, and make everybody wait, or we get the big ones out there fast and then we push in the other stuff as we have time, and as it comes online.

Scott: So one of the things that is kind of interesting is, you’ve got these individual developers in Microsoft and they’re writing code. They have worked in Microsoft for a long time and they have a feeling that, you know, what happens in Microsoft stays in Microsoft, I guess, for lack of a better term.

The whole world is not going to see the line of code that they are typing right now. Now, with the source being released, how do you think that is affecting, if at all, how individual developers are doing their job? How product teams are looking at how they build features going forward, knowing that the source for it is going to be out there.

Shawn: Even product teams that are not the framework team, they are thinking, that’s great, they did that over there, so does this mean that someday my source is actually going to be out there? Yes. Yes. They will be thinking that. And they should be thinking that.

In fact, one of my to do list items is to do some broader communications across our Division about that exact point Software developers are pretty much the same around the world. Software developers, in general, always try to do the right thing. They are also always, in general, under a certain amount of time pressure. Anything that is going to encourage them to take a little extra time to do something right, in the kind of capital ‘R’ right, way, is a good thing.

Whether that’s testing, whether that’s static analysis like FxCop, or whether that’s security reviews or code reviews, or whether that’s the code is going to be visible to their peers or the public, I think those are all good incentives for software developers to think extra hard about how they do things.

So, you know, honestly, we went through the code, we’ve done these code reviews, and these code scrubs and we found a lot less stuff than we thought we would.Actually, we found almost nothing of any real concern, which was nice.

Sean: Are there other things that Microsoft is thinking about exposing around the development process?

There are elements like threat modeling, specs, test scripts, all kinds of things that Microsoft might be thinking about exposing. Are there any thoughts or plans along those ways about how far this maybe goes, eventually?

Shawn: That is a really good question. No is the answer. [laughter]

Sean: You’ve accomplished enough by ruining one Cabo vacation. Right? You can put that post up later, you know, on your next vacation.

Shawn: I think that stuff…that is a good point. If you start doing, a curve on the cost for us to do that versus what percentage of customers that level of access is going to be valuable for. For the threat models, and even for the specs, I think that is a harder case to make. I think those things are also harder to vet, or release, than the source is. They are also less useful to a broad set of customers.

Sean: Well, one of the things that crossed my mind was test scripts, and test routines that Microsoft creates because that would be helpful to others just from an educational standpoint. That does not really expose Microsoft to too much more overhead. I am not trying to pitch the idea here, But go ahead, Scott, what were you going to ask?

Scott: No, same thing. Just, Microsoft has given an inch so watch us try to take a mile.

Shawn: Oh, yeah, we know it. Absolutely.

Scott: A lot of times that’s what people want. They say, OK, there is this API. I want an example for how would use it. I would go look in the help. Well, that example is not what I really want. Microsoft I would assume writes really comprehensive test suites.

Shawn: I do not know if the test code is what you want, either. We call it auto PMEs. The PME stands for Property Method and Event. It is super granular. A lot of that code does not relate to any real scenario. It is just wide unit testing code. There are other types of tests as well. A lot of the stuff is that. The challenge of the testing stuff.

I actually think that the testing stuff is a good point, and an interesting thing and I have been thinking about, not really in terms of us releasing test stuff for previous products. I will get into one in a second. The question is going forward, is there a way for customers to leverage the kind of stuff we do in tests more broadly?

However, the reason, going backward, most of our test stuff is written using internal test harnesses and internal tools. There is this huge slippery slope there. You need this and you need that, and it would be really a lot of work.

Scott: Right, right, so they would have this test but they would not be able to hit F5 on it and actually watch it run anyways.

Shawn: Yeah, and the tests probably call into a bunch of other libraries that are internal that we would not ship externally anyway. The tests themselves might not make any sense. Most of the tests are stored in this big system called Mad Dog, which is kind of a giant database of test cases and test code, and I do not even know how you go about extracting code from that thing.

[laughter]

Sean: So Shawn, what do you think aboutthis is just something I was noodling on, trying to think of the right way to box it in given the time in terms of a questionso, Microsoft’s a closed source company, right. By definition if not necessarily for the sake of our interview. What do you think about closed source companies perhaps at times being better able to support the exposure of the source than maybe a company or a project that simply says ‘Well, just go check it out from subversion, from xyz open source project?’

Shawn: Right.

Sean: Because I look at the Visual Studio integration, and I look at robably how this will eventually weave into PSS, interacting with customers, and saying well, here’s an odd dynamic: is a closed source company perhaps better able to actually make the exposure of the source code, when they do it, a more positive experience that actually lets people leverage it more easily? I mean, absent the ability to fork, and some of the more foundational principles of Open Source, how an outside developer would use it day to day development and how a closed source company in this case Microsoft could support that.

Shawn: That is a great question. I think that the answer isone of the things that I’ve been trying to do, and that we’ve been trying to do, is look at the changes in the industry that open source has driven and really think hard about what the real value adds are there. Is the value add really that you can recompile your kernel, or is the value add that you can debug it and understand why it’s doing something?

And there are obviously cases where the answer is ‘both’ or ‘one or the other’. However, for I think the vast majority of folks, us being able to deliver the source in a way that’s easy for folks to consume and step through as a way to make them more successful on our platform is a really big win. I think that it fits well into our overall strategy of kind of keeping developers integrated in our overall processes, and it fits well into this another milestone of transparency.

We have kind of been trying to figure out ways to open up the envelope of transparency with Developer Division particularly and I think this fits into that portfolio really wellas well as the portfolio that the Shared Source Initiative folks are building. You have CodePlex on one end which is full open source, then you have kind of our proprietary source system on the other end, and there’s a nice continuum in between. I think that we do bring some assets to bear which are difficult for companies that are purely open source or service model driven.

Sean: OK, cool. Well, that is all I’ve got, and also to be sensitive to timeScott, do you have a closing question or two?

Scott: I think that’s a good note to leave it on. The thing that we always end with is: are there things that we didn’t cover that you think are really important for people to know? So, we’ll just kind of hand you the microphone at the end if there’s something you want to get out that we didn’t specifically ask about.

Shawn:Nope, I think we covered it all.Thanks guys!

Bookmark this:

Interviewees:Blaine Wastell and Glenn Block.

In this interview with Blaine Wastell and Glenn Block of the Patterns and Practices Group at Microsoft we asked them about:

• About the Patterns and Practices Group
• What makes them different from a typical product development group
• The software development methodology they use
• Some of the things that Patterns and Practices group has produced for developers.
• The role of CodePlex for their group.

Blaine:   I’m Blaine Wastell. I’m a program manager in patterns & practices, and I’ve been in this role for a little over four years now. Most recently my focus has been in what we call the "client UX program". It’s about providing guidance to customers on developing both smart client and web client applications.

For 12 years before Microsoft I was out in the consulting world helping business customers develop enterprise line of business applications, mainly Web applications.

Glenn Block:  I’m Glenn Block. I’m a technical product planner in patterns & practices. I am a newcomer to the client team. I’ve been in p & p for about 6 months, and with Microsoft for about 2 years. Prior to p & p, I worked in Microsoft Learning and was responsible for building online developer training.

For the 10 years prior to Microsoft I was a software engineer and architect. I worked in various startups as well as a few enterprises. I’ve had a touch of consulting, but it’s mostly been really about building both packaged apps and internal apps ‑‑ both on Web and on Windows. A significant part of my background has been in developing software frameworks for those kinds of applications.

And as the technical product planner on the client team, I’m responsible for the overall strategy, understanding what customers want, and setting expectations on what kind of guidance we’re going to deliver.

Scott:  Talk a little about patterns & practices.

Blaine:  We provide "building blocks" that customers use for line of business application. Microsoft has a number of development tools and technologies which include the .NET framework, Visual Studio, SQL Server, the core operating system, etc. We help customers by providing guidance, in multiple forms, on using those technologies to build line of business applications.

Glenn:  The practices part of our name is focused on the latter, where we give guidance, best practices, and design patterns, to educate developers on known solutions so they’re not reinventing the wheel.

Scott:   I’ve used like the Enterprise Library that patterns & practices produced, and it’s really more of a framework than building blocks. Are things like the Enterprise Library are designed to be best‑practice reference architectures, or are they drop-in frameworks designed to give a productivity boost?

Blaine:  Enterprise Library, and the application blocks, are part of what we deliver. We also have written guidance for areas like security and performance.

We also have another type of deliverable called "software factories" that tie all the pieces together. They have the framework, but they also include the common patterns for a specific area. They include automation; they include what we call a "reference implementation" ‑‑ essentially a sample of how to implement proven practices using the frameworks.

Glenn:  When you talk about frameworks, at the end of the day, to us it’s all guidance. In the case of these frameworks, they are incarnations of the patterns we recommed. We think of them as an 80% solution meaning they may not provide all the answers. We write the code with the intention that people who are consuming these deliverable will rip that code up, and will look at how we’re doing stuff. It’s not simply about giving people a reusable toolkit, although there is an aspect of that.

The idea is, "Hey, look at what we’ve done. Rip out the pieces that you think are applicable for your scenario. If you’re writing something new, you can use this as a reference point and know that it incorporates well‑known patterns."

Scott:  What makes your team different from a Microsoft product team. How are you different from the teams building the .NET Framework, or SQL Server, for example.

Blaine:  We focus on what we call ‘customer connected engineering’. We start with an area that we see is challenging for customers, and then we will put together an advisory board that includes customers. As we design a solution, they’ll validate that our scope and approach is really hitting the top set of challenges that they have.

Glenn:  The big differentiator is that product groups build the platform, while the guidance we create uses the platform.

Product teams focus on taking the platform to the next level. We do guidance, as Blaine described, guidance on using the platform. We show you how to put the legos together in meaningful ways to solve particular problems that you’re facing.

In some cases, through the act of putting those things together, we may pull out some new reusable guidance that we call ‘application blocks’. But the block is not the end goal. The block is just one incarnation of the guidance. When I say ‘block’ , I’m really referring to a reusable library.

Scott:  Could you list some of the things that you’ve produced?

Blaine:  You mentioned the Enterprise Library, which has a number of blocks within it. There’s also the composite UI application block, which is a library that helps you create smart clients. There is also a set of libraries to help you create web clients. One is called the composite web application block. And we also have another one called the pageflow application block which addresses challenges around Page navigation. Blaine:  We’ve done a number of things in the web services area.

Glenn:  We have a Web service software factory; which is guidance on building web services. And we also have new guidance that we’re developing called ESB guidance which addresses utilizing BizTalk as an Enterprise Service Bus.

Scott:  Talk a little bit about the software development methodologies that you use.

Blaine:  Sure. We are an agile organization. We’re probably closer to XP. We start with the backlog that we prioritize. Glenn is essentially the proxy for the customer; the one that will prioritize the backlog for us. From that we do the standard iteration planning, and meetings. We do it once a week, and we have stand ups daily where we go through the status of the standard set of questions: What did we do yesterday, what do I plan on doing today, what blocking issues do I have.

At the end of an iteration, which is every week, we publish our results out on our CodePlex community site. Then customers can look at the CodePlex site and get an understanding of where we’re going. A lot of times they’ll say, "Hey, why are you doing this, because I have these other two sets of challenges you’re not covering," which is good. We’ll also have people say, "Hmmm. There’s a bug in line 35." It’s pretty amazing how closely customers sometimes watch us.

Glenn:  And in some cases even supply fixes.

Blaine:  Right. We’ll also do testing, development, and continuous integration. We use in test driven development, so we write the unit tests first, and then the code.

We’ll generally do pair programming. We’re also distributed team with team members located in Redmond, Buenos Aires and India. This sometimes makes it hard to always do pair programming. If we can’t do pair programming we will have somebody else do a code review before checking the code in. We do automated exception tests where they make sense. We do quite a bit of static analysis on the code that gets checked in.

Glenn:  How about code coverage?

Blaine:  We do code coverage. I have a blog entry that I just put out recently that talk about the quality checkpoints that we go through. On the quality assurance side, we’ll do things like proof and security testing, and globalization testing.

Scott:  You mentioned there’s also a community side of what you do. Talk a little about that.

Glenn:  There’s several different ways that we actually interact with the community and our customers. The simplest way is our blogs. A good portion of the team is heavily into blogging, and we try to give as much visibility as possible into what we’re up to.

Blaine mentioned that all of our stuff is on CodePlex. A big differentiator between the work that we do, and the product groups at Microsoft, is that you can get all the source code.

Also on CodePlex we have work item voting. When people come to us with feedback we say, "Great idea. Throw that on as a work item so the rest of the community can vote." And I can tell those work items absolutely determine what we do.

For example, AJAX support in the Web Client was one of the highest voted work items. As a result we’ve done about four months of work on the UI responsiveness and we have a few more to go. That’s a large investment and it’s based off the community feedback.

In some cases, there’s a work item that’s very broad, and we’ll dig in. For AJAX there’s a server side and client side aspect. When we asked which people were more interested in us investing in, people said, "Hey, we’re more interested in the server than in the client."

Another source is our direct interaction with customers. One of the things we do in p & p is bring a lot of customers in to spend time with the team. That gives us an amazing opportunity to directly interact with them, find out what they’re building, find out how our guidance in mapping against their needs, and find out what the gaps are.

We just had a customer that was here for an entire week. We do that with a number of clients and aggregate the information and learning which we inject back into our efforts.

We also do some on‑site, where we go and visit customers. Since I’ve joined, I’ve visited two customers ‑‑ one in the U.S. and one in the U.K. ‑‑ spending a week with their teams.

Further, we have general blog that generates feedback that we get. We have email feedback that we get. That covers the external customers. But p &amp p also has internal customers. Who are those? These are the product groups.

Lately we’ve been we’ve been working very heavily with Scott Guthrie’s org. We’re doing this is because we want to understand what kinds of challenges the platform is going to be addressing and how it will be addressing itWe want sure that whatever guidance we’re giving is in line with this direction and not opposed to it.

And we also key in to their demand for deep content. They say, "Hey, there’s nobody who’s really telling people how to put these pieces together." We’re also getting very involved with OBA [Office Business Applications]. We’ve had the OBA team coming to us saying, "Look, we need some solid guidance that will give customers architectural guidance on how to develop an OBA application."

Sometimes all these inputs are in conflict with each other and we have to look at the external feedback, the internal feedback, the industry trends, open source, and even our competitors, and decide what we’re going to deliver.

Finally we have our Customer Advisory Boards. This is a key aspect of our customer‑connected engineering strategy. For every one of our deliverables, we have an advisory board of individuals that are in the industry, that are or will be using this guidance. For the Web Client Software Factory, every two weeks for about the past four months, we have an hour meeting where we present to the advisory board what we’re doing, and we get their feedback.

The advisory board is a group that’s able to look at what we’re doing and raise flags and tell us, "Hey, you’re going in the wrong way." Or they may be saying, "You guys are missing this big opportunity over here that really affects all of us." And when we build this advisory board, we select a group of experts across the industry in both small and large companies, some located in the U.S., some located in Europe, some located in Australia. We try to get a really good mix.

They have different needs. They give us a proper perspective on the overall landscape so that we can make those intelligent decisions. I think that covers it. As you can imagine, it’s not an easy task to drill through all of that input and decide which way to go.

Scott:  Right now, the patterns & practices code is released under a custom license. It’s not under the Microsoft Reference license or the Microsoft Public License — which used to be the Microsoft Permissive License.

Blaine:  Everything is migrating to the MSPL.

Glenn:  Yeah. It’s essentially been MSPL without the name. It is going to be MSPL.

Scott:  OK, so the MS Public License.

Scott:  The thing that’s really interesting that now that’s an OSI‑approved license. In the past, Microsoft would say, "We put the source out there. It’s open source," but to a lot of people in the industry…

Blaine:  It wasn’t really.

Scott:  Yeah. To them, unless it’s an OSI license, they don’t know if it’s really open source the way the OSI defines it. So now it is. That leads to the question of, are you now starting to do some of the things that a traditional open source project would do? You’ve got builds. You’ve got bug tracking and issue tracking that people can participate in.

Do you ever envision a day where you’d be taking community code submissions?

Glenn:  We’ve taken steps in that direction with what we call our Contrib community.

Before I worked at Microsoft, I used to work with a bunch of different open source products. I worked with NAnt and NUnit. NAnt has what’s called a NAnt Contrib Project. The NAnt Contrib project is where developers who have the source, are extending NAnt. They’re building plug‑ins, etc., because a lot of these products have a plug‑in model. They want to have a way of driving that back in so that the community can benefit from those enhancements or extensions.

We decided about six months ago to follow the same model and we created a set of Contrib communities. Actually, we didn’t create them, and I think it’s important that we didn’t create them. We worked with the community. For example, with Smart Client we engaged with people like Kent Boogaart, who created WPF CAB, which was his own extension to CAB that would allow you to build WPF applications with CAB.

We also worked with people like Bil Simser, who’s a known MVP who developed Smart Client apps. We worked with a whole group of people, Chris Holmes is another. We brought them together and said, "Hey, we want to make you guys as successful as possible and enable you to take our deliverables, extend them how you want, without us telling you which way to go, similar to the way you can with other open-source projects. We’ll provide a place for you to do that, and we will also work with you going forward. We’ll also hear your concerns and consider how we can fold that into the product."

We do also work with external resources as part of our core guidance deliverables. We have our advisory boards which I mentioned which participate in the design and give us feedback. We have our CodePlex Workitems where the community submits ideas that influence the design. And we even have members of the community (partners, SMEs, etc) that write the codebase itself.

Scott:  Aren’t there models where other people have dealt with this? You take a look at something like the Linux kernel, right? Nobody really owns it. It’s an open source project, it’s covered under an open source license. IBM is a contributor to it. And if stuff gets found in there that infringes on somebody else’s intellectual property, a lot of times the community, or IBM, replaces that with code that doesn’t.

I realize it’s a gradual process, and Microsoft is moving towards more community involvement, but it seems like down the road there may be ways to structure this to where Microsoft is a chief contributor to these projects, but every committer may not necessarily be a Microsoft employee.

Glenn :  In the Linux kernel, yes you do have community participants, but it is not a free-for-all where anyone just comes along and checks in code. It’s a regulated process with a core set of contributors. We operate in a similar fashion. We do have externals that contribute, but there is a process around those contributions.

Scott:  Another thing that’s interesting, when you’re under an OSI license somebody could fork your code.

Glenn:  Yep. And I just want to make clear that we don’t own the Contrib project. We’re not claiming any responsibility for the kind of code that goes into that. We might suggest stuff, like, "What do you guys think about doing this or doing that," but it’s completely up to the contributors what they want to do.

It is the model that you described. We may pour some stuff into Contrib, and we’ve talked about doing that, but essentially the people that own it is the community.

Scott:  Right, right. And those different Contrib projects are essentially forks of your code, and because…

Glenn:  That’s exactly what they are. In some cases, complete rewrites. For example, if you look at the EntLib Contrib, there’s actually an EntLib refactored project where somebody who works at Microsoft ‑‑ Scott Densmore, who helped write a lot of our deliverables ‑‑ is actually ripping apart EntLib and providing a new version.

Similar kind of things are happening in the CAB space. We just recently shipped another version of Smart Client Contrib that actually contains updates to our Updater Block, to get it to work on Vista. This is completely being driven by the community and it’s taking things to places where we don’t have the resources to take them.

Scott:  That’s what I was wondering. Maybe down the road, the only thing that exists is the Contrib space, and people in Microsoft are contributors just like anybody else.

Glenn:  That’s certainly a possibility that we would be open to, but it’s not really about us and what we want, it’s about the customers. Many customers that have serious concerns about using community contributions that might contain code that violate IP. In the case of patterns & practices deliverables because it’s from Microsoft, they have less concern. Other customers however are willing to take that to take that leap of faith. These are the same customers that also use open-source deliverables like NHibernate, Log4Net, Castle Windsor etc.

Scott:  To me it’s really fascinating how the different models are shaking out. We also interview CIOs for things outside of "How Software is Built" and they’re very fond of saying, " I want one throat to choke. I want to know that I’ve got support. I want to know that I’ve got a company that’s standing behind it. That’s really important to me. It’s not about the cost of the software, it’s not even about free as in freedom. I just need stuff that works, and I need someone to hold accountable."

And then on the other hand, on the other side of it, you’ve got open source, right? Where you don’t seem to have those same assurances, but on the other hand the software really does hit the mark. There are people who are experts in their domain. There are people who build really interesting things that a "big company" wouldn’t necessarily think of. There are people who do some innovative things, or start with someone else’s idea and take it in a very different direction.

There’s benefits in all of that. So it’s curious to me what the landscape is going to look like. You take a look at a company like RedHat. They’ve got this open source product, but they don’t really have control over the Linux kernel, or Apache, or the things their customer absolutely depend on. Still, they guarantee that they’ll provide support on top of it, enterprise‑level support.

You’re saying that if p & p’s work was full open source, and the only out that existed was Contrib, some of your customers would have trepidation because they would feel like there was nobody they could really point to who would guarantee the quality of the code, who could guarantee fixes, and those sorts of things.

I guess what I’m saying is it’s a very interesting spot that you sit in.

Glenn:  In a way, it can be the best of both worlds, with some caveats of course. What it means is there may be significant functionality that will exist because of the energy of the folks in the open source community, and that functionality will show up in Contrib but not in our core deliverables. There are other things that the Contrib community might do, that we might decide to incorporate into the core in the future.

Scott:  There’s another thing that makes your position interesting. On one hand you might get feedback from the product team saying, "Hey, we’re not going to have this in the next version. It’s going to be a pain point. It’s something that you guys could take a look at."

On the other hand, you might have a lot of people in the community asking for a feature, but because you’re part of Microsoft, you know that the product team is already building that into the next version of the product. You don’t want to invest in what the community is asking for, because it’s already being built, but you can’t tell anyone it’s being built. The product team still has their plans under NDA.

Glenn:  That certainly happens all the time. That’s certainly a challenge. But what I’m happy to say is that more and more we’ve been able to drive back the experiences customers have had with our guidance to the product teams, which has resulted in many cases in product offerings that come out based on that.

Scott:  When I looked at stuff you produced a long time ago, for example, the Data Access Application Block, it seems like it was almost destined to be obsolete. Eventually Table Adapters got built into the .NET Framework, and that provided the same functionality.

Blaine:  Our goal is to actually help migrate customers from what we produce, to the core product. That’s really our ultimate goal. If the core product addresses scenarios we’ve heard from customers, that’s a good thing.

Scott:  Yeah. Like I said, you guys are in a fascinating spot in terms of being an open source project, or open source projects, you’re really pretty unique it terms of where you sit. Can you share anything about the directions in which you’re going? You’re obviously moving towards more open. There’s certain constraints on that. Anything that people should look for over the next few months?

Glenn:  One of the things I think we’re moving on, and Blaine can touch on this, is bringing more of what we’re seeing to product groups. As an example ‑‑ and we’ve been trying to collaborate much more tightly internally ‑‑ one example is this new MVC framework. Want to talk a little bit about some of the work?

Blaine:  At patterns & practices we’ve been supporting, as Glenn talked about, many of the common patterns in the UI space. MVC [Model View Controller] is a common one. MVP [Model View Presenter] is another one. And since we’ve been in the space, we’ve been working with the ASP.NET team to help define the MVC work that’s coming out of there.

A big part of this is really Scott Guthrie and his team’s vision. We are bringing some of the feedback that we’ve heard from customers to the ASP team.

Glenn:  We’ve actually been working on the design with them. It’s really been a partnership. I think those kinds of things are efforts that you’re going to continue to see.

The other big thing that I think you’re going to see, is the fruits of a transition we’ve been going through internally within patterns & practices. We’re re-evaluating carefully at how we deliver guidance and what is the best method. , and we’re also looking at taking the principles of SCRUM that we apply within our teams and bringing it up to a higher management level with p & p. This includes having having a cross p & p product backlog that we can be very visible with customers about. We want to show, "These are the things we’re looking at. This is what’s a high priority. This is what’s not."

On the subject of how we deliver guidance, although many customers have nothing but good to say about our factories, there are other customers, that say they are too complex. They also tend to not be as applicable in in what we call "Brownfield scenarios" which are existing applications. Instead, they tend to be most useful when you are starting from scratch. Another thing we’ve seen is that industry-wide there has been a shift away away from comprehensive “do-it-all” type frameworks to a more pluggable services approach. Finally we’ve had a set of customers who tell us that the biggest value we can provide is to focus more on patterns themselves and how to use them (including samples and reference implementations) rather than framework type libraries that implement those patterns. We’ve heard thse customer concerns and are taking them seriously. We’re looking at how we can break things into smaller blocks. We don’t mean exactly like the earlier application blocks, but instead how to ship our deliverables in an incremental fashion that offers customers more options as to what they use.. For example if I want to use some aspect of the factory without having to take everything, then I can. And, we are also looking at patterns and how we can make them more a first class citizen in our guidance.

Around this, we’ve put a stake in the ground with the current release of the Web Client Software Factory and with our newly announced WPF Composite Client.

By getting these things to be smaller, we can ship more of them more frequently, we can make more intelligent decisions about what we ship and what we don’t ship. We can build guidance that has more applicability in both the brownfield scenarios and well as greenfield — which is new development. We also give customers more choices as to how they incorporate our guidance within their environments.

Ultimately, we can reduce the complexity in evaluating and using these deliverables to make them approachable to a wider audience.

As you can tell, we’ve certainly got a lot of work ahead of us.

Scott: Thanks for talking. You opened up an interesting subject at the end, and maybe we could talk again in the future about it.

Blaine:  You bet.

Bookmark this:

Update: It seems that this isn’t seen as happy news by all. There’s an article on eWeek that’s just too irrational and frothing to pass up, claiming that this is all a ploy by Microsoft to kill Mono. As Microsoft is officially supporting Novell’s efforts in porting Silverlight to Linux (on top of Mono), the evidence would indicate that Microsoft is doing this to support .NET developers, and not as some clever conspiracy to kill off Mono.

Bookmark this:

Interviewers: Scott Swigart , Richard Bowler

Interviewee: Matt Gibbs

In this interview with Matt, we asked him about:

• Approaching open and closed source SDL
• Development challenges for UI Framework and Services team
• Handling security within feature development
• Incorporating SDL into ASP.NET
• Launching multiple distributions with open source
• Testing against multiple operating systems
• Testing cycle for features
• Managing bug fixes and feature stability
• Following a development cycle
• Microsoft philosophy on product development

Scott Swigart: Hi, Matt. Before we start, go ahead and introduce yourself.

Matt:I’m Matt Gibbs, and I’m the Development Manager in the UI Framework and Services at Microsoft. My team is responsible for ASP.NET and AJAX as well as parts of Silverlight.

Scott Swigart: Thanks! We’ve been commissioned to look into the premise that open source and closed source SDLs [Software Development Lifecycles] are fundamentally different, and that the differences manifest themselves in the final products in tangible ways.

That’s not to say that one is better than the other – certainly both models have their strengths and weaknesses – but it’s really designed to get information out there for two audiences: one is software vendors who are looking at building products. One of their first choices is whether they are going to go with an open source or a closed source model, and we want to look at some of the ramifications of that choice.

The second audience is people bringing products into their organizations, and we want to uncover some of the basic expectations that they could have around open and closed source, or some of the characteristics of each model that might impact their choices.

Richard Bowler: You’ve been the Development Manager for ASP.NET for about a year, right?

Matt: Yes, about a year.

Richard: What would you say are the major development challenges with that product?

Matt: I think for us, the biggest challenge is that we have a large customer base, so as we innovate, we have to be very sensitive to backwards compatibility. We have a fairly complex platform, so innovation is challenging when you’re trying to make sure that existing applications people have built with it continue working.

Richard: It’s a pretty complex project, and it’s built on another complex project, which is the CLR; to co-develop with the CLR team, do you just take what they give you and work from there, or how does that work?

Matt: We work really closely with them. In fact, we meet regularly to go through code changes we’re making with members of the CLR team, so that we’re all in sync.

Richard: It sounds like you’ve been able to direct feedback to them, saying, “We really need this kind of capability, let’s fold it into the next release,” or something like that.

Matt: Yes. And they’re pretty sensitive to how their changes impact us. They run compatibility tests and stress tests against our scenarios, as well.

Richard: It seems like a huge challenge, when you consider the number of software products that Microsoft has on the market that interoperate. It seems like a huge challenge to keep them all working together.

Matt: For the framework we’ve actually held a pretty strict line. I am part of a committee where we review things that were classified as “potentially breaking a developer’s code,” and the default was that those weren’t allowed – that was kind of the highest priority. Things like “security fixes” would be allowed. Then there always needs to be a way to get back to an earlier version’s behavior, through a configuration parameter, for example.

Richard: Speaking about security, we talked with Michael Howard, and also with James Whittaker about the SDL. It looks like it’s really paying off for you guys. Were you involved in the development internally when the SDL was being instituted at Microsoft?

Matt: Yes, actually I worked with Michael Howard on IIS where we ran into a few security issues that helped highlight the need for a more structured approach.

Richard: I know what the benefits are, because we discussed it with Michael and James. But what were the main difficulties in shoehorning those new processes over the top of existing SDL processes?

Matt: I don’t think it really came down to process difficulties, as much as it came down to just helping developers recognize that the mindset they had been using needed to be front-loaded with security all along.
In our first round, everybody recognized that we needed to pay even more attention to security, and make it a higher priority. But then the realization was that it can’t be one phase in the development process. It really needs to be something that we pay attention to all along.

Richard: To put it another way, then, you had to get a wrench on the culture. You had to make security important in everyone’s mind.

Matt: Right, and it really had to be the case that everybody came to the same conclusion. You couldn’t just tell people it was important – they had to see what kind of impact it had to customers, and recognize it themselves.

Richard: Did you get much internal resistance from individual developers, or was there pretty much across-the-board buy-in?

Matt: It was pretty wholesale – everyone recognized it. I think particularly since I’ve been involved in web platforms, and we saw a couple of places where we’ve made mistakes, everybody recognized, “OK, we have to do something different, and be more practiced in this,” because otherwise we’re not going to get the right outcome.

Richard: Where was ASP.NET in the process of instituting the SDL? Was Version 1 out before the SDL really came to the fore?

Matt: No, when the process was launched we were already in development. It was during that cycle where we decided to take a break, and we went back and said, “We need to invest time in this process.” Everybody did some extra security training, and then we essentially stopped development, and did a pass for just security. We looked at every feature, all the code, looking for specific issues, just to make sure we had done all the right things.

Scott: One of the things that I found looking at the open source side of things, is just the sheer number of distributions that they have to ship for a product. MySQL for example, I think has 16 distributions, because it has to run on all of these different flavors of Linux, none of which the people working on MySQL really have a lot of control over. Those versions of Linux are all rev’ing independently of each other, so the underlying operating system that it’s dependent on is changing all the time. You’ve got a whole ton of applications that are built on top of MySQL, and are therefore dependent on it.

I can’t even really imagine exactly how to test that, and I know that in the open source world there are issues about, “This version of MySQL isn’t compatible with this version of WordPress, which isn’t compatible with this version of Linux.” So, they are always fighting these compatibility issues as things just rev independently of each other all the time.
Microsoft has some rather similar challenges, in the sense that IIS revs with the server. ASP.NET revs on its own timeline, which isn’t tied to the base operating system, and Atlas is also on its own timeline to some degree.

Talk a little bit about the kind of test matrices that Microsoft builds, and the amount of resources in terms of hardware and testers that come to bear when you guys are getting ready to release something, or while you’re developing something to ensure compatibility with all the versions of all the things you ship.

Matt: It’s probably more expensive than you might initially think, because in the same way that there are a bunch of different Linux distributions, we also still test against a lot of different supported versions of the operating system.

We’ll test against XP and Windows 2000 and Win2k3, with different levels of the service packs and with components installed; with Visual Studio installed, without Visual Studio installed, and then different flavors of IIS. We’re already testing with IIS 7 on Vista where we’ve got a lot of integration work going on. When we go to do compatibility checking and functional testing, we have a fairly large lab. And the run time, unfortunately, is on the order of weeks to really ensure compatibility.

Scott: What I’m envisioning is that somebody pushes a big button, and a hundred machines get configured with tests. It all comes up and runs, and results get cranked out — test, pass, or fail. Is it something kind of like that?

Matt: Yes, and they categorize tests into different levels of priorities so they can say, “We’re going to do a quick sanity check on something,” and that can take a couple of days. Or, they can say, “We want to run the gauntlet, and run everything we’ve got,” like before a service pack or something, and that will take much longer.
It covers everything from different language packs, to different service packs installed. One time-consuming thing is that errors that are false positives come up, and they need to be investigated. These might be just test issues, and that takes time as well, because you don’t want to ship with something that might be a problem, so everything has to be investigated.

Scott: I know that Microsoft’s got certain terminology around this. I’ve heard of something called the BVT, which I think stands for Build Verification Test. There are things that have to happen before something gets released as a CTP, Community Tech Preview. Can you talk a little bit about that, in terms of what the different levels of testing are, and what kind of things have to get verified on those different levels?

Matt: There’s a scaled approach. On a developer’s machine, we’ll have what we typically call Developer Tests.
The developer cranks out a lot of small standalone tests against the feature he’s building, against his APIs. He’ll run those every time he goes to check in code. We’ll aggregate those for our dev team, so that you’re running a fairly lengthy set of them. But, they still run within just a few minutes, so that you can continue making fast development progress. Then we’ll run a superset of those before we check in.

Scott: So, that’s to make sure that one developer’s change didn’t break some other developer’s code.

Matt: Yeah, that the basic functionality is intact. Then, the test team runs what they call a set of nightlies. They’ll run those on every daily build that comes out, they’ll kick off an automated set of nigthlies. They’re small enough in scope that they can run overnight, and any failures can be investigated fairly quickly. They are what they call “Pri0” test cases; and that would be mainstream use-cases.
Then, the superset of that is something that’s maybe ten times the number of test cases, and that’ll run for days to verify the full functionality. That’s literally thousands upon thousands of test cases.

Richard: So, your nightlies maintain a level of sanity about the ongoing nature of the code base – yesterday it was still good, today it’s still good, tomorrow somebody’s checking in something bad. Let’s back up and fix this so you never get too far out of whack.

Matt: Then, you may find something when you go through a full test pass. When the tester originally made the test plan, they may have said it’s a fringe scenario, so it was classified as a priority two test case. You might only run it before a release, and then something breaks.

Scott: Microsoft has a lot of people who blog, so there are obviously a lot of people on the product teams who are fairly transparent about where they are and what they’re doing. I think Scott Guthrie put up a big blog post where he said, “You know, there’s a point in time where we don’t necessarily fix every bug because we have to weigh the risk of impacting the stability of the product.”

Every time we touch a line of code, there’s a risk that we’re going to hurt the stability of the product, so there’s a point where we switch from telling a certain team what we’re fixing to actually having to get permission to fix certain things because of where we are at in the stabilization process. Am I characterizing that right?

Matt: Yeah, we’ve spent a lot of time mining our bug tracking database, and you look at data points like, “For X number of bugs, how many fixes actually cause some other problem?” The closer you get to shipping, the more likely you are to look at something and say, “This is really not a very important bug, and people are just not really going to see this. It’s good that we did the due diligence and found it, but this late in the game, you run the risk of breaking something more important.” You might just leave a bug in there and fix it when you have more time to shake out the problems.

Richard: I guess if you didn’t do that, you’d never ship.

Matt: Exactly. With new engineers, you’ll regularly see them hit this kind of frustrating point when they have a bug, they want to fix it, but it’s time to ship and they get told no. Their point of view is that it’s a problem that should be fixed, and they have to be told that they can fix it, but not until the next version

Scott: I have to admit that it’s odd to think that leaving bugs alone raises quality.

Richard: In my own process over time it seems like when we get down to the mode when we’re primarily testing an implemented product and engineers are working on bug fixes and not new features and behaviors. It seems like as you get closer and closer to when you need to ship it gets harder and harder to jump through the hoop of where a bug ought to be fixed. At some point it’s got to be an absolute showstopper to stop the shipment; it’s got to be a crash or something like that, otherwise you’re just not going to fix it until the next release.

Matt: Yeah, we get to the very end where we call it recall mode, where the question is whether you would actually take the product off the shelf in order to fix this issue. At this final point in the process, unless we’d recall the product, we don’t fix it.

Scott: Another thing that’s pretty interesting to me is the whole way that Microsoft does milestones. Again, my impression of it from the outside is that somebody will say, “OK, these are the features that we’re targeting for milestone one, for M1,” and so developers will go off and they’ll be working on those features. There will be a graph of the bug count that’s just climbing, climbing, climbing; the number of test cases written are just climbing, climbing, climbing.

At some point, it seems to switch over to where new features aren’t being written, it’s focusing on stabilization for the M1 milestone. So, there’s a trend down in the bug count. It seems like there are a lot of different variables that are being looked at in those stages; what’s happening to the bug count, what’s happening to the perf numbers…
Can you talk about how all that stuff comes together? Am I characterizing these milestones correctly? Is that sort of how it works?

Matt: In the past, that is how I would have characterized it – a sort of waterfall approach where we crank out a bunch of features and accumulate bugs, and the test team cranks out test cases, and then we play catch-up and try to satisfy all those test cases and bring the bug count down isn’t the most efficient. But we’ve actually shifted in our group to a little different approach that we think is working better.

Instead, we’ve gone to what we’re calling a feature crew model where the dev, test, and PM work together from the start on defining the design, the set of test cases, and the unit tests, and they drive for more quality up front before the feature even gets checked into the product. So, they’ll work in an isolated source code branch getting a feature to a higher level of quality and then get it into the product.

Richard: It sounds like you’re moving, at least to some extent, toward a more Agile model.

Matt: Yeah, it’s ended up being a hybrid of Agile development with features of SCRUM, but you couldn’t really call it one or the other. It still falls into a sort of waterfall effect where we have a milestone of a bunch of features we’re going to try to get done by a certain time. There is still this mentality of feature crews driving for a deadline, so we picked the pieces of a bunch of different methodologies that seem to work well for us in delivering commercial software, and we are trying to make them work together.

Scott: Now, I’ve heard a little bit about these feature crews. Correct me if I’m wrong, but that seems to have been developed after the ship of Visual Studio 2005. It came out of … I think they called it like, MQ or Milestone Quality, or something like that?

Matt: Yeah, it was kind of built around the milestone for quality.

Scott: So, some of the ramifications of that are that if there’s a feature you’re dependent on to build your feature, it might take you longer to get that. But, when Microsoft releases things like Community Tech Previews, they’re going to be more stable by default, because things were only checked into the source base that passed a higher quality bar. Is that right?

Matt: That’s the goal, and we were able to benefit from the approach with the ASP.NET AJAX release. We embraced the feature crew model when it was still ramping up for the rest of the division, and it worked out pretty well for us. We learned a few things we could do better.
We also learned that some of the dependencies became a little bit difficult to manage. If you truly had a feature that needed something else to be completed, then you ended up with a staggered set of feature work waiting for future builds. You needed an equal set of things that could be done early, in order for everybody to keep being productive.

Richard: It sounds like what you’re doing is trying to make sure that your interim quality is higher by testing feature-by-feature, and by enforcing a quality standard on individual pieces instead of waiting for the entire conglomeration. Is that a correct characterization?

Matt: Yes, hold the line on quality all along. Don’t accumulate debt in order to get more features done earlier.

Richard: That seems really smart to me. There’s an old saying you can’t test quality into a product, and that’s really true. It seems really smart to me. Was that inspired by the SDL’s focus on early process involvement of quality assurance?

Matt: Yes. Before, we almost had two instances of lifecycles. You had development working on one lifecycle model, and you had the quality assurance teams running along six to eight weeks behind the dev team.
But, that wasn’t really working efficiently because all we were doing was having one team try to play catch-up with the other team. The dev team was cranking out features, and the test team was finding problems with it. Then, the dev team was trying to catch-up on the problems that the test team found.

Scott: With Visual Studio 2005 when it came out, there were a lot of issues around stability and quality, and things like that.
It was easily the most complicated development environment that Microsoft had ever built, maybe that anybody had ever built. It had amazing levels of cross-product dependencies; there were big products of their own that were put in there, like Team System.

Some of the impression, at least from the outside, was that it didn’t all come together and stabilize at the end the way that everybody had hoped. So, is that correct? Do you feel like maybe the initial version wasn’t as stable as everybody hoped, and so some of these things have been done as a response?

Matt: That may be the conclusion from my team’s perspective, focusing on the platform. I think we probably feel like we’re in pretty good shape because we have criteria and goals that were met.
But, I know in the MQ timeframes a lot of people focused on a lot of little things that weren’t quite right that they wanted to fix up for the next release. That was primarily targeted at the development tools environment, not the platform.

Scott: You work in the open source world in a sense with Atlas, right? Because there are a set of controls for Atlas that are effectively open source in that Microsoft puts the source out there. They’re maybe not open source in the sense that anybody in the world can just modify the source and submit changes to it, but please talk a little bit about this foray into open source with some of the Atlas controls.

Matt: The AJAX Toolkit Project is a separate team around the corner from us, and it is shared source. They take code submissions from the public. They have released the really snappy UI controls that everybody uses on top of the Atlas part of the platform.
It’s been well received, and people are able to grab the source, and build it, and contribute. It’s a little different for us in comparison to what we’ve done before. We’ll see how much the community itself drives lots of new controls, or if it’s something where they like the fact that the controls are out there, but they don’t really actively contribute a lot.

Scott: Certainly a key benefit of open source is that the source is out there for you to learn from. I think that even taking the Windows Forms controls and the ASP.NET controls, certainly a lot of best practices might have been gleaned from being able to look at how Microsoft did what it did.

Matt: Well, we’ve done that with the Atlas release now, too. That’s a first for us, and I’m really proud of it. When we released ASP.NET AJAX, we released the source symbols for it so you can attach a debugger, and step right through our source code.
It’s not the same as taking source code in from the community, but we look at it as being very open and transparent. It’s one of those things where in the open source community, everybody says, “I have the source, so I can do what I want with it.” From my perspective, though, people tend to say that but not do it.

Scott: I think they call it the Berkeley Conundrum. It’s a variant on “if a tree falls in the woods, but nobody hears it, it doesn’t really make a sound.” If there’s so much open source code out there, and not really enough eyeballs to actually look at it, does it really matter that it’s open source?

One of the things that seems to me like a huge challenge is that inside of Microsoft you can control exactly what people get to do to. You can control the training that they have to have around security, and performance, and best practices. You can control the unit tests they write, and you can control the process that the code goes through.
When you take a look at open source, you just get code delivered to you and you don’t have any control, or any view into the process that was used to create that code.

From Microsoft’s perspective, you’re taking submissions from the community; how do you ensure things like security, performance, and reliability, when anybody could be submitting the code, and you don’t have control over how they got there?

Matt: I’m not intimately familiar with it, but my understanding is that the team has essentially taken on the burden, so far, of ensuring some of the security themselves by essentially laying our security process on top of the code before it goes out to the public. I also think there’s an element of caveat emptor.

But, because it’s coming from Microsoft, there has been a certain expectation of quality. At this point, because we have a dedicated team working on it, they’re in the code and it’s still going through quite a bit of the Microsoft process.
Also, at this point, you can’t just submit and have it go right out to CodePlex. You submit as a team, and your submission is then under review for inclusion on CodePlex.

Richard: What do you think are the advantages of closed source over open source?

Matt: In my mind, it comes down to probably two things. One, the customer knows they have somebody to go back to, that there is a company that they’re buying software from that is backing it up. Two, there is a full-time developer that is working on advancing the software.
Open source has done fantastic things with having a community effort drive things to do good software. But, the fundamental difference is that with proprietary software you have an entity that’s responsible and people employed to push things forward.

Richard: What about the development of that process itself — the mechanics of designing, implementing, and shipping a piece of software. Do you see advantages to the closed source model, and if so, what are they?

Matt: I think you may have a little bit more of a culture of discipline but I’m not sure that it’s really that much different. I think there’s so much passion from people in the open source community in what they’re building that it may be the same kind of thing. The same kind of respect somebody has for their full-time job.

Scott: There are two myths that I frequently encounter, one related to open source, and one related to proprietary software. The open source myth that I hear again and again is that many eyeballs looking at the code ensures security.

Michael Howard really threw down the gauntlet on that and said, “Show me the data, because there’s just no data to support the idea that you can make sure code is secure just by open sourcing it and letting people look at it.”
On the closed source side, one of the things that I often hear as being an advantage is this notion that there’s a company to go back to, there’s a company that stands behind it. My impression is, I don’t have any guarantee that a bug that I find will get fixed in the product. I don’t have any guarantee that a problem that I’m running into will get solved by the company that produced it. I can file a bug, I can call product support; maybe I’ll get a resolution, maybe I won’t.
Do you think that Microsoft or any other company really offers that level of guarantee?

Matt: In my experience having worked as a quick-fix engineer where we investigated customer reported problems from PSS, it was very serious. If you couldn’t find some way to workaround it, it needed to be a product fix. Now with that being said, if there was a reasonable workaround, then you’re back to the issue of whether it’s worth the risk to go patch product code.

There’s not a case I can think of where a customer had a serious problem and we didn’t either find them a workaround or patch the product in order to get them back on track. The open source community can’t guarantee every requested change will be made either.

Scott: I guess the only caveat to that might be that obviously, it’s not going to be instantaneous, in either the closed source or the open source world. I remember a problem I found, and the answer was it’ll be fixed in Service Pack 2, but Service Pack 2 didn’t come out for another six months.

Matt: Very few fixes in proprietary or open source development are instantaneous. More complex issues may take longer, and again there is the concern about compatibility. When something like Service Pack 2 ships, it’s had many hours of testing devoted to it to ensure that even small fixes are getting good coverage.

Scott: Clearly, from your perspective, that guarantee of supportability is real, but there are just certain kinds of physical constraints on how fast the resolution might be available. But in your experience if there is a critical problem, either the customer gets a workaround for it or Microsoft starts working on a fix?

Matt: That’s been my experience in ten years here.

Richard: We talked to one of the guys that works on the Linux kernel. He basically said what happens is that people turn code in to the mailing list, and then everybody takes a look at it. If they’re interested and they snipe at it, then the developer makes some changes. Then, ultimately it either gets adopted by the maintainer – the person who owns that section of the kernel – or it doesn’t.

So, at least in some sense there’s no process to enforce quality during implementation beyond peer review. It seems to me intuitively that that’s a disadvantage. But, I just wanted to get your take on it.

Matt: Of course we’re not working on the kernel, but I know that before a piece of code I write is going to get anywhere that it’s going to get exercised quite a bit. It seems like inherently that’s an advantage.

Scott: Matt, talk a little bit about motivation inside of Microsoft. How much is it sort of computer-generated, and how much is it, “Developer Number 13 had this much code checked in, this few defects, found this many bugs…?” How does it work in the “big house?”

Matt: For us it’s all about our developer customers. It’s about the level of enthusiasm and people’s feedback, and it all feeds off of that. We get our customer’s excitement, what they want to see, what they’re going to be able to build. They say to us, “If only you added this feature you could save me time, and here’s what I want to see next.”
The people here in Microsoft are completely driven by what our customers want to see next. We may be in a bit of a unique position because our customer is a developer, and developers get really excited about the platform.

Scott: I’ve heard from people who’ve interviewed at Microsoft come back and say, “There really is a sense that by working at Microsoft you have an opportunity to change the world.” I would guess that many of Microsoft’s products … if you take a look at Office, Visual Studio, Silverlight, and how you hope those products progress over time, most of these are big bets. In most of these cases, you are part of something that you hope is going to be really big and industry-changing.

Matt: Yes, it’s the kind of thing where I’ve had people come up to me at ASP.NET conferences and say, “You changed how I do my work. It’s been an incredible experience and I can’t wait to see what you guys do next.” It makes an impact on their life. It makes it fun.

It’s different than what Scott was driving at; the “how many defects” and “how many lines of code” kind of thing. We really run off of developer-driven enthusiasm. There’s an enthusiastic, sharp, hardworking group of people that are really focused on the customer and how to make better developer experiences.

Richard: Well, thanks very much for talking with us. I appreciate it.

Matt: Sure.

Bookmark this:

Interviewers: Scott Swigart

Interviewee: Rod Johnson

In this interview with Rod, CEO of Interface 21 and founder of the Spring framework, we asked him about:

• Development methodology of Spring framework
• Contributors to Spring
• Selection of code and features for the Spring framework
• Ways in which developers support open source
• Open source development cycle
• Importance of technical support
• Code aggregation in open source development
• Choosing open or closed source within small businesses

Scott Swigart: Rod, thanks for taking the time to chat. If you don’t mind, please take a moment to introduce yourself.

Rod Johnson: I am the founder of the Spring Framework, which has become the effective standard for enterprise Java development. We also have a spring.net edition. I am CEO of Interface 21, which is the company behind Spring. I’m also the author of several books on J2EE development.

Scott: Great! Talk to me a little bit about the development methodology of the Spring framework. How does that work? Every open-source project seems to have its flavor. Tell me how Spring is built. How do ideas go from conception to features to in production or in development to release?

Rod: OK. I think when people try to understand what open source is, they instantly think about code and that the software is created by a disparate, far-flung group of people who work in their spare time. In fact, to understand the open-source model you need to step back from code and look at the totality of things, which includes documentation, includes support for users, and includes the discussion of ideas.

What we see in the core of our software is that the vast majority of the code is written by a small group of exceptionally good people. I don’t believe there is any other way to produce exceptionally good software. We have known for many years about what Fred Brooks calls the The Mythical Man-Month.

Scott: Right.

Rod: Open source doesn’t change that. We know that if you throw more people at a problem you don’t get a better solution. And it doesn’t matter what approach to distributing the source of the software you use. You’re not going to change that fundamental fact. If you want exceptionally good software, you want a cohesive team of exceptionally good people and you really want that team to be as small as possible.

Scott: How is that team motivated to work on the project?

Rod: The core people who connect to Spring daily are employees. They are being paid to do this. That model works very, very well if you look at open source in terms of the enterprise Java space and also obviously of the Linux space. You see that the model is evolving towards paying the contributors who write the code. For example, I think 21 out of the top 25 committers on Linux are employees of large corporations. The majority of the committers on Spring are Interface 21 employees. Another good example of this is JBoss, the Java application server. The majority of the code there comes from employees of JBoss, who are now employees of Red Hat.

Scott: I want to get your insight on the Linux kernel. My understanding of the Linux kernel is that unless Linus likes a certain patch, it’s not going to make it into the kernel. And yet, the bulk of the development is done by people who work for IBM, people who work for Red Hat, Intel, AMD, whoever. Right? Corporations are doing the bulk of the work. And yet what makes it in or out of the kernel isn’t really under the control of a company. Do you have any insight into how that works?

Rod: I am not an expert on Linux kernel governance. From what I see, one thing that is very, very important to developing high quality software and maintaining that software in high quality for the long-term is consequences: consequences of success, consequences of screwing up.

In the case of Linux, what we have is not a single entity, but we have a number of companies (and in Linus’ case partly also an individual, obviously) and a number of parties that are really baked into the system because there are large consequences if they screw up.

I don’t think one entity needs to be responsible for a piece of software. I think you can end up with a kind of partnership, but there have to be consequences if something is screwed up. That is one thing that scares me about the rather naive volunteerism view of open source because what are the consequences to a volunteer? Sure, in many cases they may be professionals. They may feel as a sense of personal ownership that they cannot be satisfied unless something is as good as it possibly can be. But there ultimately are problems because they don’t necessarily suffer the same kind of consequences if something isn’t taken forward.

Scott: What people have told me is that there is sort of a developer Darwinism that happens. In other words, to get your code in you have to submit it to a mailing list. You have to submit it to what one person described as a “full contact code review” and in some mailing lists you’ll just simply flat out get ridiculed and won’t be taken seriously if your code isn’t to a certain level. So, it seems reputation based, I guess. You can’t really get code in until it’s passed code review and up the chain of maintainers. So, on one hand there isn’t a lot of consequence to writing bad code, except maybe it was a complete waste of time. Your inability to do it well was pointed out for the whole world to see. So, it seems there is a certain amount of pressure to sit up straight and code well, especially if you’re going to try to get stuff into very high profile, very core open-source products like the Apache, or Linux, or some things like that.

Rod: Yeah.

Scott: One of the things that you pointed out is that people who are going to use open source shouldn’t consider it a free ride. It isn’t just something that you should just leech off if you want the stuff that you’re dependent on to continue to thrive, grow, and be there for you, and continue to be enhanced down the road.

But, at the same time, most of the core contributors to open-source products are highly skilled people who are employed by a company, and this is what they do all day long. So, if I’m going to be a user of open source, how could I best contribute?

Rod: The first thing is that I would hate to give the impression that openness means nothing. There is one interesting characteristic, for example, about a piece of software like Spring, which is overwhelmingly written by one company, compared to closed source.

You have the ability to work up to being a committer of the software. Not many people will take that up. And we may choose to approach you to see if we can hire you if you do it. And that is a way of expanding and renewing the developer base.

So, it is open in the sense that there is opportunity for people to come to be committers on the software through the community based on merit. That is interesting because if I want to be a contributor on the Java-based IBM’s WebSphere, I’m sure I could add some value based on my background to WebSphere, but I can’t put my code in there. There is no way I can do it without becoming an IBM employee.

Scott: Right.

Rod: So that is a genuine difference. There is a true meritocracy. So, coming back to your question, which was “How do you pay? How do you contribute back?”

Scott: Yeah, how do you contribute?

Rod: Well, I think the first thing is to remember that although it’s very important that there is that meritocracy based on code there are several reasons that the majority of users are never going to contribute to that code. The majority of users are not going to modify and compile it. The majority of users are going to use it essentially as if it’s a shrink-wrap product.

I make all of our consultants just use the binaries. If they’re working with our product with customers, we don’t like them to be working with the source code and making changes to the source code of our product.

The interesting thing is that it simulates more accurately how end users use it. So, often we’ll go into customers where we’ll see that they just realized that you can mount the source code and if you want to step through the same work source code…

Scott: Right.

Rod: …so, it just doesn’t make sense for most end users to even consider modifying the software. Consider that Spring is 1.2 million lines of code. Some of that code does things like transaction management, the fundamental configuration of your application. Some of it executes in virtually every runtime call set. It doesn’t matter whether its open source or closed source, you really don’t mess with code that does that kind of thing unless you know what you’re doing.

Scott: Right.

Rod: The process of knowing what you’re doing…For example, in our transaction infrastructure I, myself, as the original co-designer, made a change about a year ago. I spent almost a day conceptualizing the current state of the code (which one of our other guys had modified) and figuring out exactly how I could do these [changes] in a safe way.

I am a strong enterprise Java developer with 10 years experience. I originally co-designed this module so I know how it is meant to work, yet it still took me almost a day to feel that I can safely make a change without risking breakage of a core part of the software.

Imagine if I were someone coming from outside who didn’t have 10 years of experience, hadn’t co-designed the module. What is the probability that I’m going to make that change in a robust way unless I spend maybe five days fully getting inside it?

Now, if I am an end user and I am writing software, for example for a bank, like many of our users, they don’t want to pay me for five days to figure out how the Spring framework works internally. That is irrational. There is little economic benefit for end users. There is a high probability of being economically irrational in modifying the open source software.

Another good example is Linux. I did a lot of C-programming early in my career. I can code in C. Am I ever going to look inside the kernel and change any of the kernel? This makes no sense, whatsoever. So, code contributions and changes are not likely. However, there are many other things I can do if I wish to give something back.

One thing I can do is to help other users of the software. Most open source projects have forums and mailing lists. Every time a developer answers a question for a user that developer is distracted and can’t write code to improve the product. Both things are important but no one can do both simultaneously.

One thing that has worked brilliantly with Spring is, especially early on, we made a big effort in the core development team to answer users’ questions and be really, really helpful. That set up a virtual circle, where users saw the Spring development team was really helpful when they were first starting up. Gradually they used the software more and became experienced. Then, they see other people asking questions and they think, “Hey I know the answer to that question. Maybe Ron doesn’t have to spend time answering that question. I’ll get in there and I’ll help the other user.” That’s a valuable way of giving back.

Another thing is promoting the software because open source software, like any other form of software, can only be successful if it is popular. For example, you like a particular piece of software. You wonder whether there is a user group for that software in your city. Maybe you have time to organize the user group. Maybe you can persuade your manager to provide coffee and a room if someone else is willing to organize it.

Another way that you can contribute is through issue tracking. If you see something that you think can be better, go and report it. An issue tracker is the enhancement request. That works very, very well with open source because that is one area where access to the code really matters. We see people who make enhancement suggestions with pretty deep knowledge of the code. We might get an enhancement suggestion saying, “By the way, if this class extended some other class it would allow the possibility to do X, Y and Z. Have you considered that?”

What happens is that one of our developers looks at it and says, “Oh God, I would never extend that class. Oh, but I see what he’s getting at. Oh, yeah! He’s right. It would allow X Y and Z. So, I wouldn’t do it that way but I can see that there is a lot of value in what he suggested.” The ability to look at the software is quite interesting from that point of view. It’s not necessarily so interesting in terms of modifying the software but it’s quite interesting in terms of the ability to provide suggestions with as little or as much context as you want.

Finally, and this is the way in which I think most financial organizations will end up giving back, go find someone who is contributing to or sustaining that software and buy a support contract. Not only is that going to sustain the software very effectively but also open source software needs the ability to have maintenance, especially as open source moves into the enterprise. What happens if a system that does trading goes down at 2 AM?

There’s not much point if one of your guys gets paged. He didn’t ultimately write the software that is involved in it and the fact that it is open source doesn’t really make any difference. The need for a similar level of SLA and mission-critical support is exactly the same.

Scott: So, that’s one of the things that I want to drill into a little—two things actually. One is support and the other is the software development lifecycle.

It looks like if you take a look at something like Spring, there is nothing magic about the way it is built perhaps. You probably have internal meetings. You come up with plans. You come up with specs. Engineers are assigned. Software is written. It goes to quality assurance.

Open source gives you much tighter coupling to the user base and the community at large. They provide better issues because they look at the source code and even recommend a certain implementation and talk about the merits of a particular implementation. They file bugs directly into the same bug database that your internal people use. So, open source gives you that tighter coupling with your users. But—and I don’t think this is unique to Spring—I am finding this a lot; it is still built the way software has always been built.

Good people. Good process. Good QA. You have people assigned to do documentation. You have people assigned to do QA. Even though those are not the things that people would volunteer for, you’re not dealing with a bunch of volunteers. You’re dealing with paid employees.

Am I accurately summing it up?

Rod: I think you’ve got it. You raised one really interesting question there. You said that the community uses the same issue-tracker as we use. That is obviously true. We don’t have anything around the software in terms of management of bugs that isn’t publicly visible.

A big part of open source, to me, is free use of information. I know who wrote a piece of code, I know when it was last changed; I know what issue that person was trying to fix when they wrote that code. I can see all that information.
It also means that there is a great deal of security in adopting software that was produced by small companies. For example, at Interface 21, there are slightly under 50 people. We’re growing pretty rapidly, but that’s still a pretty small software company.

So, let’s suppose that our product was closed sourced. We would, in each deal, be dealing with escrow agreements, that kind of thing. Frankly, other people would not think of using our software. They wouldn’t necessarily know whether the internals of our software were any good. They could see it from the outside, but they wouldn’t know, for example, what’s our philosophy of bug fixes, what is the quality of the documentation internal to the code, or what is the quality of design.

I do believe that there is a very strong benefit in open source. I believe that all that information is freely available. You could look at the software, and I would strongly encourage [that for] customers making significant adoption decisions.
They’re probably never going to modify the code, but they should send some of their best guys to spend a day looking at the code. See what they think of it. See whether or not you think this code is maintainable, whether or not the language idioms reflect best practice. That’s a very interesting benefit.

Scott: One of the other things you talked about was support. That comes up a lot with open source, because the way to get support isn’t always as obvious as with closed source. If you buy something from Adobe or you buy from Microsoft or you buy from Symantec, it’s obvious where you go to get support.

In the open source world, it seems there are two models. One is, “OK, you pay MySQL, you buy their enterprise software and you get support.” It’s the same thing with RedHat and the same thing with Spring, I’m guessing – can you purchase a support agreement from Interface21?

The other option seems to be that you go to someone like Open Logic, who supports everything, and you just pay them for support across a lot of products. Now, you’re probably a little biased in your answer, but what do you think about the two different ways of getting support on a product?

Rod: There is nothing wrong with an aggregation model if you’re looking at, effectively, as a broker who is providing the ability to reach the bloke who wrote the software.

Where the aggregation model, I believe, is flawed, is in cases where essentially it’s not providing the ability to reach the people who wrote the software, and also where it is not providing the ability to pay for improvements to the software. So, what am I buying? So, let’s suppose we’ve agreed that we need to give something back. If we’re adopting free software for potentially the next 10 or 15 years, we need to give something back.

We care not only about how we get bugs fixed or how we get our production systems working, we also care that this software is still going to be mainstream in five or ten years or at least hasance of being mainstream because if it isn’t, we’re saddled with an expensive legacy. We also care, clearly, about the software evolving to accommodate new features that are coming forward in the industry over time. So that means: First, we need the ability to make critical fixes if necessary, to ensure that there’s no risk that our production systems are affected.

Second, we need to be concerned about where that investment is going to come from in terms of sustaining the product and moving it forward. So, we’re not just paying for an insurance policy today, we’re paying for the insurance company to be there tomorrow, and for the premium we’re paying to the insurance company tomorrow to be meaningful.

This is where some of the aggregation models, particularly what Open Logic calls it’s “Expert Community,” totally fall down. Because with the Open Logic model, not a cent of what you’re paying is going into writing software. You are not paying a cent towards writing software. That can never work in the long term. What you’re doing is paying, presumably- I haven’t looked at their price point, but I’m assuming their price point must be pretty low – a low price point for something that is not comparable with true enterprise support.

There are two problems with the model. One problem with the model is that it does not do anything for the health of the goose that lays the golden eggs. It says, “Well, I’m not going to feed the goose. I don’t care about feeding the goose; I’m just going to scrape up the golden eggs.” Obviously, the goose isn’t going to be there tomorrow, if that becomes the dominant approach.

The second problem is you just cannot get good quality support out there. Let’s suppose that I’ve got a piece of software that may have a lethal, critical problem in production. Now even though I think, for example, that the Spring Framework is a fantastic piece of software, the best pieces of software in the world have bugs in them. It’s impossible to write perfect software. It doesn’t matter whether it’s open source or closed source.

So, you’ve got a critical problem in production, and it’s really hard to resolve. You will see this happening with both commercial companies and closed source companies and open source companies. They all need to assign resources to work on a resolution of that problem until it’s fixed. When the developer in San Francisco clocks off work, he hands it over to the guy in Australia, who continues working on it until, probably, the guy in Romania starts work at the end of his day. And you need to be able to pay for that, and you cannot support that quality of support with essentially paying pocket money to volunteers.

Obviously, I have a vested interest. My company represents a different model. I hate it when people sell open source short. I hate the view that, “I only use this open source thing because it is going to be cheaper”. I feel so strongly about that because I used to be on the other side. I used to be the person making purchasing recommendations. In the financial industry companies where I worked, I would look at TCO. It made no sense to me to say, “Hey, I can save $100,000 here on a support contract.”

What I needed to look at was where, for example, one of the systems that I was involved in designing handles $5 trillion annually. If anything goes wrong with that system and we can’t fix it for a day, the cost to that company could be in the tens of millions of dollars. In fact, if it was down for two days in that example, the consequences to the UK economy could be quite catastrophic. So, I’m not going to be making the decision on, “Well, you know we can choose open source here and we can save $100,000 on a support contract.” I am more likely to be thinking, “OK, well if open source software seems to fit our requirements really, really well, what do I do if I have a mission critical outage? Who do I pick up the phone to call? Who can organize a team to work on this until it’s fixed?”

Scott: One follow-up: It seems that we are talking about the extremely large enterprise accounts that are handling enormous amounts of financial transactions or things like that. They have enormous vested interest in the quality and the future of the software that they are running. It seems like smaller organizations, whether they are choosing closed source or open source, kind of get to draft off of that.

Talk about the decision for the smaller organizations. For a 50-person company, how is the decision different than it is for an enormous financial exchange?

Rod: That’s an interesting point. For the smaller companies it depends on what you’re doing. For smaller companies who are not doing something that is as mission critical, it may be that the software effectively is free to them, and they don’t have the same requirements around mission-critical support.

Yet frankly, from my point of view as a chief executive of an open source company, if people who don’t need that kind of mission critical support don’t buy it because, for example, their company simply can’t afford it, good luck to them. I would love to see the 50-person company grow into a 500-person company with a website that has a huge amount of traffic for which they need to buy a support contract.

So, I think that open source expands the market and it brings the software to a new audience of people who don’t necessarily have the need or capacity to pay. Essentially, good luck to them! No one should give open source money out of the goodness of their hearts. They need to look at what they need as an organization and who can satisfy that.
So, for example, for every job where I have personally been involved, I would have been getting support contracts for the open source I was using because of the nature of those businesses.

However, let’s suppose I was working for a start-up, a dotcom that is trying to build up some kind of community site and is trying to get traffic. It doesn’t really matter if it goes down some of the time. The software that my company is developing is rapidly evolving. At that point I wouldn’t have a support contract. Let’s think about Twitter. Twitter is a great example here. Twitter clearly went through that. Then it became successful. Now they are encountering technology limitations, which are causing frustrating outage. Now you are getting into a different set of business needs. Now they are reaching the point where it is going to harm the growth of their business unless they improve the uptime of their infrastructure.

Scott: So, in other words, your goal as a start-up is to grow to the point where you need a support contract.

Rod: I have never put it in those words before but I think it’s an interesting way of thinking about it. In a way, you have a greater ability to pay the price that is appropriate to your business needs.

Scott: Right. At some point if you are a start-up and you’re not successful, then the open source that you are using is probably not as valuable to your business as if you are a start-up and you become very successful and you grow. You become an ongoing concern. Well, now your infrastructure has value because your business has value and you need to treat it as such.

Rod: Yeah. I mean what I would suggest is that in the early days people should look at purchasing things like consulting and training around open source.

Scott: Right.

Rod: I think that makes a whole lot of sense because it is simply going to reduce their total costs by making their developers more efficient.

Scott: Right.

Rod: So, the needs profile of the end-user is significantly different.

Scott: Well, I really appreciate you taking the time to chat.

Rod: No, I think that was an interesting discussion. Thank you for following up.

Scott: Thanks very much.

Rod: Thank you, Scott.

Bookmark this: