Interviewees:Shawn Burke.

In this interview with Shawn Burke of Microsoft we asked him about:

• Shawn’s motivation for helping to make the framework source code available.
• Whether there are any limits on who gets access to the source code.
• Whether there was any internal resistance initially to providing the framework source.
• Why Microsoft doesn’t just provide a zip for the source.
• Whether Microsoft’s individual developers write code differently knowing that their actual source might be available for the world to see someday.
• Whether or not Microsoft is planning on making available other development artifacts such as test scripts, threat models, etc.
• Whether a closed source company or an open source company is better positioned to support external developers who need access to the source for a given project.

Scott: All right. So if you take a second and just briefly introduce yourself, that is probably a good place to start. Then we can drill into questions specifically around the source for the.NET framework 3.5.

Shawn: OK. I am Shawn Burke; I am a director here at the.NET development platform at Microsoft. I run a group that we internally call the agility team. We run small high value projects to try to help deliver value to customers in agile ways. Not necessarily using agile technology, but just being agile about how we deliver.

Scott: :So in other words, it sounds like you are looking for sort of low hanging fruit where you can do things that are a little bit outside of the norm but would not necessarily be an enormous effort, but would deliver a lot of value to the customers quickly. Am I characterizing it correctly?

Shawn: That is right. Our team works a little bit outside the bounds of the typical projects and we are not constrained by the same rules they are. We are able to do things that are more customer focused and a little less process focused way.

Scott: Just as a point of clarification, people have said Microsoft is “open sourcing the.NET framework.”That is not what your Microsofts intention is and not really what you are doing. You are making your source code available for reference. You are not really open sourcing it the way that, you know…

Shawn: Nope, some people in the community used characterization, that is definitely not what were doing.Were making the source available as a tool that helps people be more successful with the platform in its current state.

With this effort, my goal has been from day one, a simple scenario, which is you are debugging against an API, you are getting an exception from the API, and you cannot figure out why you are getting it. It is often very difficult to figure that out just from documentation. However, your ability to step into the API often makes it really clear what you are doing wrong.

Scott: Right. In the example that Scott Guthrie put up on his site, this is something I’ve run into,? At the point where you DataBind something, all kinds of stuff happens like down in the plumbing.

Shawn: Yes.

Scott: And the exception that you get, you know, you do not really have a lot of visibility and sort of what you did wrong in your code that caused the DataBinding exception. And so that’s like…

Shawn: Yes.

Scott: That’s like the key scenario that you went after.

Shawn: Yes. That is like the thing that you are unableto do currently you know, because I work here I have the benefit of being able to do that a lot. Quite frequently, the amount of time it saved me , “Oh, Ill just grab the symbols” and you do and start to debug with them and immediately you’re like “Oh! Because I forgot to do this or that.” It becomes so clear that I really wanted to enable that sort of service for customers because its such a powerful thing

Scott: Yes. And just to back up another thing that you said, too. I never read it. I was not indicating that Microsoft was somehow trying to create the impression that they open sourcing this. I know when I looked at it, it was clear to me that Microsoft was releasing the source for reference purposes, you were not releasing it so that people could…

Shawn: Yes. I get it. I did not mean to sound defensive there!

Scott: [laughs]
I’m sure it comes up a lot.

Shawn: Yes. That was the first thing we saw on a large website that is often not terribly pro Microsoft

Scott: [laughs]

Shawn: Fortunately many people came on there and said, “No, no, no.” They never said that, that is not what they are doing. I think most people get that now.

Scott: So what you are enabling is a sort of step into debugging experience where normally, as soon as you call into the API, you could not continue to drill down and step in. Now you can.

One of the things I was not clear on is who gets this? Is this available to all versions of Visual Studio and all customers or is there a sort a subset? Who is this available to I guess?

Shawn: If you are running on Windows, it is available to anyone who can use it or to look at it.

As far as the Visual Studio integration, you need to have VS Standard or above. You cannot do it with Express versions. You know the express versions of Visual Studio?

Scott: Sure.

Shawn: Express does not get it for some kind of fundamental reasons.The main reason there is it just simply because Express does not expose the UI that you need to set this up or have some of the other facilities internally to take advantage of it

Scott: Sure. Sure.

Scott: Yes. The only reason I ask is that some of the people in the Microsoft ecosystem have made their source available kind of under premium plans with their customers. However, that is not the approach that Microsoft is taking with this. This is not sort of a premium thing, this is just generally available.

Shawn: Yes, that is Code Center Premium (CCP), this works similar to that CCP , but there you need to have an account and a Smart Card actually. And you have to go through much more security than in this system. This is a public reference release of the code, so there is really no credentials required. There is none of that.

Scott: So when you sort of floated this idea internally, it sounds like it was generally well received. People kind of jumped on board and wanted to help make it happen. Just behind the scenes, what were some of the things that it took to enable this? It is more than just dropping a zip of the source out there to make this happen.

Shawn: Yeah. Well, yeah. It is good that you brought that up. This whole thing started quite a few years ago. When I was on the Windows Forms team, which I was for seven years, I always felt like we should be releasing the source for many reasons. Others had done it. I felt like it was good for customers. When I was the Development Manager on Windows Forms in early 2005, I did a blog post about this, saying I wanted to make this happen and talking about some of the challenges to doing so. It was kind of justI was just kind of brainstorming aloud about different ways and what some of the challenges were. The response was enormous.Hundreds of comments.

A funny story, I did that blog post, and then I went down to Cabo San Lucas with my girlfriend a couple of days later. When I turned my phone on halfway through the trip, just to see if it would work in Cabo, I had a flood of text messages that said, “Please call work.”

Sean Campbell: [laughter] All right. So you learned a lesson of do not blog anything but “I’m going to Cabo” on the blog before you go to Cabo, basically.

Shawn: Yeah. I think if you would really pull it down to basics, that is it. Well, actually, I had always done tons of blog posts that no one’s…

Scott: Meanwhile, you created this firestorm of all these people running around going, “What have you done?”

Shawn: Yeah, exactly. I like to joke that it created, basically, an international incident because it got picked up by eWEEK, Mary Jo Foley, CNET, and all the rest. Quite a few people picked it up. It generated a lot of interest. That was a good impetus for me that there was a lot of excitement in the community about this.

If you fast forward a couple of years, I did some more ground work on this thing in 2005 and 2006, got busy with new role I’m in now, and dropped it for six or eight months, and then picked it back upI think it was late last year, late 2006and started looking at how we would get this done.And Ive been shepherding it along since then.Getting this out there really is the realization of a dream that I have been chipping away at for quite some time.I really hope Developers find it as useful as I think they will!

You guys mentioned pushing it out as a zip. One of the real problems was that I did not want to have a situation where every team that did this put a zip file or MSI out every four months, so the customer had to go download, install, and set up his little studio to use. What if you have a hotfix or different version? You know what I mean. I did not want to have a situation that was going to be cumbersome for us to ship and for customers to consume.That would kind of defeat the purpose.

What really got me rolling on this was I started talking with the guy who developed this technology we use internally. Internally, you can get the source and demos for almost every product we build here at Microsoft off this thing called “Source Server.” Source Server has versioning built in, making sure things do not collide so you can have side by side versioning, and it all works very smoothly. The server side piece for source server is just a web server. There is no magic there so its very simply to deploy. There really was not any barrier to making to using that technology externally.

So what we able to develop was a system where teams could publish source very, very efficiently. After a first source publish, they can continue to do source publishing at near zero cost to them, which is great. On the customer side, you have a scenario where a customer has a few simple set up steps. Then they kind of auto magically get source and symbols for what we have chosen to make available at that time for free. No searching around on MSDN or having to deal with where to put the stuff on the machine.

Scott: It sounds like, too, one of things that you are saying this accomplishes is it sort of knows exactly what the client is running and building against. If they have installed the service pack or have not installed the service pack, they get the right source. They get source that matches up with their environment.

Shawn: Yes, if it is on the servers they can get it. One of the decisions I made in getting this done was that, the cheater thing I did was I said, “You know what, I ‘m not going to define policy on this. I’m going to get a system developed where individual product groups and teams can make decisions about how often they push source.” Meaning, does it make sense to push source for every new release we do? Maybe, I dont know. Does it make sense to push it for every service pack? Maybe, I do not know. I think those might be different for different teams. I think we are going to do learning in working with the community about what really makes sense there. However, the system is set up to handle it, in any case.

Scott: Like you said, generally people were receptive to this. Were there internal objections that had to be overcome? Where there people who had concerns that, “Hey, this is crazy. Why would we do this?”

Shawn: Well, if you go back into 2005, I think I had many senior people that I knew, vice president level people that were generally supportive of this, but very cautiously supportive. For me a little bit, one of the reasons it’s been a multi year effort is I had to push carefully and let some perspective change as the industry has changed and the environment changed over the last couple of years.

It has gotten significantly easier. Something happened, I think, in the greater consciousness of the industry over the last 12 months or so. A lot ofI do not want to say a lotresistance is too strong of a word. A lot of the real cautiousness and kind of by-default-no talking-into-yes stuff dissipated and switched to actually a lot of enthusiasm. We need to be careful about protecting the IP that we spent billions of dollars developing. However, I think people really see the value of this now. It has been great for us and for our customers for a subset of products. Now I have seen teams are signing up to push their code out really quite enthusiastically.

Scott: With the.NET framework, it is in a little bit of a unique position because, on one hand, Microsoft has invested a lot in it, and generated a lot of IP. However, on the other hand, it was never really a secret what was in the framework anyways. You could use reflector and other tools to disassemble what was in the framework and sort of figure out how it was doing what it was doing. However, it did not enable this “step into it while you are debugging” experience. It did not make it super easy to learn from the framework. This might have been a little unique in that the IP was not all that protected to begin with.

Shawn: Well, yeah. Yeah, it is funny. The case that I built for this, with managed code in particular, definitely had that aspect to it, that our level of secrecy around how this stuff works, as well. One of the problems with reflector, using reflector, the EULA does not really allow for it.

Scott: Ah, interesting.

Shawn: The EULA does say that you are not supposed to reverse engineer. That said, I felt like it was a good idea to give customers a way to do this within the license, which this is.

Scott: Hah! Yes, I guess I never really realized that that might be in the EULA. I think, especially among the more advanced developers and in the blogs, and in all that kind of stuff, it is such a common practice.

So, now that you have done this, you have taken this first step and you’re going to make source code available for certain subsystems of the framework, and that’s on a feature by feature basis, right?

So, Windows Forms will decide when they push their source code in, ASP.NET will decide when they push theirs in, am I understanding that right?

Shawn: Well, we have pretty much what you would consider, if you go back to.NET 2.0 and think about what made up the Framework at that point. We’ve got pretty much all of the big pieces there. We’ve got BCL, Windows Forms, ASP.NET, Data and XML.

So those guys are already taken care of. Therefore, we’re going out with essentially what is the original.NET framework. The other stuff will fall in. It is really a matter of me getting with a given team and through the process. The code preparation process is actually fairly labor intensive. It takes time to schedule that and manage it.

I had teams that were able to do that. A lot of teams have been obviously really busy getting VS 2008 and.NET framework 3.5 finished up. We will continue to push out more code as time goes on. We wanted to get the big ones first.

We also managed to get WPF in there as well, so its not just the older stuff.Lots more is in the pipe to come online as time goes on.

The choice was either we wait a couple of months to do this, and make everybody wait, or we get the big ones out there fast and then we push in the other stuff as we have time, and as it comes online.

Scott: So one of the things that is kind of interesting is, you’ve got these individual developers in Microsoft and they’re writing code. They have worked in Microsoft for a long time and they have a feeling that, you know, what happens in Microsoft stays in Microsoft, I guess, for lack of a better term.

The whole world is not going to see the line of code that they are typing right now. Now, with the source being released, how do you think that is affecting, if at all, how individual developers are doing their job? How product teams are looking at how they build features going forward, knowing that the source for it is going to be out there.

Shawn: Even product teams that are not the framework team, they are thinking, that’s great, they did that over there, so does this mean that someday my source is actually going to be out there? Yes. Yes. They will be thinking that. And they should be thinking that.

In fact, one of my to do list items is to do some broader communications across our Division about that exact point Software developers are pretty much the same around the world. Software developers, in general, always try to do the right thing. They are also always, in general, under a certain amount of time pressure. Anything that is going to encourage them to take a little extra time to do something right, in the kind of capital ‘R’ right, way, is a good thing.

Whether that’s testing, whether that’s static analysis like FxCop, or whether that’s security reviews or code reviews, or whether that’s the code is going to be visible to their peers or the public, I think those are all good incentives for software developers to think extra hard about how they do things.

So, you know, honestly, we went through the code, we’ve done these code reviews, and these code scrubs and we found a lot less stuff than we thought we would.Actually, we found almost nothing of any real concern, which was nice.

Sean: Are there other things that Microsoft is thinking about exposing around the development process?

There are elements like threat modeling, specs, test scripts, all kinds of things that Microsoft might be thinking about exposing. Are there any thoughts or plans along those ways about how far this maybe goes, eventually?

Shawn: That is a really good question. No is the answer. [laughter]

Sean: You’ve accomplished enough by ruining one Cabo vacation. Right? You can put that post up later, you know, on your next vacation.

Shawn: I think that stuff…that is a good point. If you start doing, a curve on the cost for us to do that versus what percentage of customers that level of access is going to be valuable for. For the threat models, and even for the specs, I think that is a harder case to make. I think those things are also harder to vet, or release, than the source is. They are also less useful to a broad set of customers.

Sean: Well, one of the things that crossed my mind was test scripts, and test routines that Microsoft creates because that would be helpful to others just from an educational standpoint. That does not really expose Microsoft to too much more overhead. I am not trying to pitch the idea here, But go ahead, Scott, what were you going to ask?

Scott: No, same thing. Just, Microsoft has given an inch so watch us try to take a mile.

Shawn: Oh, yeah, we know it. Absolutely.

Scott: A lot of times that’s what people want. They say, OK, there is this API. I want an example for how would use it. I would go look in the help. Well, that example is not what I really want. Microsoft I would assume writes really comprehensive test suites.

Shawn: I do not know if the test code is what you want, either. We call it auto PMEs. The PME stands for Property Method and Event. It is super granular. A lot of that code does not relate to any real scenario. It is just wide unit testing code. There are other types of tests as well. A lot of the stuff is that. The challenge of the testing stuff.

I actually think that the testing stuff is a good point, and an interesting thing and I have been thinking about, not really in terms of us releasing test stuff for previous products. I will get into one in a second. The question is going forward, is there a way for customers to leverage the kind of stuff we do in tests more broadly?

However, the reason, going backward, most of our test stuff is written using internal test harnesses and internal tools. There is this huge slippery slope there. You need this and you need that, and it would be really a lot of work.

Scott: Right, right, so they would have this test but they would not be able to hit F5 on it and actually watch it run anyways.

Shawn: Yeah, and the tests probably call into a bunch of other libraries that are internal that we would not ship externally anyway. The tests themselves might not make any sense. Most of the tests are stored in this big system called Mad Dog, which is kind of a giant database of test cases and test code, and I do not even know how you go about extracting code from that thing.

[laughter]

Sean: So Shawn, what do you think aboutthis is just something I was noodling on, trying to think of the right way to box it in given the time in terms of a questionso, Microsoft’s a closed source company, right. By definition if not necessarily for the sake of our interview. What do you think about closed source companies perhaps at times being better able to support the exposure of the source than maybe a company or a project that simply says ‘Well, just go check it out from subversion, from xyz open source project?’

Shawn: Right.

Sean: Because I look at the Visual Studio integration, and I look at robably how this will eventually weave into PSS, interacting with customers, and saying well, here’s an odd dynamic: is a closed source company perhaps better able to actually make the exposure of the source code, when they do it, a more positive experience that actually lets people leverage it more easily? I mean, absent the ability to fork, and some of the more foundational principles of Open Source, how an outside developer would use it day to day development and how a closed source company in this case Microsoft could support that.

Shawn: That is a great question. I think that the answer isone of the things that I’ve been trying to do, and that we’ve been trying to do, is look at the changes in the industry that open source has driven and really think hard about what the real value adds are there. Is the value add really that you can recompile your kernel, or is the value add that you can debug it and understand why it’s doing something?

And there are obviously cases where the answer is ‘both’ or ‘one or the other’. However, for I think the vast majority of folks, us being able to deliver the source in a way that’s easy for folks to consume and step through as a way to make them more successful on our platform is a really big win. I think that it fits well into our overall strategy of kind of keeping developers integrated in our overall processes, and it fits well into this another milestone of transparency.

We have kind of been trying to figure out ways to open up the envelope of transparency with Developer Division particularly and I think this fits into that portfolio really wellas well as the portfolio that the Shared Source Initiative folks are building. You have CodePlex on one end which is full open source, then you have kind of our proprietary source system on the other end, and there’s a nice continuum in between. I think that we do bring some assets to bear which are difficult for companies that are purely open source or service model driven.

Sean: OK, cool. Well, that is all I’ve got, and also to be sensitive to timeScott, do you have a closing question or two?

Scott: I think that’s a good note to leave it on. The thing that we always end with is: are there things that we didn’t cover that you think are really important for people to know? So, we’ll just kind of hand you the microphone at the end if there’s something you want to get out that we didn’t specifically ask about.

Shawn:Nope, I think we covered it all.Thanks guys!

Bookmark this:

Interviewees:Blaine Wastell and Glenn Block.

In this interview with Blaine Wastell and Glenn Block of the Patterns and Practices Group at Microsoft we asked them about:

• About the Patterns and Practices Group
• What makes them different from a typical product development group
• The software development methodology they use
• Some of the things that Patterns and Practices group has produced for developers.
• The role of CodePlex for their group.

Blaine:   I’m Blaine Wastell. I’m a program manager in patterns & practices, and I’ve been in this role for a little over four years now. Most recently my focus has been in what we call the "client UX program". It’s about providing guidance to customers on developing both smart client and web client applications.

For 12 years before Microsoft I was out in the consulting world helping business customers develop enterprise line of business applications, mainly Web applications.

Glenn Block:  I’m Glenn Block. I’m a technical product planner in patterns & practices. I am a newcomer to the client team. I’ve been in p & p for about 6 months, and with Microsoft for about 2 years. Prior to p & p, I worked in Microsoft Learning and was responsible for building online developer training.

For the 10 years prior to Microsoft I was a software engineer and architect. I worked in various startups as well as a few enterprises. I’ve had a touch of consulting, but it’s mostly been really about building both packaged apps and internal apps ‑‑ both on Web and on Windows. A significant part of my background has been in developing software frameworks for those kinds of applications.

And as the technical product planner on the client team, I’m responsible for the overall strategy, understanding what customers want, and setting expectations on what kind of guidance we’re going to deliver.

Scott:  Talk a little about patterns & practices.

Blaine:  We provide "building blocks" that customers use for line of business application. Microsoft has a number of development tools and technologies which include the .NET framework, Visual Studio, SQL Server, the core operating system, etc. We help customers by providing guidance, in multiple forms, on using those technologies to build line of business applications.

Glenn:  The practices part of our name is focused on the latter, where we give guidance, best practices, and design patterns, to educate developers on known solutions so they’re not reinventing the wheel.

Scott:   I’ve used like the Enterprise Library that patterns & practices produced, and it’s really more of a framework than building blocks. Are things like the Enterprise Library are designed to be best‑practice reference architectures, or are they drop-in frameworks designed to give a productivity boost?

Blaine:  Enterprise Library, and the application blocks, are part of what we deliver. We also have written guidance for areas like security and performance.

We also have another type of deliverable called "software factories" that tie all the pieces together. They have the framework, but they also include the common patterns for a specific area. They include automation; they include what we call a "reference implementation" ‑‑ essentially a sample of how to implement proven practices using the frameworks.

Glenn:  When you talk about frameworks, at the end of the day, to us it’s all guidance. In the case of these frameworks, they are incarnations of the patterns we recommed. We think of them as an 80% solution meaning they may not provide all the answers. We write the code with the intention that people who are consuming these deliverable will rip that code up, and will look at how we’re doing stuff. It’s not simply about giving people a reusable toolkit, although there is an aspect of that.

The idea is, "Hey, look at what we’ve done. Rip out the pieces that you think are applicable for your scenario. If you’re writing something new, you can use this as a reference point and know that it incorporates well‑known patterns."

Scott:  What makes your team different from a Microsoft product team. How are you different from the teams building the .NET Framework, or SQL Server, for example.

Blaine:  We focus on what we call ‘customer connected engineering’. We start with an area that we see is challenging for customers, and then we will put together an advisory board that includes customers. As we design a solution, they’ll validate that our scope and approach is really hitting the top set of challenges that they have.

Glenn:  The big differentiator is that product groups build the platform, while the guidance we create uses the platform.

Product teams focus on taking the platform to the next level. We do guidance, as Blaine described, guidance on using the platform. We show you how to put the legos together in meaningful ways to solve particular problems that you’re facing.

In some cases, through the act of putting those things together, we may pull out some new reusable guidance that we call ‘application blocks’. But the block is not the end goal. The block is just one incarnation of the guidance. When I say ‘block’ , I’m really referring to a reusable library.

Scott:  Could you list some of the things that you’ve produced?

Blaine:  You mentioned the Enterprise Library, which has a number of blocks within it. There’s also the composite UI application block, which is a library that helps you create smart clients. There is also a set of libraries to help you create web clients. One is called the composite web application block. And we also have another one called the pageflow application block which addresses challenges around Page navigation. Blaine:  We’ve done a number of things in the web services area.

Glenn:  We have a Web service software factory; which is guidance on building web services. And we also have new guidance that we’re developing called ESB guidance which addresses utilizing BizTalk as an Enterprise Service Bus.

Scott:  Talk a little bit about the software development methodologies that you use.

Blaine:  Sure. We are an agile organization. We’re probably closer to XP. We start with the backlog that we prioritize. Glenn is essentially the proxy for the customer; the one that will prioritize the backlog for us. From that we do the standard iteration planning, and meetings. We do it once a week, and we have stand ups daily where we go through the status of the standard set of questions: What did we do yesterday, what do I plan on doing today, what blocking issues do I have.

At the end of an iteration, which is every week, we publish our results out on our CodePlex community site. Then customers can look at the CodePlex site and get an understanding of where we’re going. A lot of times they’ll say, "Hey, why are you doing this, because I have these other two sets of challenges you’re not covering," which is good. We’ll also have people say, "Hmmm. There’s a bug in line 35." It’s pretty amazing how closely customers sometimes watch us.

Glenn:  And in some cases even supply fixes.

Blaine:  Right. We’ll also do testing, development, and continuous integration. We use in test driven development, so we write the unit tests first, and then the code.

We’ll generally do pair programming. We’re also distributed team with team members located in Redmond, Buenos Aires and India. This sometimes makes it hard to always do pair programming. If we can’t do pair programming we will have somebody else do a code review before checking the code in. We do automated exception tests where they make sense. We do quite a bit of static analysis on the code that gets checked in.

Glenn:  How about code coverage?

Blaine:  We do code coverage. I have a blog entry that I just put out recently that talk about the quality checkpoints that we go through. On the quality assurance side, we’ll do things like proof and security testing, and globalization testing.

Scott:  You mentioned there’s also a community side of what you do. Talk a little about that.

Glenn:  There’s several different ways that we actually interact with the community and our customers. The simplest way is our blogs. A good portion of the team is heavily into blogging, and we try to give as much visibility as possible into what we’re up to.

Blaine mentioned that all of our stuff is on CodePlex. A big differentiator between the work that we do, and the product groups at Microsoft, is that you can get all the source code.

Also on CodePlex we have work item voting. When people come to us with feedback we say, "Great idea. Throw that on as a work item so the rest of the community can vote." And I can tell those work items absolutely determine what we do.

For example, AJAX support in the Web Client was one of the highest voted work items. As a result we’ve done about four months of work on the UI responsiveness and we have a few more to go. That’s a large investment and it’s based off the community feedback.

In some cases, there’s a work item that’s very broad, and we’ll dig in. For AJAX there’s a server side and client side aspect. When we asked which people were more interested in us investing in, people said, "Hey, we’re more interested in the server than in the client."

Another source is our direct interaction with customers. One of the things we do in p & p is bring a lot of customers in to spend time with the team. That gives us an amazing opportunity to directly interact with them, find out what they’re building, find out how our guidance in mapping against their needs, and find out what the gaps are.

We just had a customer that was here for an entire week. We do that with a number of clients and aggregate the information and learning which we inject back into our efforts.

We also do some on‑site, where we go and visit customers. Since I’ve joined, I’ve visited two customers ‑‑ one in the U.S. and one in the U.K. ‑‑ spending a week with their teams.

Further, we have general blog that generates feedback that we get. We have email feedback that we get. That covers the external customers. But p &amp p also has internal customers. Who are those? These are the product groups.

Lately we’ve been we’ve been working very heavily with Scott Guthrie’s org. We’re doing this is because we want to understand what kinds of challenges the platform is going to be addressing and how it will be addressing itWe want sure that whatever guidance we’re giving is in line with this direction and not opposed to it.

And we also key in to their demand for deep content. They say, "Hey, there’s nobody who’s really telling people how to put these pieces together." We’re also getting very involved with OBA [Office Business Applications]. We’ve had the OBA team coming to us saying, "Look, we need some solid guidance that will give customers architectural guidance on how to develop an OBA application."

Sometimes all these inputs are in conflict with each other and we have to look at the external feedback, the internal feedback, the industry trends, open source, and even our competitors, and decide what we’re going to deliver.

Finally we have our Customer Advisory Boards. This is a key aspect of our customer‑connected engineering strategy. For every one of our deliverables, we have an advisory board of individuals that are in the industry, that are or will be using this guidance. For the Web Client Software Factory, every two weeks for about the past four months, we have an hour meeting where we present to the advisory board what we’re doing, and we get their feedback.

The advisory board is a group that’s able to look at what we’re doing and raise flags and tell us, "Hey, you’re going in the wrong way." Or they may be saying, "You guys are missing this big opportunity over here that really affects all of us." And when we build this advisory board, we select a group of experts across the industry in both small and large companies, some located in the U.S., some located in Europe, some located in Australia. We try to get a really good mix.

They have different needs. They give us a proper perspective on the overall landscape so that we can make those intelligent decisions. I think that covers it. As you can imagine, it’s not an easy task to drill through all of that input and decide which way to go.

Scott:  Right now, the patterns & practices code is released under a custom license. It’s not under the Microsoft Reference license or the Microsoft Public License — which used to be the Microsoft Permissive License.

Blaine:  Everything is migrating to the MSPL.

Glenn:  Yeah. It’s essentially been MSPL without the name. It is going to be MSPL.

Scott:  OK, so the MS Public License.

Scott:  The thing that’s really interesting that now that’s an OSI‑approved license. In the past, Microsoft would say, "We put the source out there. It’s open source," but to a lot of people in the industry…

Blaine:  It wasn’t really.

Scott:  Yeah. To them, unless it’s an OSI license, they don’t know if it’s really open source the way the OSI defines it. So now it is. That leads to the question of, are you now starting to do some of the things that a traditional open source project would do? You’ve got builds. You’ve got bug tracking and issue tracking that people can participate in.

Do you ever envision a day where you’d be taking community code submissions?

Glenn:  We’ve taken steps in that direction with what we call our Contrib community.

Before I worked at Microsoft, I used to work with a bunch of different open source products. I worked with NAnt and NUnit. NAnt has what’s called a NAnt Contrib Project. The NAnt Contrib project is where developers who have the source, are extending NAnt. They’re building plug‑ins, etc., because a lot of these products have a plug‑in model. They want to have a way of driving that back in so that the community can benefit from those enhancements or extensions.

We decided about six months ago to follow the same model and we created a set of Contrib communities. Actually, we didn’t create them, and I think it’s important that we didn’t create them. We worked with the community. For example, with Smart Client we engaged with people like Kent Boogaart, who created WPF CAB, which was his own extension to CAB that would allow you to build WPF applications with CAB.

We also worked with people like Bil Simser, who’s a known MVP who developed Smart Client apps. We worked with a whole group of people, Chris Holmes is another. We brought them together and said, "Hey, we want to make you guys as successful as possible and enable you to take our deliverables, extend them how you want, without us telling you which way to go, similar to the way you can with other open-source projects. We’ll provide a place for you to do that, and we will also work with you going forward. We’ll also hear your concerns and consider how we can fold that into the product."

We do also work with external resources as part of our core guidance deliverables. We have our advisory boards which I mentioned which participate in the design and give us feedback. We have our CodePlex Workitems where the community submits ideas that influence the design. And we even have members of the community (partners, SMEs, etc) that write the codebase itself.

Scott:  Aren’t there models where other people have dealt with this? You take a look at something like the Linux kernel, right? Nobody really owns it. It’s an open source project, it’s covered under an open source license. IBM is a contributor to it. And if stuff gets found in there that infringes on somebody else’s intellectual property, a lot of times the community, or IBM, replaces that with code that doesn’t.

I realize it’s a gradual process, and Microsoft is moving towards more community involvement, but it seems like down the road there may be ways to structure this to where Microsoft is a chief contributor to these projects, but every committer may not necessarily be a Microsoft employee.

Glenn :  In the Linux kernel, yes you do have community participants, but it is not a free-for-all where anyone just comes along and checks in code. It’s a regulated process with a core set of contributors. We operate in a similar fashion. We do have externals that contribute, but there is a process around those contributions.

Scott:  Another thing that’s interesting, when you’re under an OSI license somebody could fork your code.

Glenn:  Yep. And I just want to make clear that we don’t own the Contrib project. We’re not claiming any responsibility for the kind of code that goes into that. We might suggest stuff, like, "What do you guys think about doing this or doing that," but it’s completely up to the contributors what they want to do.

It is the model that you described. We may pour some stuff into Contrib, and we’ve talked about doing that, but essentially the people that own it is the community.

Scott:  Right, right. And those different Contrib projects are essentially forks of your code, and because…

Glenn:  That’s exactly what they are. In some cases, complete rewrites. For example, if you look at the EntLib Contrib, there’s actually an EntLib refactored project where somebody who works at Microsoft ‑‑ Scott Densmore, who helped write a lot of our deliverables ‑‑ is actually ripping apart EntLib and providing a new version.

Similar kind of things are happening in the CAB space. We just recently shipped another version of Smart Client Contrib that actually contains updates to our Updater Block, to get it to work on Vista. This is completely being driven by the community and it’s taking things to places where we don’t have the resources to take them.

Scott:  That’s what I was wondering. Maybe down the road, the only thing that exists is the Contrib space, and people in Microsoft are contributors just like anybody else.

Glenn:  That’s certainly a possibility that we would be open to, but it’s not really about us and what we want, it’s about the customers. Many customers that have serious concerns about using community contributions that might contain code that violate IP. In the case of patterns & practices deliverables because it’s from Microsoft, they have less concern. Other customers however are willing to take that to take that leap of faith. These are the same customers that also use open-source deliverables like NHibernate, Log4Net, Castle Windsor etc.

Scott:  To me it’s really fascinating how the different models are shaking out. We also interview CIOs for things outside of "How Software is Built" and they’re very fond of saying, " I want one throat to choke. I want to know that I’ve got support. I want to know that I’ve got a company that’s standing behind it. That’s really important to me. It’s not about the cost of the software, it’s not even about free as in freedom. I just need stuff that works, and I need someone to hold accountable."

And then on the other hand, on the other side of it, you’ve got open source, right? Where you don’t seem to have those same assurances, but on the other hand the software really does hit the mark. There are people who are experts in their domain. There are people who build really interesting things that a "big company" wouldn’t necessarily think of. There are people who do some innovative things, or start with someone else’s idea and take it in a very different direction.

There’s benefits in all of that. So it’s curious to me what the landscape is going to look like. You take a look at a company like RedHat. They’ve got this open source product, but they don’t really have control over the Linux kernel, or Apache, or the things their customer absolutely depend on. Still, they guarantee that they’ll provide support on top of it, enterprise‑level support.

You’re saying that if p & p’s work was full open source, and the only out that existed was Contrib, some of your customers would have trepidation because they would feel like there was nobody they could really point to who would guarantee the quality of the code, who could guarantee fixes, and those sorts of things.

I guess what I’m saying is it’s a very interesting spot that you sit in.

Glenn:  In a way, it can be the best of both worlds, with some caveats of course. What it means is there may be significant functionality that will exist because of the energy of the folks in the open source community, and that functionality will show up in Contrib but not in our core deliverables. There are other things that the Contrib community might do, that we might decide to incorporate into the core in the future.

Scott:  There’s another thing that makes your position interesting. On one hand you might get feedback from the product team saying, "Hey, we’re not going to have this in the next version. It’s going to be a pain point. It’s something that you guys could take a look at."

On the other hand, you might have a lot of people in the community asking for a feature, but because you’re part of Microsoft, you know that the product team is already building that into the next version of the product. You don’t want to invest in what the community is asking for, because it’s already being built, but you can’t tell anyone it’s being built. The product team still has their plans under NDA.

Glenn:  That certainly happens all the time. That’s certainly a challenge. But what I’m happy to say is that more and more we’ve been able to drive back the experiences customers have had with our guidance to the product teams, which has resulted in many cases in product offerings that come out based on that.

Scott:  When I looked at stuff you produced a long time ago, for example, the Data Access Application Block, it seems like it was almost destined to be obsolete. Eventually Table Adapters got built into the .NET Framework, and that provided the same functionality.

Blaine:  Our goal is to actually help migrate customers from what we produce, to the core product. That’s really our ultimate goal. If the core product addresses scenarios we’ve heard from customers, that’s a good thing.

Scott:  Yeah. Like I said, you guys are in a fascinating spot in terms of being an open source project, or open source projects, you’re really pretty unique it terms of where you sit. Can you share anything about the directions in which you’re going? You’re obviously moving towards more open. There’s certain constraints on that. Anything that people should look for over the next few months?

Glenn:  One of the things I think we’re moving on, and Blaine can touch on this, is bringing more of what we’re seeing to product groups. As an example ‑‑ and we’ve been trying to collaborate much more tightly internally ‑‑ one example is this new MVC framework. Want to talk a little bit about some of the work?

Blaine:  At patterns & practices we’ve been supporting, as Glenn talked about, many of the common patterns in the UI space. MVC [Model View Controller] is a common one. MVP [Model View Presenter] is another one. And since we’ve been in the space, we’ve been working with the ASP.NET team to help define the MVC work that’s coming out of there.

A big part of this is really Scott Guthrie and his team’s vision. We are bringing some of the feedback that we’ve heard from customers to the ASP team.

Glenn:  We’ve actually been working on the design with them. It’s really been a partnership. I think those kinds of things are efforts that you’re going to continue to see.

The other big thing that I think you’re going to see, is the fruits of a transition we’ve been going through internally within patterns & practices. We’re re-evaluating carefully at how we deliver guidance and what is the best method. , and we’re also looking at taking the principles of SCRUM that we apply within our teams and bringing it up to a higher management level with p & p. This includes having having a cross p & p product backlog that we can be very visible with customers about. We want to show, "These are the things we’re looking at. This is what’s a high priority. This is what’s not."

On the subject of how we deliver guidance, although many customers have nothing but good to say about our factories, there are other customers, that say they are too complex. They also tend to not be as applicable in in what we call "Brownfield scenarios" which are existing applications. Instead, they tend to be most useful when you are starting from scratch. Another thing we’ve seen is that industry-wide there has been a shift away away from comprehensive “do-it-all” type frameworks to a more pluggable services approach. Finally we’ve had a set of customers who tell us that the biggest value we can provide is to focus more on patterns themselves and how to use them (including samples and reference implementations) rather than framework type libraries that implement those patterns. We’ve heard thse customer concerns and are taking them seriously. We’re looking at how we can break things into smaller blocks. We don’t mean exactly like the earlier application blocks, but instead how to ship our deliverables in an incremental fashion that offers customers more options as to what they use.. For example if I want to use some aspect of the factory without having to take everything, then I can. And, we are also looking at patterns and how we can make them more a first class citizen in our guidance.

Around this, we’ve put a stake in the ground with the current release of the Web Client Software Factory and with our newly announced WPF Composite Client.

By getting these things to be smaller, we can ship more of them more frequently, we can make more intelligent decisions about what we ship and what we don’t ship. We can build guidance that has more applicability in both the brownfield scenarios and well as greenfield — which is new development. We also give customers more choices as to how they incorporate our guidance within their environments.

Ultimately, we can reduce the complexity in evaluating and using these deliverables to make them approachable to a wider audience.

As you can tell, we’ve certainly got a lot of work ahead of us.

Scott: Thanks for talking. You opened up an interesting subject at the end, and maybe we could talk again in the future about it.

Blaine:  You bet.

Bookmark this:

Update: It seems that this isn’t seen as happy news by all. There’s an article on eWeek that’s just too irrational and frothing to pass up, claiming that this is all a ploy by Microsoft to kill Mono. As Microsoft is officially supporting Novell’s efforts in porting Silverlight to Linux (on top of Mono), the evidence would indicate that Microsoft is doing this to support .NET developers, and not as some clever conspiracy to kill off Mono.

Bookmark this:

Interviewee: Jay Pipes

Jay Pipes

In this interview, we speak with Jay Pipes North American Community Relations Manager at MySQL.

We talk about:

• How MySQL Community and Enterprise servers are nothing like RedHat Fedora and RHEL.
• The nuances of MySQL licensing.
• It’s built mostly like traditional software, with 120 developers on staff at MySQL.
• But that’s changing, the worklog system has been opened to the community and they’re starting to take contributions.
• For that to happen, MySQL has to become more modular and support plug-ins.
• MySQL supports so many platforms through a portability layer, which was a lot of work initially, but now it’s pretty well baked.
• Bugs = Gold.
• The tenets of MySQL: performance, reliability, and ease of use.
• It’s easier for proprietary software to make money, it’s easier for open-source to get work done.
• And finally, how to get involved.

Jay Pipes: My name is Jay Pipes. I work for MySQL. I’m the North American Community Relations Manager. Part of our job as community managers is to monitor, encourage, and grow the external MySQL ecosystem as much as possible. [This involves] responding to concerns of the community, pushing those concerns internally, being an advocate for the community within MySQL—being a liaison between free and open-source projects and companies and MySQL. Also, each of us has different responsibilities. I do a bit of development work for the MySQL Forge website. I work a bit with engineering. I do a lot of conferences and speaking engagements on performance tuning, and blogging, and writing, and that kind of thing.

Scott: I get it. You’re greasing the wheels between MySQL and the community at large, and trying to keep the information flowing friction-free in both directions.

Jay: Yes, I want to remove the friction points between contributors to MySQL and users of MySQL, developers and DBAs, and the flow of information from MySQL to make sure it’s accurate—that they understand. Especially with MySQL, we’re in a unique position. We provide this open-source software and we have this enormous, ubiquitous user community, but at the same time we’re selling products. So, because it started out as an open-source project, and the community knows that, a lot of people don’t even realize that MySQL is a company.

A lot of times we have to explain that MySQL as a company has to do certain things to provide revenue that will in turn benefit the open source community. Sometimes that’s the challenge. There’s this push and pull between commercial and the community. And the community team at MySQL, including me—a lot of times we get asked to advocate for the community within the company, but we also have to have an open mind. We are a revenue producing company, and at the same time, we want to be good for the community. That kind of balance is most of what my job is.

Scott: It’s interesting that you bring that up. There’s still this feeling like it still works the way it did six or seven years ago where open-source was people working on stuff in their basement. But if you look at any of the major projects out there, the model has really shifted. If you look at the Linux kernel itself—that’s a lot of engineers at Red Hat, IBM, Intel, and various different companies that are really doing the bulk of coding. If you look at something like RedHat/Enterprise Linux—again, this is a commercial enterprise, right? This is an ongoing business that has come up with a business model around something open source. And so, with you guys—help me understand it a little bit…

Jay: I will try and clarify something. A lot of people say, “Well, when MySQL just recently (in September or October) split between the community server and the enterprise server in MySQL…“ A lot of people equated that to RedHat/Enterprise Linux split. One of the things significantly different between MySQL and RedHat, is that within RedHat does produce its own pieces of software within the RHEL stack and the Fedora community packaging, it doesn’t have total control over its software — their software stack is dependent on upstream committers. Obviously its engineers contribute to the Linux kernel and various other things.

But MySQL has always been 99% completely written by MySQL employees. So, it’s a different model in that RedHat (until more recently where they’ve started to create more of their own software) has been more of a packager than it has been a producer of software. And there is a difference there. It’s what’s made it difficult for people to say, “Well, MySQL has done the Fedora versus RHEL split.

Scott: OK.

Jay: It is different because we are producing almost 100% of the code that we’re selling in the enterprise and then giving away in the community. Does that make sense?

Scott: Totally. And so, I’m sure you can explain this better, but if you’re using MySQL as part of an open-source project, then MySQL can be freely distributed with that project. But if you’re building something commercial on top of MySQL, then my understanding is that’s where you actually have to purchase a license. Or do I have it all wrong?

Jay: Well, it depends. A lot of people are confused by the licensing. Some of the confusion stems from the fact that it is GPL. Some confusion is that we sell commercially-licensed MySQL for those OEMs and ISVs that embed MySQL within their product and then distribute it. So, GPL is all about distribution—reciprocity and distribution.

If you distribute your product with MySQL, and your product is not GPL or a GPL-compatible license and is not an open-source product, then yes, you are required to pay a licensing fee to My SQL. However, a lot of people will say, “I’m distributing my product as a non-open-source piece of software. it can connect to MySQL.” So, if you have a MySQL server running on your network, my application can connect to that server and run against it.

There are different ways of using MySQL, but a lot of where the licensing comes in is when you’re an original equipment manufacturer and an independent service vendor—or whatever ISV is anymore—and you’re embedding and distributing MySQL as an integral part of your application.

That’s where the licensing comes into play. But not for web applications, when you have MySQL installed on the server and you’re a service-oriented application provider. That’s not where the licensing comes in. That’s where we sell the MySQL Enterprise edition, which is the support and services offering.

Scott: Got you. So, just to make sure I understand that: If I’m an ISV and I’m selling my software, and I package MySQL with it, I have two options. I can distribute my software under the GPL license, or I can pay a licensing fee to MySQL and distribute my source under a proprietary license. Is that it?

Jay: That is correct.

Scott: OK. OK, good.

Jay: And in the past, there’s been confusion about the protocol, but that’s not the issue. I think people have blown that up out of proportion. What you just said is exactly what it is. If you are distributing your application packaged with MySQL and you’re not releasing under GPL or a GPL-compatible license, you have to buy a license for each copy of your software that you distribute, because you are then distributing MySQL. Now, the licensing costs have also been blown completely out of proportion by a lot of people. People say, “Oh, it costs $595 per distribution.” That’s not right either..

The pricing depends on whatever the sales team at MySQL and you agree on. It’s just like any other company. But that is the case when you need to buy licensing, when you embed MySQL and you do not want to release your source code as GPL or compatible licensing.

Scott: Cool. And I’m just not involved enough in the community to really even be aware of what have obviously been contentious conversations, probably over the years. So it’s just me trying to understand it, coming…

Jay: No, no. I understand where you’re coming from. I was referring to this kind of myth that MySQL licensing is overly complicated. I work for MySQL, so I’m biased. But I don’t particularly think that is a complex process to understand. I think that the complexity really stems from the fact that there are just a ton of open-source licenses out there that all have these weird idiosyncrasies to them. And I think that complexity lends itself to, “Well, MySQL is open source, so it’s going to be complicated.”

Scott: Right. So, the main thing we’re focusing on is how software is built. We’re looking at things like Apache and the Linux Kernel. From the outside it kind of looks like they’re built over a mailing list. In other words, there are these core mailing lists, people post code to them, it gets reviewed, there’s a maintainer who decides whether it gets checked into the main tree or not.

Jay: That’s absolutely true, with Linux, Apache and Eclipse. Eclipse is more bureaucratic than that; they have hierarchies and procedures and policies and incubation periods and all that stuff—as does Apache.

Scott: With you guys, it looks more like a traditional proprietary shop. I’m guessing that you sit down and have meetings. You discuss what features are going to be slated for the next release. You come up with project plans; you come up with specs.

Jay: If only it were that simple. Yeah, on the outside is does look like a traditional software house, in that we have maybe 120 engineers working on various teams, from people that work on the connectors and the GUI tools to people that work on the server runtimes or the backup and replication folks.

Scott: Which is a lot. When you talk about 120 engineers,that’s a mid-size software company.

Jay: Absolutely. Absolutely. And yes it’s true that we have scrums internally. We have internal roadmaps. And up until, I would say, December or January of this year, what has been more of a cathedral-type model, meaning more of a closed-source model for development at MySQL. It’s now starting to open up significantly.

Recently, we opened up our internal work log system, which is as close as you’re going to get to a list of roadmap tasks that we’re working on. This is for anything from MySQL 5.2 and up to MySQL 8.1 and beyond from all sorts of crazy wish-list ideas to stuff that’s actually going into the code at this point. We’ve opened that up publicly on our MySQL Forge (http://forge.mysql.com/worklog/). People can comment on these tasks, provide suggestions and vote on things that they’d like to see. We’ve also started accepting more contributions from the outside community. So, it’s starting to be more of mix of an open-source project and a commercial company model. We’ll see how it goes. I’m obviously pushing for more of the open-source development model, having more outside committers and contributors that are providing both external tools to the MySQL server, but also fixing bugs and provided patches for small features within the server itself.

Scott: That’s interesting that you’re transitioning from the cathedral to the bazaar, to some degree.

One of the things that I don’t understand about things like the Linux Kernel is how really big sub-systems get built or worked on. At the point where IBM decides, “OK, we just need this in the kernel.” They don’t ultimately kind of get to say, right? If Linus doesn’t like it, it doesn’t make it in. At the same time, corporations are sort of doing the bulk of the development.

So, with MySQL, how do you see that shaking out? Certainly MySQL, the company, is going to control the direction of the product, you’re opening it up to take more community input both in terms of suggestions and ideas and in terms of actual feature code and things like that. But I’m guessing there will still be a pretty significant section of the product that will be spec-ed out. A team of engineers will be put on it to build out a feature. It’ll go through…

Jay: I think it will be a mix of both. And we’re still going through these growing pains of figuring out how this is going to work. The community team is going to be pushing more and more contributions from the community, and MySQL doesn’t dictate those in any way. Someone can hop on there and say, “You know what, I want to implement check constraints, and here’s the patch for it.” What will be the issue is which version of MySQL will that patch make it into. And will it be a module that will be marked as experimental? Will it be something that will be patched into the core kernel?

That’s the process that we’re currently going through. We’re still in these growing pains. We’re still really just now figuring out how to handle these kinds of contributions. So, over the next year or two, I think we’ll start to hammer that out, and understand, “This is going to go into the community server, and this is going to go into the core kernel.” I think, as we make the core server more modular, that issues like that are going to start to disappear a little more, because someone can provide a module, just like mod_ssl for Apache.

Scott: Right.

Jay: It can be a self-contained component that isn’t necessarily going to kill the main, core kernel of MySQL. And so it’s not going to be as big of an issue, because we can package and version up that module separately from the core kernel, and the community person can have it out there. Until we get to that modular core piece of MySQL, it’s going to be a little bit of a difficult road, as we decide how to patch that stuff in. But, on the commercial side, we’re always going to have companies that will provide us with what we call NRE, non-recurring engineering.

Scott: Right.

Jay: Which is basically, someone’s paying us to put a feature into MySQL that is vital, or mission-critical, for their business. So recently, a lot of that type of work has gone into the NDB Cluster tool, which is our high-availability tool. Telecom companies that extensively use MySQL Cluster would like certain things, and they’re paying for those things to get included. And we’ve got those type of projects going on all the time. In the next year or two, we’ll start to see a bigger balance of community driven activity, in engineering, and commercially driven activity.

Scott: Looking at successful open source software, I think you’re exactly right. Modularity seems to be completely essential. I might have trouble getting something into the Apache core, but I wouldn’t have any trouble writing a module and just putting it out there, and if people like it…

Jay: Absolutely, absolutely. And that’s the key to the community driven coding, is that once we get that architecture completed — the plugin interface — where people can write add-ons and extensions to MySQL – that problem of, “OK, which version of the server? Can we put this in there without destabilizing the core runtime?” Those kinds of questions will cease to be an issue. And so will packaging issues, because the community person can put it on their website: “Hey, this is my module for MySQL. Go download and install lit.”

Scott: Right.

Jay: Just like you would with any of the weird Apache modules that are floating out there.

Scott: Right. Apache, Eclipse. All of these things have a modular architecture. To be an open-source project that’s taking community contributions, it seems essential to have that modular architecture.

Jay: I think it is, yeah. It’s going to be a long ways to go. From my understanding, from the engineering team, it’s not something that happens overnight and, certainly, is going to take lower precedence to some of the commercial work that we need to get done on the server. And obviously, our roadmap is years ahead of time. [laughs] The stuff that’s going into MySQL 6.0 and 7.0 is already on the block.

Scott: Right.

Jay: Bringing stuff up like the modularization of the core kernel, we’re looking at two years down the road. But it’s still, in my opinion, vital to start thinking about this now if we’re going to really get to a point where a community is actively contributing to MySQL.

Scott: So, from a software development standpoint, that seems like it would be a particular challenge for MySQL is just that MySQL runs on so many different platforms. You guys have, I don’t know how many kind of distros for a given version. You run on Windows. You run on Mac. You run on a whole bunch of different Linux flavors. How much of the engineering effort do you feel like goes into features themselves, versus how much of the engineering effort goes into making a distribution that runs on such an enormously wide variety of platforms?

Jay: Yeah. I’m not privy to the exact numbers. I can take a guess, though. I would say that the portability layer that allows us to run on these various platforms is fairly stable. Not that it runs perfectly on all the platforms, but that we do run on all the major platforms. And the reason we can do that is an underlying subsystem that takes care of the portability between those systems. That’s been around for a while, so, unless we’re talking about newer things, like Windows 64-bit running on Falcon — which is our new storage engine coming out — I think a lot of that’s already been done. So, most of the work is really in the features, and a lot less in the portability layers.

Scott: OK.

Jay: And I would say that is the case with, say, PHP or Apache or Python, or many of the major open source projects. That core portability layer was a key thing early on, and it’s stabilized pretty dramatically recently, so that’s not really what people are working on; it’s more of the feature-wise stuff.

Scott: Got you. Got you. So, the only time the portability layer really needs significant engineering, like you said, is if you’re porting to a whole different architecture, like 64-bit, or something like that.

Jay: Or when you’re specifically profiling bottlenecks on a specific architecture.

Scott: Right.

Jay: But that’s more of a performance thing and less of, “Will it work on the platform?”

Scott: So, talk to me about some of the other stuff that goes into building a product, things like testing and QA. How does that work? MySQL, I’m assuming that it’s got pre-release beta builds, or daily builds, or things like that, that you can pull down.

Jay: Yeah. In fact, the release schedule of MySQL, on the way it’s built, I don’t think is going to be much different from most other open-source projects. We have an internal build team, which I think there’s four or five people on it, maybe. They are responsible for the overall release management: making sure that the builds compile on all platforms, that the binaries are stable on the major platforms. And also, building up the release notes, making sure the flags and switches that are relevant for each platform are turned on or off depending on what’s needed, and that everything runs through our internal push build system, which, essentially, is an automated system that says, “Will this build on this architecture?” And then, we also have a QA and testing team.

It used to be a single team. Now, we have one man, Omer Bar Nir, who’s the QA architect over the whole thing. But we have QA engineers, now, attached to each of the development teams. And so, they are focused specifically on the QA and testing of, say, backup and replication, or the storage engines, and things like that. So, where it used to be that the QA was across the board, now they’ve split up and focused on specific pieces. And I don’t think that’s very different from any other open source project. The way we release is, we use a tool called Bit Keeper for our source control, and we do nightly or daily snapshots from that, which you can take and build the source code yourself.

And then, once in a while, we’ll package up the source code into tar balls, or zip files, depending on what platform you need. And then, depending on what version of MySQL, whether it’s Enterprise or Community, they’re built into binaries and then distributed. All the distros that I know of don’t use the binaries at all. All the Linux distributions, they actually take the source, from either BitKeeper, or from the source tar balls for a release, and then modify it to suit their needs, mostly by where the configuration files go on install, what’s in there by default, all that kind of stuff. And then they package it up into a. DEB or an RPM, or whatever it is.

Scott: Right.

Jay: Now, for Windows folks, the vast majority of Windows users don’t have the ability to compile software locally on Windows. So, it’s much more important that we provide binaries for MySQL on Windows than it is for the Linux folks. So, that’s most of why MySQL has been providing binaries for so long. Also, we say, “If we built the binary, we’re assuring you that it’s stable on that platform.” And to be honest, most of the Linux distros are very stable as well. The same goes for Mac and Windows. We built the binaries so people can download them and install them.

Scott: So, what percentage do you feel like of the QA or of the bugs that are found and posted for you guys to fix, how much is found by internal QA versus how critical is the community to wringing those bugs out of the product while you’re posting the daily builds, moving towards release?

Jay: That’s a good question. I would have to refer to Omer and the MySQL group, they kind of have these stats. But I would say that, internally, probably 10% to 20% of the bugs are found by MySQL engineers or support engineers. And then you’re going to come across this gray border between who’s a user and who’s a customer.

A lot of users are also customers. Sometimes we’ll get a fairly large installation, say, Yahoo Finance or Google, that submits a bug on a specific version of MySQL. But a lot of times we’ll get larger installations from users as well.

We also have something called the Quality Contributor Program, which is for users that are really our bug seekers. They’re actively trying to find edge cases where stuff just blows up. And so we have a program for people like that. But overall I’d say it’s fairly spread out between internal folks finding bugs, customers finding bugs, and then the larger communities finding bugs.

Scott: Got you.

Jay: But we do get a ton of bugs. The majority are small. In other words, documentation type stuff. Whenever I’m giving a talk on MySQL, I talk about community. And I always say that bugs are gold to MySQL. We value them just as much as anything else from the community—especially a reproducible bug case.

Scott: Right.

Jay: Because it saves so much time for the engineers. Run this code and there, it crashes. That kind of thing is gold to MySQL. So, I’m always encouraging people, “If you ever find a bug in MySQL, don’t ignore it. Send it in.”

Scott: That makes perfect sense. Talk about the work that MySQL—I mean, obviously it’s used so pervasively now, and lots of mission-critical stuff is built on it—what things do you do around security, reliability, all of the “itys” that people talk about with software?

Jay: All the “itys.” [laughter]

Scott: Stability, reliability.

Jay: That’s a good… Performanceability. [laughter] Usability.

Scott: Usability, scalability.

Jay: The three things that MySQL always strived for are performance, reliability, and ease of use. Those are the three binding principles of how our engineers kind of evaluate how well we’re doing. Is it easy to use? Does it perform well? Is it reliable? As far as security and stuff, as an open-source project we tend to worry a little bit less about security. There are just so many people looking for security holes in the software, because they can see the source code and look at it. They can see major problem areas and we usually get notified quickly and respond very quickly to those kind of things. Let’s see… Scalability. I’m biased, but I think we scale very, very well. And it’s always something we’re thinking about internally, because performance doesn’t necessarily mean scalability. You can get a hundred concurrent connections for doing web pages or responses at half a millisecond, but if you can get 10,000 concurrent connections at 0.7 seconds, it’s less performance but better scalability. And there’s sort of this constant refactoring process going on. How can we make this better? How can make it scale better. All that kind of stuff.

Scott: Cool.

Jay: Which I’m sure is the same with any closed-source software house and any open-source project as well. You’re always thinking about all those “itys.”

Scott: Right, right. Well, you’ve got a mature product, so you’re not engineering it from scratch to have all of those, but you’re evolving it, and a lot of the work now is more in terms of…Either there’s a well-defined opportunity to rewrite something and increase performance and scalability, or you’re really just — as you add new features — trying to make sure you don’t negatively impact those areas that are already good.

Jay: Right. As you increase the features in the code base, you increase the code complexity, and you always look out for performance regressions because of that. And there’s a way to combat that, but the general rule of thumb is, the more code you add to something, you’re going to impact the performance. So, there’s always this balance. Do we need this feature? Because the last thing I think MySQL wants to become is—no offense to Oracle or PostgreSQL—a database that has a million features that no one uses.

Scott: Right.

Jay: And that’s something that does go through the mind of the software architects at MySQL. Is this a critical functionality that the majority of users are going to use and are going to value, or is it going to be passed over and just slow down the code?

Scott: That seems to be a key challenge of closed-source proprietary companies, is that they really have to guess. They have to shoot in the dark in terms of… First they have to come up with big features, because if they don’t, they can’t compel people to buy the upgrade.

Jay: Right. Which is the opposite of how MySQL sells our stuff. We’re not trying to be this enormous feature-rich database piece. We’re trying to be the best and fastest online database. And so the features that we’re adding are designed for highly-scalable web applications and online databases.

Scott: And I would guess, too, as you open it up to more and more community input, it will become easier to identify features which will be widely used. Because, the worst thing in the world is to write a feature, nobody uses it, but you can’t ever cut it because it’s actually not that nobody uses it, it’s that three people use it.

Jay: Yeah. And this goes back to that, what’s commercial versus what’s community? And I think that actually closed-source software companies have more of a problem with this, in that a large customer really, really wants this feature in there. And it’ll be added into the code base and sometimes significantly affect performance of another piece, but it’s been bought and paid for by a customer and will stay in there. And unless the software is written in a modular fashion, it will impact adversely everyone else who will never use that feature. And I think the open-source model, which tends to lean towards a more modular architecture, can handle that dilemma better.

Scott: And also, in things that are very, very open source, where most of the code is coming from community contributors, there are no features that are being written because somebody thinks someone else will want them. The only features that are being written are, “I’m writing this feature because I need it.” So, that’s at least a little validation that the feature is needed by somebody.

Jay: Sure.

Scott: But, doesn’t MySQL have just exactly the problem you talked about? Because you mentioned that you guys do some nonrecurring engineering.

Jay: Yeah. And that’s the dilemma that all commercial software companies are faced with. Which is why I’ve said as we move more and more into that mix of community input and also making that core kernel much more modular, I think that we can significantly offset the disadvantage of that, or the drawbacks of having nonrecurring engineering work done.

Scott: So, initially, I’m guessing the main driver of having MySQL be open source was just so that it would be used and accepted. In other words, it’s a much more difficult proposition to sell something that’s completely closed-source proprietary that targets Linux as a primary platform. And so, it seems like a lot of companies open-source their software and they derive their revenue off other things, support and that kind of stuff.

Jay: And packaging.

Scott: Yeah, and packaging. Otherwise you’re just not in the game.

Jay: Right. Well, one part of it is the revenue, right? When you look at open source versus commercial, commercial has just an enormous advantage from a revenue perspective because of their control over their product, right? The open-source company doesn’t necessarily have that. From the exact opposite end of the spectrum, the open-source company doesn’t have nearly the amount of cost involved in R&D, QA, and testing that a closed-source company does.

So, the big shift that’s happened is that you’re going to see closed-source companies start to open source products that they are tired of spending money supporting, and let the open-source community take on the cost of that support. Now, I’ve read recently that Microsoft is open sourcing Visual FoxPro, which I thought was awesome. And then I started thinking about it. I’m like, “Well, they’re probably just tired of supporting it. And just give it to the open-source community and see what they do with it.” And I think that’s where we’ll start to see the first major shift with closed source companies that open-source products because they realize that the cost benefit of doing that, and letting that source out there to the open-source community to test and QA and support, is just so much more worth it than keeping an older product in-house that’s really is not making any revenue.

Scott: Well, and one of the places where you see something similar to that is Adobe open-sourcing the Flex SDK. And to them it makes sense because it wasn’t something that they ever sold anyways.

Jay: Right.

Scott: So, it was free to begin with. There doesn’t seem to be a lot of downsides in open sourcing it.

Jay: And there is a difference between free as in no-cost and open source. And MySQL is open source, and free and open source as in free as in freedom. But it doesn’t necessarily mean that just because something is GPL or is open source that it’s free of cost.

Scott: Right.

Jay: And the original definition of free and open-source software really had nothing to do with cost.

Scott: Right.

Jay: Right. So, when I talked about Microsoft or other companies open-sourcing and making free software, I meant it more in the sense of free as in freedom, so that the developers can get their hands on it, and tool with it, and tweak it, and completely change it, and support it themselves. It had less to do with charging for it.

Scott: But it seems to me like a place where companies are looking at open-sourcing products and making them free as in freedom, are places where the product was already free as in cost, or maybe it wasn’t free but it’s just not generating a lot of revenue.

Jay: Exactly. It’s costing more for them to support it than it would the open-source community. And that’s where I think the first round of closed source becoming open source is going to happen.

Scott: So, MySQL started out as being open-source, but developed by a for-profit company. And at this point you’re moving to more of a traditional open-source model. And I’m guessing it’s because as the product has grown, and as it’s gotten more complex, and there are more and more features to it, there’s a need to kind of force it to be more modular. There’s a need to get more community involvement in the development…

Jay: Yeah, and input. Right.

Scott: And just to really shape the direction because it’s not just a single little standalone database anymore, right? I mean, this is a pretty massive product that you have. And, you’re at that point where you need that cost savings that the community can provide. And you need the input, and direction, and expertise that the community can provide to really move the product forward in the best possible way. Am I summing it up correctly?

Jay: Yeah. Yeah, although I don’t think the plan was ever to push it towards the Apache or Linux model. But we want more of a balance, just like you said, so that we can get the benefits of the community. But also, so we can give more back to the community, and have them happier with the product that we’re providing.

Scott: Sure, sure. There’s more buy-in when it’s something that you can actually work on.

Jay: Right.

Scott: And there’s more buy-in when you touch the product, so to speak, I guess.

Jay: Right.

Scott: OK. Well, I guess, what else would you like to say? I’ve gone through the questions that I had. I’ll go ahead and hand you the microphone. And any message that you think is important to get out that I didn’t cover, feel free.

Jay: Yeah. Well, I definitely did want to point out that the MySQL community headquarters is becoming the MySQL Forge website. And that is http://forge.mysql.com. And, right now, what we have in there is a list of open-source and MySQL-related projects in a project directory (http://forge.mysql.com/projects/). A whole directory of code snippets in various languages on how to use MySQL, or coded in C++, PHP, Perl, SQL,.NET, you name it is in there (http://forge.mysql.com/snippets/). And then we also have our public worklogs, which I mentioned earlier. It essentially represents our big roadmap. And it’s broken into specific tasks that are unassigned or assigned to a specific developer. And the developers really want input from the community. So there’s a way of commenting on those work blogs that I highly encourage people that are interested in MySQL to go and give your input to the developer of that specific piece, and let them know what you think about it. So, forge.mysql.com. And then there’s also a huge Wiki that we’re developing as well. That would be the last thing I’d say.

Scott: So then, if people want to contribute code to MySQL, if they want to actually work on it, how set up are you to take community contributions at this point? Or what do you think the timeline is to where you’ll really…

Jay: We’ve accepted, I think, about 80 contributions already this year—contributions meaning one-line patches to semi-large features. So, we’re already set up to do that. If anyone is ever interested in doing that. I would highly suggest going to http://forge.mysql.com/wiki/Contributing.

Scott: Got you.

Jay: And, that is how you can get all sorts of information on the various ways you can contribute, the different IRC channels where the hackers hangout, and where you can get help, how you can build a test case, all that kind of stuff. How you can build locally, and all that kind of information is all in that page?

Scott: Jay, thanks for taking the time to chat.

Jay: Thanks.

Bookmark this:

Phil Costa

In this interview, Scott Swigart, Sean Campbell, and Richard Bowler interview Phil Costa who is the Director of Product Management for Flex and ColdFusion at Adobe, with responsibility for product definition and strategy of the Flex product line. Prior to joining the Flex team, he was product manager at both Macromedia and Allaire and led XML and Internet middleware research at Giga Information Group. Phil has a Master’s degree in English from Boston University and an undergraduate degree from Swarthmore College.

In this interview, Phil talks about Adobe’s decision to open-source the Flex SDK. In specific, Phil talked about:

• Background on the Adobe Flex SDK
• Why open-source the Flex SDK
• Risks in open-sourcing the Flex SDK
• Why the MPL license was chosen
• Advantages in open-sourcing the Flex SDK
• The risks of people forking the build

Scott Swigart: We were commissioned by Microsoft to do an investigation into the differences between how closed source and open source software are built, and how those changes in software development methodologies manifest themselves in the final products.

[Adobe] Flex was particularly fascinating to us, because it started out as a closed source product, and now the Flex SDK is being open sourced. We wanted to get some insight about that. That obviously made sense for Adobe to do, but it isn’t necessarily obvious to us what goes into a decision like that. Why does open-sourcing Flex make sense?

Phil Costa: It might help to give you a little bit of background on Flex. We announced about three weeks ago now that we were going to be open sourcing the Flex SDK. That actually hasn’t happened yet, but it has been a process we’ve been going through for the past two, two-and-a-half years. It has really accelerated in the last year.

The Flex product line started as a server-based product that we sold to a very small number of high-end customers, and we’ve been gradually opening it up, both from a license standpoint, in terms of making it free or parts of it free, and then also opening up from a source code standpoint.

We started publishing all the source code about ten months ago, and we’ve been gradually more and more open about the roadmap we’re taking with the product. Actually making the license open source was a logical next step which we’ve always envisioned taking.

Scott: What are the advantages of going with more of an open source model — what are the advantages of taking something that hasn’t been open source and open sourcing it?

Phil: There are a few different elements to it. First, there’s the nature of the product itself. The core part of the product is try hard to internally test, and get all the bugs out, because it’s a development framework. By its nature, it gets used in a million different ways. It’s very hard to actually set up tests for all those ways and to chase down all the bugs.

So having an open source model will actually help us by having more people looking at the code and suggest changes based on their particular use-case. In one respect, we view it as a way of magnifying the QA resources we have, and also magnifying some of the bug-fixing resources. So, that’s a very tactical thing, but that is a proven advantage of an open source project.

The second element refers more to the evolution of the product. We’ve always tried to be very customer-centric in terms of designing the product, working with early adopters, niche users, and new users, to determine what things we should be adding to the product, to make sure that it fits their needs, both on the learning-curve side as well as the power-user side.

We decided that having a group of people who can directly influence that — or at least feel more deeply invested in it — was a good way to continue to evolve the product. With a developer product, the people who develop the product are also the users of the product, and there’s a very efficient feedback loop.

The third piece is more PR and marketing focused. Because people have to make a substantial investment in Flex — in terms of spending a lot of time developing an application and then making their application dependent on the Flex framework — they’re looking for an open source project or a set of standards. In the rich-ended application space, there aren’t really any standards, per se. They’re mostly looking for open source or de facto standards.

It’s become increasingly apparent, as the market has grown up, that to be successful as a platform you need to be an open source project, so that people view the product as being bigger than one individual company, or in some cases one individual product team. In a lot of ways, that was another requirement that more and more customers were raising; they love Flex but they wanted it to be bigger than just Adobe.

Scott: What are some of the risks that you saw, if any, in open sourcing Flex? It was an evolutionary process, it’s a track that you’ve been on, but were there risks or concerns that were identified where people thought that this might not be a good idea?

Phil: I think some of them are unique to a product that is going from being a closed source product to an open source product, and some are more cultural.

On one hand, we looked closely at taking all of the current business relationships we have with people around Flex, not only from a revenue standpoint but also from a contract standpoint, and made sure that as we move to a different model around the Flex framework that we could either maintain the revenue streams, grow the revenue streams, or were willing to give them up to get other benefits from moving to an open source model.

Then, on an individual basis, we had to make sure that none of our customers were going to be left in the lurch because of the particular license we chose or the fact that we were going to open source at all. Because Flex is used not only by developers within a company for building an application for their web site, but also in a lot of commercial software products.

For example, SAP uses Flex, as do a lot of smaller ISVs like Dorado and other companies who have their own very specific processes and policies around the use of open source software that they can incorporate into their products. We needed to make sure that there were options available for them that would not cause all kinds of problems. That’s why we chose a dual license model, so that we could address our goals of becoming an open platform while also enabling companies that needed a traditional, closed-source license to also use Flex.

Richard Bowler: I’m guessing that’s why you guys picked the MPL license, because they could combine a proprietary module with the open source modules, and not be forced to open-source their product?

Phil: Exactly. We wanted one that was strong in the sense that it was viral around the core part of the product, but not so strong that no one would use it in a commercial product, because we wanted people to use Flex in commercial products.

We looked a little at LGPL. But our own history as a commercial vendor shows that even though the spirit of LGPL is that it can be combined with other things and it doesn’t affect them, in practice a large percentage of ISVs just won’t touch it.

Whereas things like MPL, ETL, EPL, those licenses were explicitly designed to be more commercial friendly while still carrying the basic spirit of open source.

Richard: I presume that in the past, other commercial products that are using Flex have paid for that privilege, and now anybody that wants to use it in a commercial product is going to be able to. Is that true?

Phil: That’s true. Originally, no one could use it without paying for it. But about ten months ago — last June — we made the core part of the product free, but it was still closed-source. We made the SDK, which was free and it included the source code, and there were specific cases in which you could modify and redistribute the source code, but it was all done under a traditional commercial license.

By making the move to open source, we’re making even more parts of the product free because not only can you use it for free but you can modify and redistribute the SDK for free. Of course, there are still lots of companies that will pay us to obtain support and escalations or for the additional Flex products we sell. If they’re making a big investment in Flex development, they want to make sure someone is on the other end of the phone to help them out if they run into issues.

Richard: It seems that part of the motivation for open sourcing and making it free, was it just to get wider market acceptance?

Phil: That was definitely part of it. There were lots of interesting open source projects that wanted to use Flex that didn’t feel like they could because of their practical or ideological limitations.

The other part was that, as the developer community has gotten bigger and more active, people are looking for ways to contribute back and to participate in Flex. And there wasn’t really a good way to do that, because it wasn’t being run as an open source project.

Richard: It’s an interesting paradigm, because Flex is a developer tool, so your users are the kinds of guys that are extending their own presence on the web by using your product. They are going to have very strong ideas about how your product should grow and what it should become. I imagine you’ve got a lot of feedback from your users already about how they want Flex to grow.

Is this also a way to remove some of the blockades to collaboration so that the user community can be a little more in control of how the product changes over time?

Phil: Absolutely. The way you use Flex is you write ActionScript, which is a JavaScript derivative, and MXML, which is an XML layout language for the UI. The Flex framework itself is entirely written in ActionScript, so people who get deeply involved in Flex are very deeply involved with the tools that actually created Flex. They’re the perfect people to provide feedback and to add things to it to support their needs.

What we’re trying to do is set up a process where they can do that, but it’s still run in a way that preserves the overall philosophy of the product design, and the goals of the product. The compatibility and stability of the framework is maintained over time.

Richard: I imagine that the gatekeeper for this project is ultimately going to be someone other than Adobe.

Phil: The initial plan is that all the committers will be the Flex framework product team. But over time, we plan to add external people who demonstrate skill, judgment, and commitment to the overall mission of the product. It is a big, complicated product, and we’re going to take the evolution of the project one step at a time.

Scott: One of the challenges that open source products face is that somebody decides they want a change in the product, but the people who control the source decide that they’re not going to incorporate that change, so somebody just forks their own build. And now you’ve got two of these out there, three of these out there, and so on. Is that something that’s a concern at all with this project?

Phil: It was one of the explicit concerns we talked about, and it’s still possible, certainly. I don’t expect that you will get a lot of strong forks. You might get weak forks where they take a particular area of the product and create an alternative implementation of a particular component or a particular subsystem.

Scott: It seems like your risk might be lower too, just because this is so tied into Flash and other things that are closed source proprietary Adobe. There’s only so far you can really go with your own fork.

Phil: It’s true that it’s a little bit different for those reasons. In the end, they can’t change the Flash Player API, but Flex does offer a lot of levels above the Flash Player that are design choices that we made.

I think more of what made us comfortable in the end with the question of forking was on the one hand, generally, the practice has been that people do not make major forks lightly. Having a community that can maintain and develop new projects, and taking a lot of resources and trying to split them just means that both projects have fewer resources.

And on the other hand, if you look at it optimistically, if you manage the project well and you have clear reasons why some things are being accepted and some things are not, then things that are not accepted in many cases can be moved over into a sub-project area or a related project area where they have a chance to evolve, even if they’re not becoming a replacement part of the project. In general, people submit to the group project’s will.

Richard: Are you worried about people writing extensions to it and keeping those extensions closed source and trying to pump up the value of code by just adding a neat widget?

Phil: That’s actually one of the things we encourage people to do. To give you an example, today we sell the core part of the framework, which includes the application model, a bunch of core classes, and then a library of components, and that’s the part we give away for free. On top of that, we have a pretty advanced charting and graphing package that we actually sell.

We’ve been working with a bunch of vendors, and there are already some of them out there who have built other component libraries and are selling them. So we encourage people to innovate around the core. The core we’ve made free, but we expect that more specialized visualization tools and so forth would be commercial products.

That was one of the other reasons we chose MPL — because of the nature of the framework, it gets blended in with your application. We wanted to be able to both have open source versions of that that are more community driven, but commercial versions that range all the way from individual components to full blown applications.

Scott: We appreciate you taking the time to chat.

Phil: Thank you.

Bookmark this: