Interviewee: David Campbell

In this interview with David Campbell we talked to him about:

• How the development methodology of SQL Server has changed.
• How the changes affect the ability of customers to give feedback.
• How virtualization makes it easier for users to test-drive SQL Server.
• As products and open-source projects grow, the developers have to realize that they are not the target user.
• David’s belief that the best product doesn’t necessarily have the most knobs.
• Building a community around a closed-source product like SQL Server.

Scott Swigart: David, thanks for taking the time to chat with us again. One of the things you mentioned in the first interview was the common engineering criteria. I’ve seen that mentioned in a number of different places, but I have never seen it explained. And I’m guessing that the common engineering criteria impact a product like SQL Server. It might be good to start off just by explaining what they are.

David Campbell: It started in the Windows Server system. We’ve been telling customers that one of the advantages of our stack is that it works better together. It does, but we started really challenging ourselves by asking, “OK, what are the real things we’re doing to actually make it be better together?” The way I think about it is, “What are the things that would add value for customers who require coordination or consistency across groups that otherwise wouldn’t happen organically?”

We started this a number of years back, when Paul Flessner was running the Windows Server group. We have a set of requirements, which are updated every year, and all the enterprise products within the Server and Tools Business (STB) have to adhere to those requirements. These are things like having a best practice analyzer, having a Management Pack for System Center before you RTM, having consistent user education, continuous publishing, a community engagement program, that kind of stuff.

Scott: Previously when we talked, you also mentioned that SQL Server went through a pretty big change in its development methodology between 2005 and 2008. Talk about that a little bit.

David: This is something that a small group started, and it ultimately took great effort across the division to pull off. It will never be “done,” but it has been an amazing, fascinating transformation. I’ve learned more in the last few years about culture and change management in a large organization than probably anything else.

I’ve been working on the product for a long time. Back in the good old days, we had something like 20 people who knew the whole server, end to end. Back then, as you worked on things, if you had an issue you just walked across the hall, sketched something out on somebody’s whiteboard, and went back and coded it up. Because things were so intimate, you could keep things moving at a rapid pace.

To somebody who understands the standard Microsoft development methodology, one of the things that’s very important is the daily build. We continuously build and test the product, and one major vital sign is the health of the daily build. Back when the product was small and there were 50 to 100 people working on it, you could do the daily build live, with everyone just checking stuff in and keeping it together.

But as the team grew and the size of the code base grew by several orders of magnitude, you just couldn’t have as many hands in the soup at the same time and get anything that tasted good.

In SQL 2005, we had sort of a standard large-team, waterfall model. If we were doing a feature that spanned several component groups, we would have one group get together and they’d do everything by the book. We had a process and everyone followed it. They’d write the spec. They’d develop a test plan. They’d review the test plan. They’d write the code. They’d test it. They’d check it in.

Then the next component team in the sequence, maybe it was the client data access team, would pick it up and they’d start working with it. And they’d go, “Hey, wait a minute. This interface isn’t quite right. There is something wrong here.” They’d go back to the first team and ask them to change it. Of course, the first team had moved on to something else, so if they had to go back and do major surgery, it was a hassle.

We had one feature in particular that, to be honest, we built two or three times during the course of SQL 2005 and we still didn’t have it right. So I said to the program manager at one point, “Look, either we take it out of the product or you get everyone, end to end, involved, get them in a room and just sit till you work it out and get the thing going.”

This involved the storage engine, the language parser, the query processor, the client data access libraries, the management tools. They had done it serially several times. They just never had it work correctly, end to end, when integrated.

They all got in a room. Of course, you know how that works. They were able to figure it all out and get it all done. We said, “OK, what can we learn from this?” We completely threw the development process up in the air and changed it around. How do we go “back to the future” and try to capture the intimacy of what we had in the “good ol’ days” now that the team and product are many times larger and more complex.

When we had a small team rebuilding the database engine for SQL Server 7.0, a number of people had the whole system in their heads. Furthermore, when we did SQL Server 7.0, we really didn’t need to think about the market that much. We were building a new relational database engine and introducing an OLAP system. The playbooks were already written for these technologies and the markets were already established. Now, we did it in a way that was different; we tried to make the product much easier to use so we could actually grow the market and bring the technology to a much larger base. For SQL Server 2000 and, to a lesser extent, SQL Server 2005, we were building out on the architectural base we had established with SQL Server 7.0.

The original model worked great for several releases, but when we got to SQL Server 2005, the product was much bigger, the team was much bigger, and the market expectations were much higher. Things just weren’t fitting together in terms of consistency across the product. So the other piece that we had to add for the SQL Server 2008 process was, instead of defining the release from the bottom up in terms of what we were going to do, we had to create a definition process that looked both at the market, top-down, and at what needed to happen to the technology base of the product and what people wanted to do, bottom-up, and bring them together.

The way we framed it to the team was, “Look. We want to have the PowerPoint presentation that we’re going to use in the keynote for the launch before we write the first line of code.” So we developed a set of themes for the release. With each theme we had this notion of scenarios that were end-to-end value propositions for customers. Underneath those scenarios, we had a number of what we called improvements, instead of features.

An improvement is what we would consider an engineering transaction. The idea is that we bring virtual teams together to work on an improvement. An improvement team is cross-discipline—development, testing, and program management—and it’s cross-component. For example, the database engine, data access, and management tools teams can come together to work on a single improvement. So if a particular improvement needs focus from four different components, they’ll all come together as a virtual team.

We assign an improvement team lead, who’s got a set of requirements they need to meet to integrate the result into the source base, but we’re not very prescriptive about how they run the team. If they want to use Scrum and have stand-up meetings and do a series of sprints, that’s fine. If they want to use a test-driven development methodology, etc., that’s up to them. We’ve got tools and a playbook that we encourage them to use, but we’re pretty liberal in terms of how they go about it. There’s a very strict contract in terms of what needs to be done before it is integrated into the main branch of the product source code, however.

These improvement teams work in their own separate branches in the code management system. Once they believe that the improvement is ready to ship to customers, then and only then will we integrate it into the product. What’s interesting, if you think about it from outside SQL Server, from the customer’s perspective the release looks much, much different than it used to look.

It used to be the case that—let’s say we were going to do 300 features. Everyone gets started on them. We put a bunch of stuff together and put it out for Beta 1. Everything would work about 30 to 50 percent. Then we’d keep going for Beta 2. We realized, “We’re not going to be able to finish 20 percent of these things, let’s cut them.” With Beta 2, things are working about 70 to 80 percent correctly. Then you have Beta 3, a bit better. And in the end, you’re still trimming stuff out, cutting it, and refining it to get it ship ready for customers.

We were constantly in the mode of having stuff that kind of worked in Beta and trying to get feedback from customers. We flipped it around with the new process. What customers see now in a particular drop is very high quality. They used to see the 120 percent of the ultimate features in Beta 1 in some measure. Now they only see 20 or 30 percent of the features early, but they all work. We also changed the name from Beta to Community Tech Preview, or CTP.

As we go through each successive CTP, more and more of these improvements show up. They’re baked, end to end, and they accumulate in number. What a customer is seeing is that the quality is actually very good the whole way through, so it’s an interesting change.

Scott: It seems as if the SQL Server team went through the same process the Visual Studio team did: Instead of putting code out there that is 30 percent functional and people can’t really get it to work end to end, they decided it’s better to wait until a given feature is 100 percent even if that means customers see it much later.

But there has to be a trade-off, because your window for getting customer feedback is smaller, and you have potentially invested a good deal more time in the original code before you get any feedback. Talk about that balance.

David: That’s a great question. One aspect of the previous releases is that we were frankly playing catch-up in an established market. We knew what we wanted to build and we knew how we wanted to differentiate in the market so we really didn’t need a lot of customer feedback on the feature set for SQL 7.0 because in many ways we were rearchitecting and reimplementing the existing product. This approach worked for SQL Server 7.0 and 2000, but one consequence is that we hadn’t built up muscle around how to engage customers earlier in the design process for completely new features. In some sense, we built stuff and threw it against the wall to see if it would stick. This worked well for awhile, but it made much less sense starting in SQL Server 2005.

Basically, you could say that the old process was quite inefficient, in the sense that we built a bunch of stuff, actually committed code, before we even knew what would stick in the market. Now we’re trying to do that through more concept testing, more up-front work, spending more time with customers to look at their real pain points. Are you going to build a set of features that’s going to be compelling? That’s one piece of the puzzle. Are you building something all up that will be valuable to your customers to you customers end to end considering they way they really work? That’s another pieces of the puzzle.

But then the question becomes, have you built each feature in a way that makes sense to the customer and is useful? So it’s less about the value proposition of having a particular capability and more about have you designed and built each thing correctly, on an improvement-by-improvement basis.

One fallacy around the Beta process, in my opinion, is that we would throw a Beta out there, and tens or hundreds of thousands of people would download it. A small fraction would really play with it, a smaller fraction would actually think about how they’d use a new feature, and a fraction of those people would actually give us any real feedback. It wasn’t very structured. Our TAP program tried to target specific features and scenarios but it mostly kicked in after we had written the code.

I believe it’s a bit different in the Developer division, because you’re close to developers. One of the challenges we have with SQL Server is that, for our real validation, you need to get closer to a production environment. And it’s very difficult to get the feedback from the production team.

Scott: Right. With SQL Server, a single dev banging away isn’t a real test. You could get somebody banging away at the feature, but that doesn’t really tell you whether it will scale, whether it will be reliable, whether it will be secure.

David: Right, exactly. So what happened with this release was, we had some teams that said, “Oh, crap. What are we going to do to get feedback for this really complex improvement?” For the database engine there’s an improvement called a Resource Governor. It allows you to constrain queries that consume specific resources so you can limit a particular connection to 10 percent of the CPU, and that kind of stuff.

The question is the mechanics of getting the architecture and user interface right. We think very hard architecturally about separating mechanism and policy. In this case, the mechanism would be things like how do we keep track of the various resources and who do we attribute the resource usage to. The policy would be how the user describes her intent in a way that allows the mechanism to enforce it. In the case of the Resource Governor, there’s a complex interaction between the policy and the mechanism, – not the least of which is that we needed to build new mechanisms to hold and enforce various policies. This is a great example, because what we would have done before was stare at the whiteboard, do a design, and throw something together. We’d ship it out in a Beta and hear, “No, that’s not what we need.” So it turns into a rock fetch. “No, this thing stinks. … This thing stinks. … You’re getting warmer …,” etc. It’s a pretty inefficient process all up.

In the case of the Resource Governor, we got together with some of our MVPs and key customers who had a need for the improvement and did more of an outside-in design process starting from use cases. What’s interesting is that this outside-in approach is completely natural for many people building applications, but for a development team that’s been creating low-level system software for 10-plus years it’s quite different.

The bottom line is that we definitely needed to figure out how to engage customers differently with our new development process, and there are opportunities for design councils, concept testing, early prototypes, and early builds. In fact, things like virtual machine technology allow an improvement team to create a build of their early thinking and test it with a small group in a focused way without having to integrate everything else that’s in various states of completion, like we used to do with the standard Beta process. They just create a VHD with their bits and have some customers try it. With this model they can iterate faster than was possible with the old Beta process. Ultimately, I think we’ll get better improvement feedback by finding the right 10 to 20 customers and working closely with them rather than putting it together in Beta and throwing it out to 100,000 people hoping they’ll give us valid feedback. I’m overstating things a little bit to make the point, but the general concept still holds, although not everyone here agrees with me on this one.

Sean Campbell: I’m curious about the point you just brought up, because Scott and I have been knee-deep in virtualization for a long time. The point you just made about letting people evaluate it on a VM, how has that impacted the overall development process for the SQL Server team? Has it given you the ability to basically give, let’s say, broken bits out earlier and get more people working with it earlier because they don’t have to install it, they don’t have to configure it? I’m sure in the past people would try to install it on top of whatever else they already had. You could tell them a hundred times, right? And they’re going to try to install it as a multiple instance on top of what they’ve already got.

David: It has, in exactly the ways you said. These applications are so complex, and the state that they put on the machine is so complex, and we work so hard to make sure uninstall really uninstalls all the stuff that it put there in the first place.

The other piece is that it’s hard for us to build and test the setup and the installation. With VHDs, we can basically handcraft them and then clone them, and they don’t pollute any other state on an existing machine. So it’s much easier to get together. The bulk of the savings is in the setup, for us to construct both the setup and the uninstall, to make sure we don’t pollute the machine state.

So I think from that perspective, it’s very, very helpful. I think we saw that, but we haven’t really perfected it as a process yet. I think we’ll probably go a little bit deeper on it in the next release.

Scott: In talking to a lot of people in the open source world, especially community-driven open source, like the Linux kernel and the Apache Web Server, I hear there is a pretty strong correlation between your ability to request a feature and your ability to actually code it. You will get on a mailing list and say, “Hey, it’d be great if the Web server did this.” And the response is basically, “Yeah, that would be cool. When will you have the code ready?”

What you have talked about is very different from that, in terms of interviewing customers, interviewing influencers, and putting together a team. Talk a little bit about how a specific feature happened from end to end, how the feature was initially conceptualized, how you validated it, the portions of the product it touches, etc.

David: Well, I’ll talk about it in the meta form, and then I’ll talk about a particular feature. It goes back to the thing I was saying about technology evolution. And it’s actually one of the challenges.

One of the challenges the open source community faces, and this will be a sweeping statement that I get flamed for, is that they don’t yet generally appreciate the difference between the consumer and the producer. By that I mean, as a technology matures, the gap between who’s using and who’s producing it becomes greater. It’s a challenge we face at Microsoft; we are no longer simply building products for people like us, the engineers. In one of the talks I give to the product development community at Microsoft, I stress, “Look. We’re no longer just building products by engineers, for engineers. The PC is moving toward a consumer electronics device for many users and we must build a product that interacts with them on their terms, not ours.” I then show screen shots of Task Tray bubbles talking about “Virtual Memory” and “Pagefiles,” etc. My friends and family shouldn’t have to know what virtual memory is.

We’ve seen the same thing in the database space. Twenty years ago, every DBA had to be a wizard and needed to know how to fiddle with hundreds of knobs, figure out how to manage raw disks, and learn all sorts of low-level details just to create and maintain a database. When we were building SQL Server 7.0, I used to troll the database newsgroups an awful lot. You’d see people asking perfectly reasonable questions and you’d see responses like “RTFM” or “You don’t belong here if you don’t know that.” So we started this campaign to fix what I called the “you dummy” questions. Basically, if an expert responded to a question with a “you dummy” tone, we saw it as an opportunity to address the issue by engineering—teach the product to do it rather than teaching hundreds of thousands of DBAs. We turned the “you dummy” on ourselves and did something about it.

I don’t have to be an expert on my automobile to keep the thing running, and that’s fine. If Toyota wanted to come to me, I wouldn’t have to talk to them about ignition advance, dwell time, and those things anymore. I’d talk more about my experience and what I wanted the car to do for me. I think that’s just an evolution, and you can certainly see it in the database space.

To get back to the specific feature, I think the Resource Governor is a good one, where there was a complex set of mechanisms inside our system that we needed to go off and engineer, but how we constructed those and how we presented them to users was the challenge, right? What is the policy? How does someone describe a policy around constraining resources in a particular query or session? And then how does someone bind different policies to different classes of users or applications or queries or times? What degrees or what dimensions do they need?

Another way to think about this, from the perspective of design, is that the software development community has not made the leap yet from expressing the product control surface as knobs on the underlying mechanism to capturing the user’s intent or objective and then acting to achieve that. A simple example is: Rather than having an administrator configure how many dirty database pages will trigger a database checkpoint, we can capture the user’s intent in terms of the trade-off between fast recovery in the event of a failure, which can be achieved by increased checkpoint frequency, or better run time throughput, which can be achieved by fewer checkpoints. Having a control that captures how long the administrator is willing to wait for recovery to complete in the event of a restart looks directly at the business intent rather than fiddling at the level of the mechanism. I mean, how many dirty log blocks can I have on a particular database and still have it recover in 60 seconds, anyway? Instead of publishing some equation in the documentation, capture the user’s intent and treat that as a constraint during run time.

So to go back to the Resource Governor—we go to the customer and ask, “OK. What sorts of things do you want to constrain? What dimensions are most important? Are they applications? Are they time of day? Are they this or that?” And then gather all that, design something that’s consistent from the user intent perspective, and then figure out how we build a mechanism underneath to marry up with it.

Certainly, there’s a lot of success in the open source community, using open source technology in consumer electronics. But I don’t think that they’ve gone end to end in terms of doing a great job of user-centered design.

Sean: Well, that’s one of the things I’ve talked to people about, too, and that I personally feel pretty passionate about, innovation in the usability area.

I am curious, from a closed source company perspective, about what processes you feel should be put in place in order for a development team to excel in terms of usability. You go out and talk to people about potential features to help make sure that a feature doesn’t turn into engineers building it just for engineers, and that you actually end up with something—like the Ribbon in Office—that honestly has been a success, if only because, after release, nobody complained about it.

It’s the proverbial tree that fell in the forest and nobody heard it, but everybody was freaked out about it. And I keep telling people, “Well, think about this. Microsoft changed the UI on the one application people use the most, on a daily basis, and yet nobody really complained once Office launched.” There must have been something that really went into the thinking, in terms of usability.

For you guys, especially with a product like SQL Server, where I’m sure you can lose yourself in B-tree discussions until the end of time, how do you ensure usability in the product?

David: It’s a great question and a great challenge. It’s funny, but one of the things I’ve been doing for awhile that’s had a good effect is to hold the mirror up to the product development community at Microsoft, to get them to see that we’re no longer building products for people like us.

I have a set of slides that I’ve been using for six years or so, where I have collected some crazy error messages and I have real-life scenarios. I’ve got a screen shot where, in Windows XP, if it extends to your page file, you get this balloon coming out of the system tray that says, “Your virtual memory is low. We’re extending the page file. Blah blah blah.” I came home one night and my wife hit this and she said, “Hey, I think my computer’s broken. It says something here.” So I started describing to her virtual memory and page files, and she said, “Shut up! Just fix it.”

It dawned on me that perhaps 99 percent of the people using PCs these days are not engineers, yet we’re still throwing these exceptions and doing these sorts of things as if they were. And we ridicule the users instead of ourselves. I go help my neighbors fix their machines, and they all start with, “Oh, I feel like such an idiot around computers.” And I just want to say, “No, you’re not the idiot. We are, because we’re not building computers for you yet.”

And so I have a set of rules, or things that I’ve captured, that I talk about, such as the “principle of least astonishment,” where a reasonable user action should do something reasonable with least astonishment. The example I use is another story from my wife. I don’t know how many years ago, when I first got her a PC, someone emailed her some pictures. She said, “How do I look at these pictures?” I said, “Just double-click on the filename right here.”

So, of course, up comes the picture in the picture viewer. And there are little buttons down on the bottom of the viewer: “Next picture,” “Previous picture.” What do you think happens if you click on “Next picture”? You don’t get the next picture in the set of attachments. You get the next picture of whatever the next file was in your IE cache or whatever it happens to open.

And she’s going, “What the hell is this?” She gets some random icon from the temp directory or something like that. So I started talking to her about temporary directories, the IE cache, and blah blah blah, and again she’s like, “Shut up! Just tell me why it’s broken.”

Sean: She’s saying, “I’m looking at photos. I want the next photo.”

David: Exactly. And you see that even in the database space. We’ve taken the market from tens of thousands of servers out to millions or tens of millions of servers. You can’t have every one of them require a highly skilled DBA. We just couldn’t do it economically. And so you have to get out of the mind-frame of an engineer and go adopt a perspective of what the customer needs and how the customer wants to express their intent.

That’s the other sort of piece I talk about: In terms of the degrees of control and the dimensions of control, get rid of all of those dimensions of control that don’t capture any user intent and figure out which dimensions of control are orthogonal with respect to the customer’s intent, and then figure out how you build your system, and express things that way. That’s been the most effective thing I’ve found to motivate the change.

The bottom line is that the software development community doesn’t teach “design,” certainly not with a capital D. Most developers believe that if they think for 10 minutes before coding up some function, then that’s design. It’s not. This isn’t a development methodology thing, but rather a cultural change that we need to go through across the entire industry.

Sean: That’s great. The chief creator of a project called Quicksilver for the Mac was giving a talk at Google, and I just saw the video of it the other day. His theme for the project is “act without doing,” which has an interesting similarity to what you are talking about. If you saw the app, you would probably see some interesting things in it.

To follow along that line, I have a question about Oracle. Oracle, through the years, treated the product as though the more complex the cockpit, the better it was. If you walked in and it looked like an old 727 cockpit with 50,000 switches, that was nirvana. And you were paid to know every position of every switch, right?

David: Yep.

Sean: The open source community has, to some degree, a similar bent, right? Not in everything, but in a fair amount of places. But even in a product like Ubuntu, which is seen as the ultimate of end-user experience—and it has made some great strides—there’s a massive list of steps just to join that machine to a Windows domain, for example, if that is something you wanted to do.

David: Yep, yep.

Sean: So, do you see any similarities with this line of thought in the way Microsoft helps you to administer a database and the way open source helps you with administration in projects like MySQL?

David: That’s another good question. Yes, Oracle—I think I mentioned it earlier, they marketed our ease of use against us, frankly. They said, “How can this thing be a real enterprise database product? It doesn’t have nearly as many knobs as ours does.” And when we left Digital—this is an interesting little story—we had a product, RDB, that had roughly as many knobs as Oracle did at that point.

And like engineers, we were saying, “Oh, shoot! Hardly anyone knows how to turn the knobs on this thing.” So we wrote more software, something we called RDB Expert, which was a physical database design and some other things, to recommend and actually turn some of the knobs for you.

I think the first step that you go through in this evolution is to put a facade on it, whether it’s a system that recommends turning the knobs or a GUI to cover up knobs in config files. But really, at that point, you haven’t got a good end-to-end design, because you haven’t, in my opinion, eliminated those points of interaction or control. You haven’t captured user intent.

It’s like having a TV that still has horizontal and vertical control, but you’ve put something on the TV that can watch it and turn the knobs for you. So I think the next step in the evolution is to have the system software itself become more adaptive.

Scott: That’s true. TVs used to have horizontal and vertical knobs on the back. They don’t have those knobs anymore, even though TVs are much more complex, and no one misses them.

David: And what you wind up with—here’s the key point, and this is something I didn’t understand a priori, but after the fact it really made a lot of sense. In SQL Server 7, when we did some of the memory work to have the system automatically adapt to the environment and adjust the amount of memory it consumed, we changed the system so that you could adjust the amount of memory that it was using dynamically. And the mechanism was dynamic as well, so we had an automated policy that was a closed loop, but for those people who wanted to get under the covers, they could change it themselves.

But the neat thing about the way we reimplemented it top to bottom, and not just put a facade on it, was that the underlying mechanism itself was dynamic. By that I mean you could change the amount of memory that SQL Server used while it was running and not have to shut it down and restart it to get it to adopt a new value.

It’s back to the separation of mechanism and policy. We made the mechanism dynamic, we made the policy automated, and, where it made sense to try to capture user intent, we allowed user intent to be captured and actually influence the policy. And that takes architecture, top to bottom. I think the evolution is: You recognize it’s an issue, you put a facade on it, whether it’s a GUI or whether it’s some other code, and then you have to step back and architect the thing, top to bottom, to do it correctly.

Scott: One of the attitudes I ran into back in my C programming days was, “Look, programming was hard for me. It’s hard for a reason. It should be hard. If it isn’t hard, anybody could do it, and that’s the last thing in the world you’d want.” I run into that attitude whenever I look at systems administration and other technical issues, where people say, “Look, this is complicated. It should be hard. You should have to be a little bit of a rocket scientist, otherwise you have no business doing it.”

So one of the complaints is, “Well, yeah, Microsoft’s made it so that any idiot can set up a Web server, therefore every idiot does, and they don’t keep it patched, and it’s vulnerable.” Same thing with the database: “Sure, people could install the database and set it up, but they won’t create their indexes correctly, and they won’t keep it patched.”

So there are certain people who would say you should stop more people at the front door and not let them in. Because if you do let them in, they get farther down the path, thinking that they know what they’re doing, and then they run into a disaster. What would be your response to that kind of attitude?

David: I think it all comes down to, or goes back to, design in the sense that if you have the means within the technology to bring it out to a broader market and let people do the job, then I think it’s incumbent on you to try to figure out a way to do that. It’s a great example and a great point around physical design of the database—how do I design the index set? It’s very difficult.

One of the things we did here is, we designed the Database Tuning Advisor. This was work with Microsoft research where we’ll look at a workload and actually propose indexes based upon the workload, and do physical design that way.

And the way we test that is kind of interesting. We bring databases in, we measure their response against a real user workload. Then we strip all the indexes and we use the Tuning Advisor to rebuild the index set. More often than not, the Tuning Advisor does a better job than the guy who gets paid a lot of money to do it by hand. Of course we can’t see the reporting jobs that are running at the end of the month if they haven’t been part of the workload, but it does a pretty darn good job. So I think it’s a matter of how far can you take the technology.

Another sort of thought experiment here is if you go back to the TV front. Go back to 20 or 30 years ago. Every small town had two or three TV repairmen. The technology is more complex now than it was, but it’s perhaps 50 or 100 times more reliable. So you don’t see the TV repairman in every small town anymore. The TVs just go and go and go until you decide you want a slick new plasma one. I think that’s the way software is going to move as well.

Sean: Obviously, one of the strengths of open source is the community. I think that is one contest they tend to win hands-down, if for no other reason than that open-source projects were driving community long before closed source companies were. But at the same time, I know Microsoft has been making some significant efforts in this area too.

You mentioned in the previous conversation about CodePlex. I knew about CodePlex, but I have not really looked at it with an SQL Server focus before. A brief glance at it showed me some of the activity that is SQL Server related. Talk to me a little bit about what is happening there, how that activity is affecting the product team, etc.

David: I agree with you. I think I mentioned earlier that tapping into the energy of the community is super valuable. I think of a lot of these things in terms of closed loops and how quickly can you run the loop, how fast can you get the feedback, and how fast can you incorporate it. Tapping into the community is a super way to do that. There are lots of examples of closing a loop, like Watson, to have the machine send back error information and actually close the loop and automate things to enable continuous improvement.

As I think about community, absolutely I give the open source guys credit for tapping into that phenomenon first, but I don’t think it requires open source in terms of the development or distribution model to engage the community. I think about community engagement in terms of layers. With the product itself there’s a way of tapping into the community by harnessing their feedback to improve the product over time. I think with the documentation content and general knowledge around how to use the product, there’s a deeper way of tapping into the community. As we publish content for the product, can we allow people to modify it, link to it?

Some of our content could provide a spine for community activity. For example, we could build a collaborative site around the product error messages and allow the community to link their responses into each error message to help one another out when they hit a particular issue. Today much of this happens in forums and newsgroups, but it’s not done in a way that closes the loop effectively. If we did it in the way I just mentioned, it would be much easier for the product development team to review and mine the activity to improve the product going forward.

I think the next level beyond documentation would be extensions and tools. Things that you could build around the product where the core of the product is still in a closed source model but you could open up and distribute add-ons, add-ins, and other sorts of tools and actually build the community around that.

Sean: This has been a great conversation, David. Thanks for taking the time to chat.

David: Thank you.

Bookmark this:

Interviewee: Justin Erenkrantz

In this second part of a two part interview with Justin Erenkrantz we talked to him about:

• How the Apache project ensures good collaboration.
• The Apache Foundation’s philosphy of having no single person as the leader.
• Apache’s security committee.
• The process of removing someone from a position of responsibility within the Apache Foundation.
• What would make someone want to be part of the Apache Foundation’s group of projects.

Scott Swigart:  In this part of the interview, we wanted to dig into some of the tenets, if you would call it that, of the Apache way. And the first of those of course is collaborative software development.

So talk a little bit, if you would, about how Apache does collaborative software development. I’m sure some things are very traditional and similar to the way that other open source projects might do it, and there are probably things that also might just be a little bit unique to Apache. So how do you try to insure good collaboration?

Justin Erenkrantz:  So the main center point of all of the collaborative software development that we do in Apache is the mailing list. That’s where pretty much everything happens. As one of the guys mentioned before, the maxim has been: "If it didn’t happen on the mailing list, it didn’t happen." And that generally tends to be true.

Basically, if you follow the mailing list for a particular project, then we’re expecting that you should know what’s going on within the project. Within that, those are pretty much all public lists. Everybody can subscribe, even people who are committers, people who are just users, people who just work on another project that may consume the Web server or maybe PHP modules or something like that.

It’s pretty much an open forum. Anybody can voice their ideas. Generally though, just the way things work, most of the traffic tends to be from the core developers who are active at that time. The traffic patterns of the list change, and you get an idea of how much discussion. You generally see peaks and valleys for the mailing list discussion. Things get really heated or things are just chugging along and there’s not much traffic on the list.

So that’s where all the discussion should happen. And then of course there is the source code repository. And in Apache we always had the thought of having shared repository. A lot of projects now are starting to ‑‑ Git and Mercurial and all of these distributed version control systems to be centralized version control systems.

And that’s something that, within Apache, that goes against what our thoughts are, because we want to all be agreeing on, "This is the Apache version."

Scott:  Sure, no problem.

Justin:  So there’s no leader within the ASF. There’s no person, if you look at say, Linux and you say, "This is Linux’ tree," or this is Andrew’s tree or this is Alan’s tree. Instead there is just the Apache tree. So there’s really no concept of, "This is Justin’s tree," or someone else.

Scott:  Is the feeling then that essentially there should just be intensive discussion before something gets checked in? So when you’ve reached a point of consensus, it should get checked in rather than the way other projects work where different things get checked into different people’s trees. And then when it’s time to build a release, you have to pull stuff from these different sources to figure out what’s going to be in the release and what isn’t.

Justin:  Right. So, generally, what tends to happen is there are two states a tree can be in within Apache. One of them is "commit then review." And then there is "review then commit." And you’ll see some projects differ on particular trees.

For example for the HTTP Server, the trunk ‑‑ which is the main development ‑‑ is usually always under the "commit then review," which basically means that anybody who has commit access can feel free to go and make changes and they basically have the benefit of the doubt that the change is going to be good. And there’s generally an implied threshold: that if you’re going to make a really big change, go discuss it on the list. But if it’s a minor change or adding a little feature, that’s probably not going to be controversial, go ahead and commit that to trunk.

But for our stable releases, generally things that have already been released and we’re doing maintenance on those, are under the "review then commit" model. So that’s going to be RTC, and that means that any change to those trees has to be pre‑approved. That means you need to get three binding votes from other committers to say, "Yes, this is good change and no one has vetoed it." Technically you would use a file called STATUS, just a plain text file that some projects will use, that basically tracks all of the things that are under discussion to be back‑ported or added into this tree.

Scott:  Got you. Like any open source project, and this one is democratic, there is a vote to commit. Well, there is a vote if it is a release product. If it’s not a release product and people later decide it wasn’t a good thing to do, it can be reverted out.

And you mentioned before that part of your governance is that out of these 60‑something maintainers, one of them can essentially veto a change if they want to. So talk a little bit about how conflict resolution usually works. For example, when you have people who are—and I understand it doesn’t happen often—adamant one way and another person who’s adamant a different way. How do you see it play out that those eventually get resolved and things move forward?

Justin:  Generally what happens is, like before, the veto just tends to be a last resort. So let’s just say that someone makes a change to the trunk and I don’t like it. I might just say, "You know, we should talk about this change," or, "Here’s the problem with this." Generally, the person who has committed that says, "Oh, yeah. Here’s why I did it this way," and comes up with an explanation, and then through a process on the mailing list, figures out and resolves those conflicts.

That’s what you tend to see happen. And it’s all tends to be, for the most part, "I’m sorry, I forgot about that particular corner case or that." And everything tends to get resolved very naturally.

The veto tends to be when someone says, "No. I have to do it this way," and someone else says, "No, that’s wrong." That’s basically when the community is at risk of breaking down. But generally it doesn’t get to that point. Everybody is, "We’re all going to go in this direction; this is the right direction for us to go in. We want to add this feature. Let’s work through whatever issue you may have about this particular commit."

Scott:  One last thing before we move on to the next point. Other than the fact that everybody has their own private tree, is there anything else about the collaborative nature that you think is somewhat unique to Apache?

Justin:  We tend to do a roll call before the release. So at this point there’s been a review of everything. But then let’s just say that I want to release the next Apache 2.4 or whatever. Then basically what I will do as a release manager is say, "OK, I’m marking this as 2.4," and I produce all the artifacts. I produce the tarballs, [inaudible]generated files, whatever. And then I send it to the list and say, "Hey, is everybody happy with that?" And then that goes to a voting process.

There’s review at all stages, but there’s also a review at the point where you do a release, and you have to get, at least, three people to approve a release. One thing that’s different is that it’s not possible to veto a release. It’s strictly majority rule on the release.

Scott:  So in other words, there is veto capability on the individual check‑ins. But, as you said, when it comes down to doing a release it is a majority rule vote.

Justin:  Yes. You’ll tend to see someone voting ‘no’ on a release if it doesn’t work on Linux or something, and generally you’ll see it get stalled and it then gets fixed. But there have been a couple of cases when we did the release even though we knew that it didn’t work on a particular platform, and so we made a release note. But the veto does not apply to releases.

Scott:  Interesting. So moving on to licensing, one of the key things with Apache is the commercial‑friendly standard license. Talk a little bit about what that means.

Justin:  Basically within Apache, we like to have a big tent where everybody can come in and play with us. We think that the community that we developed within Apache is going to be the motivation for you to stay involved. For example, within the HTTP Server community, you have all these experts and Web servers, and if you’re part of this community, you get the benefits from them. And so there’s an incentive to play within the community, so that I don’t have to hire five guys and do a whole team; I can leverage the other people within the community.

But in turn, all those people who are part of the community say, "Whatever you want to do with the code is fine. We’re not going to get hung up if you make it a commercial product or an open source project. We created it and it served our needs, and if it serves your needs, that’s great."

Scott:  What springs to mind are things like GPL. Am I seeing it right in that Apache is more commercial‑friendly than GPL, V2 or V3?

Justin:  There are companies built around GPL licensed software. But what we tend to see are two classes of GPL products: In one, there is a real community, maybe within Linux, and they’re all happy to make all their changes available to everybody, and that’s a very good community. The other community you tend to see has a single stakeholder that has a prevailing interest in the GPL product, and they basically have an unfair share.

You can see this with some of the GPL projects that require copyright assignment. In order to participate, you license your changes in the GPL and you have to give a copyright assignment to the principal stakeholder. Now they are then free to release the commercial closed source based on your work because they have the copyright or whatever legal mechanism. There is an imbalance there when you look at those two.

Generally when you think about the GPL, you’re divided, with broad strokes, into those two groups. This is a real community but the other groups are aware and want to be clear about which one has a dominant role, and that’s one thing within Apache we don’t like to see. As our projects go through incubation and get added to the Foundation, one of the things we do is make sure that the community is diverse. In fact, there is not a single dominant stakeholder that can direct the project in any untoward way.

Scott:  Right. I can make changes to it, distribute it as part of a commercial product, and I would not be required to contribute those changes back to Apache. But under a GPL license, any modifications made require you to make the source code available. You cannot have closed source proprietary extensions or modifications of it. Any modifications you make, you have to open source and it has to be under the same license. So that’s the key differentiator?

Justin:  Yeah, our philosophy is that the community is what’s going to bring you and keep you there, and that’s why you’re going to stick around. If you released a commercial product around one of our projects it’s going to be to your benefit, to basically keep your commercial project as close to whatever we’re releasing.

Scott:  Right.

Justin:  You can pick up all the bug fixes and whatever improvements; you get those as a free rider. But in a sense, you are contributing whatever changes you’re making voluntarily back into the greater community.

Scott:  Yeah, that makes sense. Do you have examples of companies that have used different Apache projects because of the commercial‑friendly licensing, where they probably wouldn’t have it if the license weren’t so commercial‑friendly? Is that a topic that comes up?

Justin:  Absolutely. You see companies like IBM that release their versions of the Apache Web server or Geronimo under different names, but in the core, they are Apache projects. We’ll see that even with smaller companies such as Covalent that does commercial support. Basically, they added in a couple of extra things that provide support to their users.

One thing that the Apache community really does not focus on providing is 7/24 support. Covalent goes in with their business model and provides the support and training around these particular Apache projects. You will see businesses like JBoss using Tomcat. So you see all of these commercial companies using things that are Apache projects under the covers.

Scott:  Right, so that freedom has led people to be a lot more creative about how they structure their business. They have a lot more options in how they participate with the different Apache projects, how they contribute back and how they structure their own products. What is the relationship between the Apache Software Foundation and the Free Software Foundation? Is there any or are those fairly separate endeavors?

Justin:  There is no formal relationship. I’ve never had a conversation with Richard Stallman, but I’ve had conversations with Bradley Kuhn who used to be, at that time, the Executive Director of the Free Software Foundation. So there’s an informal get‑together of foundations to compare notes, and that’s generally a very good thing. How do we keep our ears open to what Mozilla’s doing? If they’re doing this new technique, then we can give them a call and ask, "What are you doing? We’d like to follow on it."

One thing we’ve been doing with the Eclipse is a joint conference. There’s going to be a conference in Asia that’s now scheduled for 2008. So it’s a way for us to get the communities talking to each other.

Scott:  Sure. So basically, if I can summarize, you guys get together around joint events and joint things where it makes sense. You share information because you’re all part of the open source community. Philosophically you may agree to disagree in terms of the details of licensing, commercial friendly, and that kind of stuff.

Justin:  Well and you ought to be using more…projects have different circumstances. Apache‑‑we have a very vocal membership and we have this and you compare that to let’s say the Mozilla which has a completely different governance structure. But if you look at Brian Behlendorf, he’s been on the Mozilla board for a very long time and he was one of the founders of Apache.

Scott:  Gotcha.

Justin:  So you have this intermingling of the communities. So someone like Brian Behlendorf who was brought in through the Mozilla and says here’s how we did things within Apache, and here’s his expertise and his experience that he got, he can share that with the other people within Mozilla.

Scott:  Gotcha, gotcha.

Sean:  Let me refer to one of the other tenants of the Apache way. I’m curious about this just because I was thinking about the conversation we were having. To state the obvious, you’re focused on producing software. I notice that you have a security committee that—if I’m reading it right and for lack of a better phrase—provides a service to all of the projects that are part of the foundation. And it looks like those projects can turn to the security committee and ask security related questions or possibly look for guidance from them, regardless of whether they’re Tomcat or some other piece of the foundation? Is that accurate or is that not accurate?

Justin:  It’s somewhat so. We have a security team, which I believe is currently a Board committee. But basically what they’re responsible for doing is ensuring our security at Apache.org mailing address gets responded to. And these are generally people who are very security savvy.

But there tends to be some people from Tomcat, from the HTTP Web server, from a higher profile project on this internal mailing list. So let’s say that, to give you an example, let’s say there was a security vulnerability in Derby and they could parse to those reps and say, "Hey, we have a security vulnerability. What do we do?" And so there’s expertise and, "OK. Here’s what you do. Here is your administrative contact. Make sure your mailing list… Go talk to…" Kind of a shared resource. But we’re not getting the focus on producing the fixes for the project but it’ll be "OK, here’s the responsible disclosure policy and an attribution policy." So that’s generally what their role is.

Sean:   Are they providing fairly prescriptive guidance but just not down to the ‘I’m going to change you’re code’ level because they don’t know the individual projects at that level? Would that group essentially be the center for discussions around a security development lifecycle for the Apache Software Foundation? And an attempt to pull those best practices together?

Justin:  Yeah. I think basically our project concern…we have something. What’s the process? What do we do? And that’s as an advisory role. OK, here’s the process and the procedures to follow.

Sean:  But it’s purely advisory, right? I mean one of the projects where they feel that their code is "secure enough", or they’ve looked at it long enough or they feel that they’ve handled it.  Then the advisory committee comes back and says, "Well, we really think you could take a look at this again." That’s where the communication would stop and it would be up to the individual project whether they want to take that under advisement or not.

Justin:  Yeah. I think so. I think record security, can you maybe at that point write back to your original reporter and say, "We looked at it and we don’t feel there’s a security vulnerability here." That may be…that has happened where we look at things and we say, "No. This is not an issue." But generally, really the security team is more of a reactive. So they’re not proactively performing security analysis on our code or anything like that.

Scott:  I just want to clarify. It sounds like they have a little bit of an all‑up policy for somebody sending an email to that address; somebody reports what they perceive as vulnerability or reports some kind of issue. They do a little air traffic control. They route it to the project.

Justin:  Exactly, exactly.

Scott:  There’s a general process that the different projects would follow. Basically that sort of happens and that’s one of the things that the security group advises the other projects on. Well this is generally a ‘way we do it’ sort of thing.

Sean:  Let’s go to a different piece of the Apache way, the emphasis on a technical‑based interaction. One of the things that I find fascinating about open‑source projects is the way that they exorcise community members that maybe aren’t following those rules.

[laughter]

Sean:  Because we got some interesting responses when we talked to people about it. It’s like, fine. We understand that everybody is an adult. We understand that everybody will try to handle themselves in an appropriate manner. But if anybody’s worked on a software project they know that not everybody does, right? So…

[laughter]

Sean:  Considering that you can learn a lot from a story…if you’ve got a story or two about either fully pulling the ejection handle on somebody that would be interesting to hear? Or a scenario where it just took serious counseling to get somebody pointed in the right direction.

I would be curious to see how you guys handle that. You obviously have procedures in place. But at times you have to go beyond those with some amount of intervention and I’m just curious how that played out.

Justin:  Yeah. So there’s one case that comes into mind but I’m trying to reserve the right to figure out how much of this has been disclosed.

Sean:  Yeah, Sure

Justin:  So I’ll tell the story and then leave people’s names out of it but I have to go back and see how much of this has been told. So recently within one of our larger, well‑known projects, there was a bout, to use the word, between two committers. And they basically ended up vetoing each other on everything. It’s like no, no, no and tempers got flared. And it got into a very unhealthy situation. They’re two very strong‑willed individuals.

And basically what happened is the PMC, so it was the PMC responsible for this particular activity, basically had to step in and say, "OK; we need to come up with some policy or come up with some new rules to get everybody back to ground zero. So it wasn’t a matter of ejecting anybody. It was never really an option that…basically what happened was they said, "Here are the ground rules. Here is…if you’re going to go do this you have to follow this set of rules. If you’re going to go do that, you’ve got to go follow this set of rules." Basically the community agreed to say we’re going to go and we’re going to voluntarily adopt these rules. But as a settlement process; lots of flames and a lot of innocent people getting…

Sean:  Right.

Justin:  …accused of things and that process. The other one that’s in our history was a project called Avalon. And this is one that’s definitely well known so this won’t be any issue about this one. Avalon was a container framework. And what happened was two individuals just did not get along. And they were ending up in what we would call a commit war, where they would basically be reverting each other’s changes as soon as they came in. And it is just this whole really poisonous environment and basically in that case, the community wasn’t able to deal with it. And so basically what happened there was they fractured.

And so that one of the individuals went off and he took his code and you know, we wished him luck and said have a nice life and he went off and then it was kind of some other people came along and they did a project called Excalibur, which was basically the remnants of this whole Avalon project. You will just see if you look at the mailing list traffic, you will just see this giant peak and then this sudden nothingness because the project got shut down because no one could play well with each other.

Sean:  Right.

Justin:  By the way what it is interesting is some of the veterans of Avalon, they really got involved in the greater Apache community, one of our new directors this year. He was in the middle of all of this, but during that whole experience, he was one of the people trying to keep things level and stuck around and this year he got elected to the board of directors.

Sean:  Since Apache is a large foundation, I’m curious about a different point. If you were a closed source company and you feel you are short on testers or short of security experts, you simply go to HR and put out a requisition and hopefully some good folks come back.

So has the foundation had to answer requests from projects where they say, "OK, look, we think we are geniuses on nine out of 10 of the things we need to do, but this one thing we really need people to help." How does the foundation help with staffing up a project in this type of case?

Justin:  It is more bottom‑up than that I think in the sense of the culture that we have. You will see some overlap between the HTTP Server committers and the Tomcat committers because you will see that ‑‑ sometimes the Tomcat committers came over and they say, "Hey, we need some help with HTTP." Well lucky us, we have some of the world’s foremost experts in HTTP server, and that basically got them within the communities.

So I think our community’s diverse enough to, "Hey I am looking for a person who knows SQL." OK, I am going to go on the Derby mailing list and say, "Hey, I need some help with SQL" or for something for build systems, I’ll go to Ant community. And there will be some of the people who are the foremost experts in that. That’s actually one thing of having such a large diverse community is that you can pretty much find someone who understands something about something somewhere within the Foundation.

Sean:   I figure that’s one of the advantages. You’ve got a massive talent base but at the same time it is segmented into the project, so the Foundation can help orient a little bit of that knowledge of where the talent base is.

Justin:  Yeah and as I said if you look out the social graph and it is a weird mix of people who are committers on Cocoon, maybe committers on Mina, then maybe on Gump and so you will see that developers themselves, the committers aren’t necessarily staying in their silo, there are some who do, but there are also just as many who will go to other communities and work on other projects.

Sean:   Well I have couple more questions but Scott can go ahead. I want to give you an opportunity to jump back in.

Scott:  So talk a little bit about standards because one of the other tenets or pieces of philosophy is faithful implementation of standards. Talk a little bit about what that means for Apache?

Justin:  Right. So this was initially when we started off there was HTTP and there was the IETF standards and that was when you have editor of the HTTP standard is one of the people behind the code base, there is the knowledge is going both ways in a sense is that we are that able to influence the standards process. But at the same point we also have some of us involved with the standard process and feeding those changes back and to the development and supporting those standards.

But since then, the initial, you see our participation within some of the key web server specifications, then probably most importantly our participation in the Java Community Process and that is, as you know, we have so many Java projects and that so many of our projects are implementing some JSR specification and our involvement within the JCP has been to ensure that we can implement the specifications and we have projects have representation on these expert groups that device these standards.

Scott:  Right. So it isn’t just like the standard shows up and then you figure out how to implement it. There is this two‑way street where you are shaping the standard because you guys have such a big, real world implementation. And meanwhile, the standard is telling what you guys do because they have their own stakeholders, but they are considering your recommendations and you want to conform to what they eventually approve.

Justin:  Correct.

Scott:  Yeah, I don’t know. I don’t have anything else specific. Sean, do you?

Sean:  No, not right now. I think this led us into some new stuff and from our end, we really enjoyed chatting about it. Justin, do you have anything you’d want to add or things you think we should address overall based on the theme of where we were going?

Justin:  No, I mean you did a good job of asking me the questions.

Scott:  I guess there is one final question. When somebody is starting or has an open source project, they can pick the license they want  they can do what they want for their community, things like that. What makes people want to be, in your mind, an Apache project? What is the draw I guess?

Justin:  Well, I think from my perspective, the draw is we handle a lot of the mundane governing structures and all this and the infrastructure and the licenses. And that is all essentially managed and I think you see a lot of open source projects like, "Oh we need to go get a foundation."

And that’s a lot of overhead, there is a lot of overhead to create a corporation, handle donations and handle essential infrastructure and that is what our goal at the Foundation at the broadest level is to provide support. So that these people who are working on all these different projects all they have to worry about is doing code. They don’t have to worry about, "Oh I need to go and buy a new server, how we are doing to deal with this donation or this tax policy." We try to deal with all of that. So I think there is a critical mass that works in our favor.

Scott:  Right, right and let them focus on the piece of it that they really enjoy, which is whatever this project is that they have come up with. They have a passion, like you said, for not having to worry about all the housekeeping stuff.

Bookmark this:

Interviewees:Blaine Wastell and Glenn Block.

In this interview with Blaine Wastell and Glenn Block of the Patterns and Practices Group at Microsoft we asked them about:

• About the Patterns and Practices Group
• What makes them different from a typical product development group
• The software development methodology they use
• Some of the things that Patterns and Practices group has produced for developers.
• The role of CodePlex for their group.

Blaine:   I’m Blaine Wastell. I’m a program manager in patterns & practices, and I’ve been in this role for a little over four years now. Most recently my focus has been in what we call the "client UX program". It’s about providing guidance to customers on developing both smart client and web client applications.

For 12 years before Microsoft I was out in the consulting world helping business customers develop enterprise line of business applications, mainly Web applications.

Glenn Block:  I’m Glenn Block. I’m a technical product planner in patterns & practices. I am a newcomer to the client team. I’ve been in p & p for about 6 months, and with Microsoft for about 2 years. Prior to p & p, I worked in Microsoft Learning and was responsible for building online developer training.

For the 10 years prior to Microsoft I was a software engineer and architect. I worked in various startups as well as a few enterprises. I’ve had a touch of consulting, but it’s mostly been really about building both packaged apps and internal apps ‑‑ both on Web and on Windows. A significant part of my background has been in developing software frameworks for those kinds of applications.

And as the technical product planner on the client team, I’m responsible for the overall strategy, understanding what customers want, and setting expectations on what kind of guidance we’re going to deliver.

Scott:  Talk a little about patterns & practices.

Blaine:  We provide "building blocks" that customers use for line of business application. Microsoft has a number of development tools and technologies which include the .NET framework, Visual Studio, SQL Server, the core operating system, etc. We help customers by providing guidance, in multiple forms, on using those technologies to build line of business applications.

Glenn:  The practices part of our name is focused on the latter, where we give guidance, best practices, and design patterns, to educate developers on known solutions so they’re not reinventing the wheel.

Scott:   I’ve used like the Enterprise Library that patterns & practices produced, and it’s really more of a framework than building blocks. Are things like the Enterprise Library are designed to be best‑practice reference architectures, or are they drop-in frameworks designed to give a productivity boost?

Blaine:  Enterprise Library, and the application blocks, are part of what we deliver. We also have written guidance for areas like security and performance.

We also have another type of deliverable called "software factories" that tie all the pieces together. They have the framework, but they also include the common patterns for a specific area. They include automation; they include what we call a "reference implementation" ‑‑ essentially a sample of how to implement proven practices using the frameworks.

Glenn:  When you talk about frameworks, at the end of the day, to us it’s all guidance. In the case of these frameworks, they are incarnations of the patterns we recommed. We think of them as an 80% solution meaning they may not provide all the answers. We write the code with the intention that people who are consuming these deliverable will rip that code up, and will look at how we’re doing stuff. It’s not simply about giving people a reusable toolkit, although there is an aspect of that.

The idea is, "Hey, look at what we’ve done. Rip out the pieces that you think are applicable for your scenario. If you’re writing something new, you can use this as a reference point and know that it incorporates well‑known patterns."

Scott:  What makes your team different from a Microsoft product team. How are you different from the teams building the .NET Framework, or SQL Server, for example.

Blaine:  We focus on what we call ‘customer connected engineering’. We start with an area that we see is challenging for customers, and then we will put together an advisory board that includes customers. As we design a solution, they’ll validate that our scope and approach is really hitting the top set of challenges that they have.

Glenn:  The big differentiator is that product groups build the platform, while the guidance we create uses the platform.

Product teams focus on taking the platform to the next level. We do guidance, as Blaine described, guidance on using the platform. We show you how to put the legos together in meaningful ways to solve particular problems that you’re facing.

In some cases, through the act of putting those things together, we may pull out some new reusable guidance that we call ‘application blocks’. But the block is not the end goal. The block is just one incarnation of the guidance. When I say ‘block’ , I’m really referring to a reusable library.

Scott:  Could you list some of the things that you’ve produced?

Blaine:  You mentioned the Enterprise Library, which has a number of blocks within it. There’s also the composite UI application block, which is a library that helps you create smart clients. There is also a set of libraries to help you create web clients. One is called the composite web application block. And we also have another one called the pageflow application block which addresses challenges around Page navigation. Blaine:  We’ve done a number of things in the web services area.

Glenn:  We have a Web service software factory; which is guidance on building web services. And we also have new guidance that we’re developing called ESB guidance which addresses utilizing BizTalk as an Enterprise Service Bus.

Scott:  Talk a little bit about the software development methodologies that you use.

Blaine:  Sure. We are an agile organization. We’re probably closer to XP. We start with the backlog that we prioritize. Glenn is essentially the proxy for the customer; the one that will prioritize the backlog for us. From that we do the standard iteration planning, and meetings. We do it once a week, and we have stand ups daily where we go through the status of the standard set of questions: What did we do yesterday, what do I plan on doing today, what blocking issues do I have.

At the end of an iteration, which is every week, we publish our results out on our CodePlex community site. Then customers can look at the CodePlex site and get an understanding of where we’re going. A lot of times they’ll say, "Hey, why are you doing this, because I have these other two sets of challenges you’re not covering," which is good. We’ll also have people say, "Hmmm. There’s a bug in line 35." It’s pretty amazing how closely customers sometimes watch us.

Glenn:  And in some cases even supply fixes.

Blaine:  Right. We’ll also do testing, development, and continuous integration. We use in test driven development, so we write the unit tests first, and then the code.

We’ll generally do pair programming. We’re also distributed team with team members located in Redmond, Buenos Aires and India. This sometimes makes it hard to always do pair programming. If we can’t do pair programming we will have somebody else do a code review before checking the code in. We do automated exception tests where they make sense. We do quite a bit of static analysis on the code that gets checked in.

Glenn:  How about code coverage?

Blaine:  We do code coverage. I have a blog entry that I just put out recently that talk about the quality checkpoints that we go through. On the quality assurance side, we’ll do things like proof and security testing, and globalization testing.

Scott:  You mentioned there’s also a community side of what you do. Talk a little about that.

Glenn:  There’s several different ways that we actually interact with the community and our customers. The simplest way is our blogs. A good portion of the team is heavily into blogging, and we try to give as much visibility as possible into what we’re up to.

Blaine mentioned that all of our stuff is on CodePlex. A big differentiator between the work that we do, and the product groups at Microsoft, is that you can get all the source code.

Also on CodePlex we have work item voting. When people come to us with feedback we say, "Great idea. Throw that on as a work item so the rest of the community can vote." And I can tell those work items absolutely determine what we do.

For example, AJAX support in the Web Client was one of the highest voted work items. As a result we’ve done about four months of work on the UI responsiveness and we have a few more to go. That’s a large investment and it’s based off the community feedback.

In some cases, there’s a work item that’s very broad, and we’ll dig in. For AJAX there’s a server side and client side aspect. When we asked which people were more interested in us investing in, people said, "Hey, we’re more interested in the server than in the client."

Another source is our direct interaction with customers. One of the things we do in p & p is bring a lot of customers in to spend time with the team. That gives us an amazing opportunity to directly interact with them, find out what they’re building, find out how our guidance in mapping against their needs, and find out what the gaps are.

We just had a customer that was here for an entire week. We do that with a number of clients and aggregate the information and learning which we inject back into our efforts.

We also do some on‑site, where we go and visit customers. Since I’ve joined, I’ve visited two customers ‑‑ one in the U.S. and one in the U.K. ‑‑ spending a week with their teams.

Further, we have general blog that generates feedback that we get. We have email feedback that we get. That covers the external customers. But p &amp p also has internal customers. Who are those? These are the product groups.

Lately we’ve been we’ve been working very heavily with Scott Guthrie’s org. We’re doing this is because we want to understand what kinds of challenges the platform is going to be addressing and how it will be addressing itWe want sure that whatever guidance we’re giving is in line with this direction and not opposed to it.

And we also key in to their demand for deep content. They say, "Hey, there’s nobody who’s really telling people how to put these pieces together." We’re also getting very involved with OBA [Office Business Applications]. We’ve had the OBA team coming to us saying, "Look, we need some solid guidance that will give customers architectural guidance on how to develop an OBA application."

Sometimes all these inputs are in conflict with each other and we have to look at the external feedback, the internal feedback, the industry trends, open source, and even our competitors, and decide what we’re going to deliver.

Finally we have our Customer Advisory Boards. This is a key aspect of our customer‑connected engineering strategy. For every one of our deliverables, we have an advisory board of individuals that are in the industry, that are or will be using this guidance. For the Web Client Software Factory, every two weeks for about the past four months, we have an hour meeting where we present to the advisory board what we’re doing, and we get their feedback.

The advisory board is a group that’s able to look at what we’re doing and raise flags and tell us, "Hey, you’re going in the wrong way." Or they may be saying, "You guys are missing this big opportunity over here that really affects all of us." And when we build this advisory board, we select a group of experts across the industry in both small and large companies, some located in the U.S., some located in Europe, some located in Australia. We try to get a really good mix.

They have different needs. They give us a proper perspective on the overall landscape so that we can make those intelligent decisions. I think that covers it. As you can imagine, it’s not an easy task to drill through all of that input and decide which way to go.

Scott:  Right now, the patterns & practices code is released under a custom license. It’s not under the Microsoft Reference license or the Microsoft Public License — which used to be the Microsoft Permissive License.

Blaine:  Everything is migrating to the MSPL.

Glenn:  Yeah. It’s essentially been MSPL without the name. It is going to be MSPL.

Scott:  OK, so the MS Public License.

Scott:  The thing that’s really interesting that now that’s an OSI‑approved license. In the past, Microsoft would say, "We put the source out there. It’s open source," but to a lot of people in the industry…

Blaine:  It wasn’t really.

Scott:  Yeah. To them, unless it’s an OSI license, they don’t know if it’s really open source the way the OSI defines it. So now it is. That leads to the question of, are you now starting to do some of the things that a traditional open source project would do? You’ve got builds. You’ve got bug tracking and issue tracking that people can participate in.

Do you ever envision a day where you’d be taking community code submissions?

Glenn:  We’ve taken steps in that direction with what we call our Contrib community.

Before I worked at Microsoft, I used to work with a bunch of different open source products. I worked with NAnt and NUnit. NAnt has what’s called a NAnt Contrib Project. The NAnt Contrib project is where developers who have the source, are extending NAnt. They’re building plug‑ins, etc., because a lot of these products have a plug‑in model. They want to have a way of driving that back in so that the community can benefit from those enhancements or extensions.

We decided about six months ago to follow the same model and we created a set of Contrib communities. Actually, we didn’t create them, and I think it’s important that we didn’t create them. We worked with the community. For example, with Smart Client we engaged with people like Kent Boogaart, who created WPF CAB, which was his own extension to CAB that would allow you to build WPF applications with CAB.

We also worked with people like Bil Simser, who’s a known MVP who developed Smart Client apps. We worked with a whole group of people, Chris Holmes is another. We brought them together and said, "Hey, we want to make you guys as successful as possible and enable you to take our deliverables, extend them how you want, without us telling you which way to go, similar to the way you can with other open-source projects. We’ll provide a place for you to do that, and we will also work with you going forward. We’ll also hear your concerns and consider how we can fold that into the product."

We do also work with external resources as part of our core guidance deliverables. We have our advisory boards which I mentioned which participate in the design and give us feedback. We have our CodePlex Workitems where the community submits ideas that influence the design. And we even have members of the community (partners, SMEs, etc) that write the codebase itself.

Scott:  Aren’t there models where other people have dealt with this? You take a look at something like the Linux kernel, right? Nobody really owns it. It’s an open source project, it’s covered under an open source license. IBM is a contributor to it. And if stuff gets found in there that infringes on somebody else’s intellectual property, a lot of times the community, or IBM, replaces that with code that doesn’t.

I realize it’s a gradual process, and Microsoft is moving towards more community involvement, but it seems like down the road there may be ways to structure this to where Microsoft is a chief contributor to these projects, but every committer may not necessarily be a Microsoft employee.

Glenn :  In the Linux kernel, yes you do have community participants, but it is not a free-for-all where anyone just comes along and checks in code. It’s a regulated process with a core set of contributors. We operate in a similar fashion. We do have externals that contribute, but there is a process around those contributions.

Scott:  Another thing that’s interesting, when you’re under an OSI license somebody could fork your code.

Glenn:  Yep. And I just want to make clear that we don’t own the Contrib project. We’re not claiming any responsibility for the kind of code that goes into that. We might suggest stuff, like, "What do you guys think about doing this or doing that," but it’s completely up to the contributors what they want to do.

It is the model that you described. We may pour some stuff into Contrib, and we’ve talked about doing that, but essentially the people that own it is the community.

Scott:  Right, right. And those different Contrib projects are essentially forks of your code, and because…

Glenn:  That’s exactly what they are. In some cases, complete rewrites. For example, if you look at the EntLib Contrib, there’s actually an EntLib refactored project where somebody who works at Microsoft ‑‑ Scott Densmore, who helped write a lot of our deliverables ‑‑ is actually ripping apart EntLib and providing a new version.

Similar kind of things are happening in the CAB space. We just recently shipped another version of Smart Client Contrib that actually contains updates to our Updater Block, to get it to work on Vista. This is completely being driven by the community and it’s taking things to places where we don’t have the resources to take them.

Scott:  That’s what I was wondering. Maybe down the road, the only thing that exists is the Contrib space, and people in Microsoft are contributors just like anybody else.

Glenn:  That’s certainly a possibility that we would be open to, but it’s not really about us and what we want, it’s about the customers. Many customers that have serious concerns about using community contributions that might contain code that violate IP. In the case of patterns & practices deliverables because it’s from Microsoft, they have less concern. Other customers however are willing to take that to take that leap of faith. These are the same customers that also use open-source deliverables like NHibernate, Log4Net, Castle Windsor etc.

Scott:  To me it’s really fascinating how the different models are shaking out. We also interview CIOs for things outside of "How Software is Built" and they’re very fond of saying, " I want one throat to choke. I want to know that I’ve got support. I want to know that I’ve got a company that’s standing behind it. That’s really important to me. It’s not about the cost of the software, it’s not even about free as in freedom. I just need stuff that works, and I need someone to hold accountable."

And then on the other hand, on the other side of it, you’ve got open source, right? Where you don’t seem to have those same assurances, but on the other hand the software really does hit the mark. There are people who are experts in their domain. There are people who build really interesting things that a "big company" wouldn’t necessarily think of. There are people who do some innovative things, or start with someone else’s idea and take it in a very different direction.

There’s benefits in all of that. So it’s curious to me what the landscape is going to look like. You take a look at a company like RedHat. They’ve got this open source product, but they don’t really have control over the Linux kernel, or Apache, or the things their customer absolutely depend on. Still, they guarantee that they’ll provide support on top of it, enterprise‑level support.

You’re saying that if p & p’s work was full open source, and the only out that existed was Contrib, some of your customers would have trepidation because they would feel like there was nobody they could really point to who would guarantee the quality of the code, who could guarantee fixes, and those sorts of things.

I guess what I’m saying is it’s a very interesting spot that you sit in.

Glenn:  In a way, it can be the best of both worlds, with some caveats of course. What it means is there may be significant functionality that will exist because of the energy of the folks in the open source community, and that functionality will show up in Contrib but not in our core deliverables. There are other things that the Contrib community might do, that we might decide to incorporate into the core in the future.

Scott:  There’s another thing that makes your position interesting. On one hand you might get feedback from the product team saying, "Hey, we’re not going to have this in the next version. It’s going to be a pain point. It’s something that you guys could take a look at."

On the other hand, you might have a lot of people in the community asking for a feature, but because you’re part of Microsoft, you know that the product team is already building that into the next version of the product. You don’t want to invest in what the community is asking for, because it’s already being built, but you can’t tell anyone it’s being built. The product team still has their plans under NDA.

Glenn:  That certainly happens all the time. That’s certainly a challenge. But what I’m happy to say is that more and more we’ve been able to drive back the experiences customers have had with our guidance to the product teams, which has resulted in many cases in product offerings that come out based on that.

Scott:  When I looked at stuff you produced a long time ago, for example, the Data Access Application Block, it seems like it was almost destined to be obsolete. Eventually Table Adapters got built into the .NET Framework, and that provided the same functionality.

Blaine:  Our goal is to actually help migrate customers from what we produce, to the core product. That’s really our ultimate goal. If the core product addresses scenarios we’ve heard from customers, that’s a good thing.

Scott:  Yeah. Like I said, you guys are in a fascinating spot in terms of being an open source project, or open source projects, you’re really pretty unique it terms of where you sit. Can you share anything about the directions in which you’re going? You’re obviously moving towards more open. There’s certain constraints on that. Anything that people should look for over the next few months?

Glenn:  One of the things I think we’re moving on, and Blaine can touch on this, is bringing more of what we’re seeing to product groups. As an example ‑‑ and we’ve been trying to collaborate much more tightly internally ‑‑ one example is this new MVC framework. Want to talk a little bit about some of the work?

Blaine:  At patterns & practices we’ve been supporting, as Glenn talked about, many of the common patterns in the UI space. MVC [Model View Controller] is a common one. MVP [Model View Presenter] is another one. And since we’ve been in the space, we’ve been working with the ASP.NET team to help define the MVC work that’s coming out of there.

A big part of this is really Scott Guthrie and his team’s vision. We are bringing some of the feedback that we’ve heard from customers to the ASP team.

Glenn:  We’ve actually been working on the design with them. It’s really been a partnership. I think those kinds of things are efforts that you’re going to continue to see.

The other big thing that I think you’re going to see, is the fruits of a transition we’ve been going through internally within patterns & practices. We’re re-evaluating carefully at how we deliver guidance and what is the best method. , and we’re also looking at taking the principles of SCRUM that we apply within our teams and bringing it up to a higher management level with p & p. This includes having having a cross p & p product backlog that we can be very visible with customers about. We want to show, "These are the things we’re looking at. This is what’s a high priority. This is what’s not."

On the subject of how we deliver guidance, although many customers have nothing but good to say about our factories, there are other customers, that say they are too complex. They also tend to not be as applicable in in what we call "Brownfield scenarios" which are existing applications. Instead, they tend to be most useful when you are starting from scratch. Another thing we’ve seen is that industry-wide there has been a shift away away from comprehensive “do-it-all” type frameworks to a more pluggable services approach. Finally we’ve had a set of customers who tell us that the biggest value we can provide is to focus more on patterns themselves and how to use them (including samples and reference implementations) rather than framework type libraries that implement those patterns. We’ve heard thse customer concerns and are taking them seriously. We’re looking at how we can break things into smaller blocks. We don’t mean exactly like the earlier application blocks, but instead how to ship our deliverables in an incremental fashion that offers customers more options as to what they use.. For example if I want to use some aspect of the factory without having to take everything, then I can. And, we are also looking at patterns and how we can make them more a first class citizen in our guidance.

Around this, we’ve put a stake in the ground with the current release of the Web Client Software Factory and with our newly announced WPF Composite Client.

By getting these things to be smaller, we can ship more of them more frequently, we can make more intelligent decisions about what we ship and what we don’t ship. We can build guidance that has more applicability in both the brownfield scenarios and well as greenfield — which is new development. We also give customers more choices as to how they incorporate our guidance within their environments.

Ultimately, we can reduce the complexity in evaluating and using these deliverables to make them approachable to a wider audience.

As you can tell, we’ve certainly got a lot of work ahead of us.

Scott: Thanks for talking. You opened up an interesting subject at the end, and maybe we could talk again in the future about it.

Blaine:  You bet.

Bookmark this:

Interviewers: Scott Swigart , Richard Bowler

Interviewee: Matt Gibbs

In this interview with Matt, we asked him about:

• Approaching open and closed source SDL
• Development challenges for UI Framework and Services team
• Handling security within feature development
• Incorporating SDL into ASP.NET
• Launching multiple distributions with open source
• Testing against multiple operating systems
• Testing cycle for features
• Managing bug fixes and feature stability
• Following a development cycle
• Microsoft philosophy on product development

Scott Swigart: Hi, Matt. Before we start, go ahead and introduce yourself.

Matt:I’m Matt Gibbs, and I’m the Development Manager in the UI Framework and Services at Microsoft. My team is responsible for ASP.NET and AJAX as well as parts of Silverlight.

Scott Swigart: Thanks! We’ve been commissioned to look into the premise that open source and closed source SDLs [Software Development Lifecycles] are fundamentally different, and that the differences manifest themselves in the final products in tangible ways.

That’s not to say that one is better than the other – certainly both models have their strengths and weaknesses – but it’s really designed to get information out there for two audiences: one is software vendors who are looking at building products. One of their first choices is whether they are going to go with an open source or a closed source model, and we want to look at some of the ramifications of that choice.

The second audience is people bringing products into their organizations, and we want to uncover some of the basic expectations that they could have around open and closed source, or some of the characteristics of each model that might impact their choices.

Richard Bowler: You’ve been the Development Manager for ASP.NET for about a year, right?

Matt: Yes, about a year.

Richard: What would you say are the major development challenges with that product?

Matt: I think for us, the biggest challenge is that we have a large customer base, so as we innovate, we have to be very sensitive to backwards compatibility. We have a fairly complex platform, so innovation is challenging when you’re trying to make sure that existing applications people have built with it continue working.

Richard: It’s a pretty complex project, and it’s built on another complex project, which is the CLR; to co-develop with the CLR team, do you just take what they give you and work from there, or how does that work?

Matt: We work really closely with them. In fact, we meet regularly to go through code changes we’re making with members of the CLR team, so that we’re all in sync.

Richard: It sounds like you’ve been able to direct feedback to them, saying, “We really need this kind of capability, let’s fold it into the next release,” or something like that.

Matt: Yes. And they’re pretty sensitive to how their changes impact us. They run compatibility tests and stress tests against our scenarios, as well.

Richard: It seems like a huge challenge, when you consider the number of software products that Microsoft has on the market that interoperate. It seems like a huge challenge to keep them all working together.

Matt: For the framework we’ve actually held a pretty strict line. I am part of a committee where we review things that were classified as “potentially breaking a developer’s code,” and the default was that those weren’t allowed – that was kind of the highest priority. Things like “security fixes” would be allowed. Then there always needs to be a way to get back to an earlier version’s behavior, through a configuration parameter, for example.

Richard: Speaking about security, we talked with Michael Howard, and also with James Whittaker about the SDL. It looks like it’s really paying off for you guys. Were you involved in the development internally when the SDL was being instituted at Microsoft?

Matt: Yes, actually I worked with Michael Howard on IIS where we ran into a few security issues that helped highlight the need for a more structured approach.

Richard: I know what the benefits are, because we discussed it with Michael and James. But what were the main difficulties in shoehorning those new processes over the top of existing SDL processes?

Matt: I don’t think it really came down to process difficulties, as much as it came down to just helping developers recognize that the mindset they had been using needed to be front-loaded with security all along.
In our first round, everybody recognized that we needed to pay even more attention to security, and make it a higher priority. But then the realization was that it can’t be one phase in the development process. It really needs to be something that we pay attention to all along.

Richard: To put it another way, then, you had to get a wrench on the culture. You had to make security important in everyone’s mind.

Matt: Right, and it really had to be the case that everybody came to the same conclusion. You couldn’t just tell people it was important – they had to see what kind of impact it had to customers, and recognize it themselves.

Richard: Did you get much internal resistance from individual developers, or was there pretty much across-the-board buy-in?

Matt: It was pretty wholesale – everyone recognized it. I think particularly since I’ve been involved in web platforms, and we saw a couple of places where we’ve made mistakes, everybody recognized, “OK, we have to do something different, and be more practiced in this,” because otherwise we’re not going to get the right outcome.

Richard: Where was ASP.NET in the process of instituting the SDL? Was Version 1 out before the SDL really came to the fore?

Matt: No, when the process was launched we were already in development. It was during that cycle where we decided to take a break, and we went back and said, “We need to invest time in this process.” Everybody did some extra security training, and then we essentially stopped development, and did a pass for just security. We looked at every feature, all the code, looking for specific issues, just to make sure we had done all the right things.

Scott: One of the things that I found looking at the open source side of things, is just the sheer number of distributions that they have to ship for a product. MySQL for example, I think has 16 distributions, because it has to run on all of these different flavors of Linux, none of which the people working on MySQL really have a lot of control over. Those versions of Linux are all rev’ing independently of each other, so the underlying operating system that it’s dependent on is changing all the time. You’ve got a whole ton of applications that are built on top of MySQL, and are therefore dependent on it.

I can’t even really imagine exactly how to test that, and I know that in the open source world there are issues about, “This version of MySQL isn’t compatible with this version of WordPress, which isn’t compatible with this version of Linux.” So, they are always fighting these compatibility issues as things just rev independently of each other all the time.
Microsoft has some rather similar challenges, in the sense that IIS revs with the server. ASP.NET revs on its own timeline, which isn’t tied to the base operating system, and Atlas is also on its own timeline to some degree.

Talk a little bit about the kind of test matrices that Microsoft builds, and the amount of resources in terms of hardware and testers that come to bear when you guys are getting ready to release something, or while you’re developing something to ensure compatibility with all the versions of all the things you ship.

Matt: It’s probably more expensive than you might initially think, because in the same way that there are a bunch of different Linux distributions, we also still test against a lot of different supported versions of the operating system.

We’ll test against XP and Windows 2000 and Win2k3, with different levels of the service packs and with components installed; with Visual Studio installed, without Visual Studio installed, and then different flavors of IIS. We’re already testing with IIS 7 on Vista where we’ve got a lot of integration work going on. When we go to do compatibility checking and functional testing, we have a fairly large lab. And the run time, unfortunately, is on the order of weeks to really ensure compatibility.

Scott: What I’m envisioning is that somebody pushes a big button, and a hundred machines get configured with tests. It all comes up and runs, and results get cranked out — test, pass, or fail. Is it something kind of like that?

Matt: Yes, and they categorize tests into different levels of priorities so they can say, “We’re going to do a quick sanity check on something,” and that can take a couple of days. Or, they can say, “We want to run the gauntlet, and run everything we’ve got,” like before a service pack or something, and that will take much longer.
It covers everything from different language packs, to different service packs installed. One time-consuming thing is that errors that are false positives come up, and they need to be investigated. These might be just test issues, and that takes time as well, because you don’t want to ship with something that might be a problem, so everything has to be investigated.

Scott: I know that Microsoft’s got certain terminology around this. I’ve heard of something called the BVT, which I think stands for Build Verification Test. There are things that have to happen before something gets released as a CTP, Community Tech Preview. Can you talk a little bit about that, in terms of what the different levels of testing are, and what kind of things have to get verified on those different levels?

Matt: There’s a scaled approach. On a developer’s machine, we’ll have what we typically call Developer Tests.
The developer cranks out a lot of small standalone tests against the feature he’s building, against his APIs. He’ll run those every time he goes to check in code. We’ll aggregate those for our dev team, so that you’re running a fairly lengthy set of them. But, they still run within just a few minutes, so that you can continue making fast development progress. Then we’ll run a superset of those before we check in.

Scott: So, that’s to make sure that one developer’s change didn’t break some other developer’s code.

Matt: Yeah, that the basic functionality is intact. Then, the test team runs what they call a set of nightlies. They’ll run those on every daily build that comes out, they’ll kick off an automated set of nigthlies. They’re small enough in scope that they can run overnight, and any failures can be investigated fairly quickly. They are what they call “Pri0” test cases; and that would be mainstream use-cases.
Then, the superset of that is something that’s maybe ten times the number of test cases, and that’ll run for days to verify the full functionality. That’s literally thousands upon thousands of test cases.

Richard: So, your nightlies maintain a level of sanity about the ongoing nature of the code base – yesterday it was still good, today it’s still good, tomorrow somebody’s checking in something bad. Let’s back up and fix this so you never get too far out of whack.

Matt: Then, you may find something when you go through a full test pass. When the tester originally made the test plan, they may have said it’s a fringe scenario, so it was classified as a priority two test case. You might only run it before a release, and then something breaks.

Scott: Microsoft has a lot of people who blog, so there are obviously a lot of people on the product teams who are fairly transparent about where they are and what they’re doing. I think Scott Guthrie put up a big blog post where he said, “You know, there’s a point in time where we don’t necessarily fix every bug because we have to weigh the risk of impacting the stability of the product.”

Every time we touch a line of code, there’s a risk that we’re going to hurt the stability of the product, so there’s a point where we switch from telling a certain team what we’re fixing to actually having to get permission to fix certain things because of where we are at in the stabilization process. Am I characterizing that right?

Matt: Yeah, we’ve spent a lot of time mining our bug tracking database, and you look at data points like, “For X number of bugs, how many fixes actually cause some other problem?” The closer you get to shipping, the more likely you are to look at something and say, “This is really not a very important bug, and people are just not really going to see this. It’s good that we did the due diligence and found it, but this late in the game, you run the risk of breaking something more important.” You might just leave a bug in there and fix it when you have more time to shake out the problems.

Richard: I guess if you didn’t do that, you’d never ship.

Matt: Exactly. With new engineers, you’ll regularly see them hit this kind of frustrating point when they have a bug, they want to fix it, but it’s time to ship and they get told no. Their point of view is that it’s a problem that should be fixed, and they have to be told that they can fix it, but not until the next version

Scott: I have to admit that it’s odd to think that leaving bugs alone raises quality.

Richard: In my own process over time it seems like when we get down to the mode when we’re primarily testing an implemented product and engineers are working on bug fixes and not new features and behaviors. It seems like as you get closer and closer to when you need to ship it gets harder and harder to jump through the hoop of where a bug ought to be fixed. At some point it’s got to be an absolute showstopper to stop the shipment; it’s got to be a crash or something like that, otherwise you’re just not going to fix it until the next release.

Matt: Yeah, we get to the very end where we call it recall mode, where the question is whether you would actually take the product off the shelf in order to fix this issue. At this final point in the process, unless we’d recall the product, we don’t fix it.

Scott: Another thing that’s pretty interesting to me is the whole way that Microsoft does milestones. Again, my impression of it from the outside is that somebody will say, “OK, these are the features that we’re targeting for milestone one, for M1,” and so developers will go off and they’ll be working on those features. There will be a graph of the bug count that’s just climbing, climbing, climbing; the number of test cases written are just climbing, climbing, climbing.

At some point, it seems to switch over to where new features aren’t being written, it’s focusing on stabilization for the M1 milestone. So, there’s a trend down in the bug count. It seems like there are a lot of different variables that are being looked at in those stages; what’s happening to the bug count, what’s happening to the perf numbers…
Can you talk about how all that stuff comes together? Am I characterizing these milestones correctly? Is that sort of how it works?

Matt: In the past, that is how I would have characterized it – a sort of waterfall approach where we crank out a bunch of features and accumulate bugs, and the test team cranks out test cases, and then we play catch-up and try to satisfy all those test cases and bring the bug count down isn’t the most efficient. But we’ve actually shifted in our group to a little different approach that we think is working better.

Instead, we’ve gone to what we’re calling a feature crew model where the dev, test, and PM work together from the start on defining the design, the set of test cases, and the unit tests, and they drive for more quality up front before the feature even gets checked into the product. So, they’ll work in an isolated source code branch getting a feature to a higher level of quality and then get it into the product.

Richard: It sounds like you’re moving, at least to some extent, toward a more Agile model.

Matt: Yeah, it’s ended up being a hybrid of Agile development with features of SCRUM, but you couldn’t really call it one or the other. It still falls into a sort of waterfall effect where we have a milestone of a bunch of features we’re going to try to get done by a certain time. There is still this mentality of feature crews driving for a deadline, so we picked the pieces of a bunch of different methodologies that seem to work well for us in delivering commercial software, and we are trying to make them work together.

Scott: Now, I’ve heard a little bit about these feature crews. Correct me if I’m wrong, but that seems to have been developed after the ship of Visual Studio 2005. It came out of … I think they called it like, MQ or Milestone Quality, or something like that?

Matt: Yeah, it was kind of built around the milestone for quality.

Scott: So, some of the ramifications of that are that if there’s a feature you’re dependent on to build your feature, it might take you longer to get that. But, when Microsoft releases things like Community Tech Previews, they’re going to be more stable by default, because things were only checked into the source base that passed a higher quality bar. Is that right?

Matt: That’s the goal, and we were able to benefit from the approach with the ASP.NET AJAX release. We embraced the feature crew model when it was still ramping up for the rest of the division, and it worked out pretty well for us. We learned a few things we could do better.
We also learned that some of the dependencies became a little bit difficult to manage. If you truly had a feature that needed something else to be completed, then you ended up with a staggered set of feature work waiting for future builds. You needed an equal set of things that could be done early, in order for everybody to keep being productive.

Richard: It sounds like what you’re doing is trying to make sure that your interim quality is higher by testing feature-by-feature, and by enforcing a quality standard on individual pieces instead of waiting for the entire conglomeration. Is that a correct characterization?

Matt: Yes, hold the line on quality all along. Don’t accumulate debt in order to get more features done earlier.

Richard: That seems really smart to me. There’s an old saying you can’t test quality into a product, and that’s really true. It seems really smart to me. Was that inspired by the SDL’s focus on early process involvement of quality assurance?

Matt: Yes. Before, we almost had two instances of lifecycles. You had development working on one lifecycle model, and you had the quality assurance teams running along six to eight weeks behind the dev team.
But, that wasn’t really working efficiently because all we were doing was having one team try to play catch-up with the other team. The dev team was cranking out features, and the test team was finding problems with it. Then, the dev team was trying to catch-up on the problems that the test team found.

Scott: With Visual Studio 2005 when it came out, there were a lot of issues around stability and quality, and things like that.
It was easily the most complicated development environment that Microsoft had ever built, maybe that anybody had ever built. It had amazing levels of cross-product dependencies; there were big products of their own that were put in there, like Team System.

Some of the impression, at least from the outside, was that it didn’t all come together and stabilize at the end the way that everybody had hoped. So, is that correct? Do you feel like maybe the initial version wasn’t as stable as everybody hoped, and so some of these things have been done as a response?

Matt: That may be the conclusion from my team’s perspective, focusing on the platform. I think we probably feel like we’re in pretty good shape because we have criteria and goals that were met.
But, I know in the MQ timeframes a lot of people focused on a lot of little things that weren’t quite right that they wanted to fix up for the next release. That was primarily targeted at the development tools environment, not the platform.

Scott: You work in the open source world in a sense with Atlas, right? Because there are a set of controls for Atlas that are effectively open source in that Microsoft puts the source out there. They’re maybe not open source in the sense that anybody in the world can just modify the source and submit changes to it, but please talk a little bit about this foray into open source with some of the Atlas controls.

Matt: The AJAX Toolkit Project is a separate team around the corner from us, and it is shared source. They take code submissions from the public. They have released the really snappy UI controls that everybody uses on top of the Atlas part of the platform.
It’s been well received, and people are able to grab the source, and build it, and contribute. It’s a little different for us in comparison to what we’ve done before. We’ll see how much the community itself drives lots of new controls, or if it’s something where they like the fact that the controls are out there, but they don’t really actively contribute a lot.

Scott: Certainly a key benefit of open source is that the source is out there for you to learn from. I think that even taking the Windows Forms controls and the ASP.NET controls, certainly a lot of best practices might have been gleaned from being able to look at how Microsoft did what it did.

Matt: Well, we’ve done that with the Atlas release now, too. That’s a first for us, and I’m really proud of it. When we released ASP.NET AJAX, we released the source symbols for it so you can attach a debugger, and step right through our source code.
It’s not the same as taking source code in from the community, but we look at it as being very open and transparent. It’s one of those things where in the open source community, everybody says, “I have the source, so I can do what I want with it.” From my perspective, though, people tend to say that but not do it.

Scott: I think they call it the Berkeley Conundrum. It’s a variant on “if a tree falls in the woods, but nobody hears it, it doesn’t really make a sound.” If there’s so much open source code out there, and not really enough eyeballs to actually look at it, does it really matter that it’s open source?

One of the things that seems to me like a huge challenge is that inside of Microsoft you can control exactly what people get to do to. You can control the training that they have to have around security, and performance, and best practices. You can control the unit tests they write, and you can control the process that the code goes through.
When you take a look at open source, you just get code delivered to you and you don’t have any control, or any view into the process that was used to create that code.

From Microsoft’s perspective, you’re taking submissions from the community; how do you ensure things like security, performance, and reliability, when anybody could be submitting the code, and you don’t have control over how they got there?

Matt: I’m not intimately familiar with it, but my understanding is that the team has essentially taken on the burden, so far, of ensuring some of the security themselves by essentially laying our security process on top of the code before it goes out to the public. I also think there’s an element of caveat emptor.

But, because it’s coming from Microsoft, there has been a certain expectation of quality. At this point, because we have a dedicated team working on it, they’re in the code and it’s still going through quite a bit of the Microsoft process.
Also, at this point, you can’t just submit and have it go right out to CodePlex. You submit as a team, and your submission is then under review for inclusion on CodePlex.

Richard: What do you think are the advantages of closed source over open source?

Matt: In my mind, it comes down to probably two things. One, the customer knows they have somebody to go back to, that there is a company that they’re buying software from that is backing it up. Two, there is a full-time developer that is working on advancing the software.
Open source has done fantastic things with having a community effort drive things to do good software. But, the fundamental difference is that with proprietary software you have an entity that’s responsible and people employed to push things forward.

Richard: What about the development of that process itself — the mechanics of designing, implementing, and shipping a piece of software. Do you see advantages to the closed source model, and if so, what are they?

Matt: I think you may have a little bit more of a culture of discipline but I’m not sure that it’s really that much different. I think there’s so much passion from people in the open source community in what they’re building that it may be the same kind of thing. The same kind of respect somebody has for their full-time job.

Scott: There are two myths that I frequently encounter, one related to open source, and one related to proprietary software. The open source myth that I hear again and again is that many eyeballs looking at the code ensures security.

Michael Howard really threw down the gauntlet on that and said, “Show me the data, because there’s just no data to support the idea that you can make sure code is secure just by open sourcing it and letting people look at it.”
On the closed source side, one of the things that I often hear as being an advantage is this notion that there’s a company to go back to, there’s a company that stands behind it. My impression is, I don’t have any guarantee that a bug that I find will get fixed in the product. I don’t have any guarantee that a problem that I’m running into will get solved by the company that produced it. I can file a bug, I can call product support; maybe I’ll get a resolution, maybe I won’t.
Do you think that Microsoft or any other company really offers that level of guarantee?

Matt: In my experience having worked as a quick-fix engineer where we investigated customer reported problems from PSS, it was very serious. If you couldn’t find some way to workaround it, it needed to be a product fix. Now with that being said, if there was a reasonable workaround, then you’re back to the issue of whether it’s worth the risk to go patch product code.

There’s not a case I can think of where a customer had a serious problem and we didn’t either find them a workaround or patch the product in order to get them back on track. The open source community can’t guarantee every requested change will be made either.

Scott: I guess the only caveat to that might be that obviously, it’s not going to be instantaneous, in either the closed source or the open source world. I remember a problem I found, and the answer was it’ll be fixed in Service Pack 2, but Service Pack 2 didn’t come out for another six months.

Matt: Very few fixes in proprietary or open source development are instantaneous. More complex issues may take longer, and again there is the concern about compatibility. When something like Service Pack 2 ships, it’s had many hours of testing devoted to it to ensure that even small fixes are getting good coverage.

Scott: Clearly, from your perspective, that guarantee of supportability is real, but there are just certain kinds of physical constraints on how fast the resolution might be available. But in your experience if there is a critical problem, either the customer gets a workaround for it or Microsoft starts working on a fix?

Matt: That’s been my experience in ten years here.

Richard: We talked to one of the guys that works on the Linux kernel. He basically said what happens is that people turn code in to the mailing list, and then everybody takes a look at it. If they’re interested and they snipe at it, then the developer makes some changes. Then, ultimately it either gets adopted by the maintainer – the person who owns that section of the kernel – or it doesn’t.

So, at least in some sense there’s no process to enforce quality during implementation beyond peer review. It seems to me intuitively that that’s a disadvantage. But, I just wanted to get your take on it.

Matt: Of course we’re not working on the kernel, but I know that before a piece of code I write is going to get anywhere that it’s going to get exercised quite a bit. It seems like inherently that’s an advantage.

Scott: Matt, talk a little bit about motivation inside of Microsoft. How much is it sort of computer-generated, and how much is it, “Developer Number 13 had this much code checked in, this few defects, found this many bugs…?” How does it work in the “big house?”

Matt: For us it’s all about our developer customers. It’s about the level of enthusiasm and people’s feedback, and it all feeds off of that. We get our customer’s excitement, what they want to see, what they’re going to be able to build. They say to us, “If only you added this feature you could save me time, and here’s what I want to see next.”
The people here in Microsoft are completely driven by what our customers want to see next. We may be in a bit of a unique position because our customer is a developer, and developers get really excited about the platform.

Scott: I’ve heard from people who’ve interviewed at Microsoft come back and say, “There really is a sense that by working at Microsoft you have an opportunity to change the world.” I would guess that many of Microsoft’s products … if you take a look at Office, Visual Studio, Silverlight, and how you hope those products progress over time, most of these are big bets. In most of these cases, you are part of something that you hope is going to be really big and industry-changing.

Matt: Yes, it’s the kind of thing where I’ve had people come up to me at ASP.NET conferences and say, “You changed how I do my work. It’s been an incredible experience and I can’t wait to see what you guys do next.” It makes an impact on their life. It makes it fun.

It’s different than what Scott was driving at; the “how many defects” and “how many lines of code” kind of thing. We really run off of developer-driven enthusiasm. There’s an enthusiastic, sharp, hardworking group of people that are really focused on the customer and how to make better developer experiences.

Richard: Well, thanks very much for talking with us. I appreciate it.

Matt: Sure.

Bookmark this:

Interviewers: Scott Swigart

Interviewee: Rod Johnson

In this interview with Rod, CEO of Interface 21 and founder of the Spring framework, we asked him about:

• Development methodology of Spring framework
• Contributors to Spring
• Selection of code and features for the Spring framework
• Ways in which developers support open source
• Open source development cycle
• Importance of technical support
• Code aggregation in open source development
• Choosing open or closed source within small businesses

Scott Swigart: Rod, thanks for taking the time to chat. If you don’t mind, please take a moment to introduce yourself.

Rod Johnson: I am the founder of the Spring Framework, which has become the effective standard for enterprise Java development. We also have a spring.net edition. I am CEO of Interface 21, which is the company behind Spring. I’m also the author of several books on J2EE development.

Scott: Great! Talk to me a little bit about the development methodology of the Spring framework. How does that work? Every open-source project seems to have its flavor. Tell me how Spring is built. How do ideas go from conception to features to in production or in development to release?

Rod: OK. I think when people try to understand what open source is, they instantly think about code and that the software is created by a disparate, far-flung group of people who work in their spare time. In fact, to understand the open-source model you need to step back from code and look at the totality of things, which includes documentation, includes support for users, and includes the discussion of ideas.

What we see in the core of our software is that the vast majority of the code is written by a small group of exceptionally good people. I don’t believe there is any other way to produce exceptionally good software. We have known for many years about what Fred Brooks calls the The Mythical Man-Month.

Scott: Right.

Rod: Open source doesn’t change that. We know that if you throw more people at a problem you don’t get a better solution. And it doesn’t matter what approach to distributing the source of the software you use. You’re not going to change that fundamental fact. If you want exceptionally good software, you want a cohesive team of exceptionally good people and you really want that team to be as small as possible.

Scott: How is that team motivated to work on the project?

Rod: The core people who connect to Spring daily are employees. They are being paid to do this. That model works very, very well if you look at open source in terms of the enterprise Java space and also obviously of the Linux space. You see that the model is evolving towards paying the contributors who write the code. For example, I think 21 out of the top 25 committers on Linux are employees of large corporations. The majority of the committers on Spring are Interface 21 employees. Another good example of this is JBoss, the Java application server. The majority of the code there comes from employees of JBoss, who are now employees of Red Hat.

Scott: I want to get your insight on the Linux kernel. My understanding of the Linux kernel is that unless Linus likes a certain patch, it’s not going to make it into the kernel. And yet, the bulk of the development is done by people who work for IBM, people who work for Red Hat, Intel, AMD, whoever. Right? Corporations are doing the bulk of the work. And yet what makes it in or out of the kernel isn’t really under the control of a company. Do you have any insight into how that works?

Rod: I am not an expert on Linux kernel governance. From what I see, one thing that is very, very important to developing high quality software and maintaining that software in high quality for the long-term is consequences: consequences of success, consequences of screwing up.

In the case of Linux, what we have is not a single entity, but we have a number of companies (and in Linus’ case partly also an individual, obviously) and a number of parties that are really baked into the system because there are large consequences if they screw up.

I don’t think one entity needs to be responsible for a piece of software. I think you can end up with a kind of partnership, but there have to be consequences if something is screwed up. That is one thing that scares me about the rather naive volunteerism view of open source because what are the consequences to a volunteer? Sure, in many cases they may be professionals. They may feel as a sense of personal ownership that they cannot be satisfied unless something is as good as it possibly can be. But there ultimately are problems because they don’t necessarily suffer the same kind of consequences if something isn’t taken forward.

Scott: What people have told me is that there is sort of a developer Darwinism that happens. In other words, to get your code in you have to submit it to a mailing list. You have to submit it to what one person described as a “full contact code review” and in some mailing lists you’ll just simply flat out get ridiculed and won’t be taken seriously if your code isn’t to a certain level. So, it seems reputation based, I guess. You can’t really get code in until it’s passed code review and up the chain of maintainers. So, on one hand there isn’t a lot of consequence to writing bad code, except maybe it was a complete waste of time. Your inability to do it well was pointed out for the whole world to see. So, it seems there is a certain amount of pressure to sit up straight and code well, especially if you’re going to try to get stuff into very high profile, very core open-source products like the Apache, or Linux, or some things like that.

Rod: Yeah.

Scott: One of the things that you pointed out is that people who are going to use open source shouldn’t consider it a free ride. It isn’t just something that you should just leech off if you want the stuff that you’re dependent on to continue to thrive, grow, and be there for you, and continue to be enhanced down the road.

But, at the same time, most of the core contributors to open-source products are highly skilled people who are employed by a company, and this is what they do all day long. So, if I’m going to be a user of open source, how could I best contribute?

Rod: The first thing is that I would hate to give the impression that openness means nothing. There is one interesting characteristic, for example, about a piece of software like Spring, which is overwhelmingly written by one company, compared to closed source.

You have the ability to work up to being a committer of the software. Not many people will take that up. And we may choose to approach you to see if we can hire you if you do it. And that is a way of expanding and renewing the developer base.

So, it is open in the sense that there is opportunity for people to come to be committers on the software through the community based on merit. That is interesting because if I want to be a contributor on the Java-based IBM’s WebSphere, I’m sure I could add some value based on my background to WebSphere, but I can’t put my code in there. There is no way I can do it without becoming an IBM employee.

Scott: Right.

Rod: So that is a genuine difference. There is a true meritocracy. So, coming back to your question, which was “How do you pay? How do you contribute back?”

Scott: Yeah, how do you contribute?

Rod: Well, I think the first thing is to remember that although it’s very important that there is that meritocracy based on code there are several reasons that the majority of users are never going to contribute to that code. The majority of users are not going to modify and compile it. The majority of users are going to use it essentially as if it’s a shrink-wrap product.

I make all of our consultants just use the binaries. If they’re working with our product with customers, we don’t like them to be working with the source code and making changes to the source code of our product.

The interesting thing is that it simulates more accurately how end users use it. So, often we’ll go into customers where we’ll see that they just realized that you can mount the source code and if you want to step through the same work source code…

Scott: Right.

Rod: …so, it just doesn’t make sense for most end users to even consider modifying the software. Consider that Spring is 1.2 million lines of code. Some of that code does things like transaction management, the fundamental configuration of your application. Some of it executes in virtually every runtime call set. It doesn’t matter whether its open source or closed source, you really don’t mess with code that does that kind of thing unless you know what you’re doing.

Scott: Right.

Rod: The process of knowing what you’re doing…For example, in our transaction infrastructure I, myself, as the original co-designer, made a change about a year ago. I spent almost a day conceptualizing the current state of the code (which one of our other guys had modified) and figuring out exactly how I could do these [changes] in a safe way.

I am a strong enterprise Java developer with 10 years experience. I originally co-designed this module so I know how it is meant to work, yet it still took me almost a day to feel that I can safely make a change without risking breakage of a core part of the software.

Imagine if I were someone coming from outside who didn’t have 10 years of experience, hadn’t co-designed the module. What is the probability that I’m going to make that change in a robust way unless I spend maybe five days fully getting inside it?

Now, if I am an end user and I am writing software, for example for a bank, like many of our users, they don’t want to pay me for five days to figure out how the Spring framework works internally. That is irrational. There is little economic benefit for end users. There is a high probability of being economically irrational in modifying the open source software.

Another good example is Linux. I did a lot of C-programming early in my career. I can code in C. Am I ever going to look inside the kernel and change any of the kernel? This makes no sense, whatsoever. So, code contributions and changes are not likely. However, there are many other things I can do if I wish to give something back.

One thing I can do is to help other users of the software. Most open source projects have forums and mailing lists. Every time a developer answers a question for a user that developer is distracted and can’t write code to improve the product. Both things are important but no one can do both simultaneously.

One thing that has worked brilliantly with Spring is, especially early on, we made a big effort in the core development team to answer users’ questions and be really, really helpful. That set up a virtual circle, where users saw the Spring development team was really helpful when they were first starting up. Gradually they used the software more and became experienced. Then, they see other people asking questions and they think, “Hey I know the answer to that question. Maybe Ron doesn’t have to spend time answering that question. I’ll get in there and I’ll help the other user.” That’s a valuable way of giving back.

Another thing is promoting the software because open source software, like any other form of software, can only be successful if it is popular. For example, you like a particular piece of software. You wonder whether there is a user group for that software in your city. Maybe you have time to organize the user group. Maybe you can persuade your manager to provide coffee and a room if someone else is willing to organize it.

Another way that you can contribute is through issue tracking. If you see something that you think can be better, go and report it. An issue tracker is the enhancement request. That works very, very well with open source because that is one area where access to the code really matters. We see people who make enhancement suggestions with pretty deep knowledge of the code. We might get an enhancement suggestion saying, “By the way, if this class extended some other class it would allow the possibility to do X, Y and Z. Have you considered that?”

What happens is that one of our developers looks at it and says, “Oh God, I would never extend that class. Oh, but I see what he’s getting at. Oh, yeah! He’s right. It would allow X Y and Z. So, I wouldn’t do it that way but I can see that there is a lot of value in what he suggested.” The ability to look at the software is quite interesting from that point of view. It’s not necessarily so interesting in terms of modifying the software but it’s quite interesting in terms of the ability to provide suggestions with as little or as much context as you want.

Finally, and this is the way in which I think most financial organizations will end up giving back, go find someone who is contributing to or sustaining that software and buy a support contract. Not only is that going to sustain the software very effectively but also open source software needs the ability to have maintenance, especially as open source moves into the enterprise. What happens if a system that does trading goes down at 2 AM?

There’s not much point if one of your guys gets paged. He didn’t ultimately write the software that is involved in it and the fact that it is open source doesn’t really make any difference. The need for a similar level of SLA and mission-critical support is exactly the same.

Scott: So, that’s one of the things that I want to drill into a little—two things actually. One is support and the other is the software development lifecycle.

It looks like if you take a look at something like Spring, there is nothing magic about the way it is built perhaps. You probably have internal meetings. You come up with plans. You come up with specs. Engineers are assigned. Software is written. It goes to quality assurance.

Open source gives you much tighter coupling to the user base and the community at large. They provide better issues because they look at the source code and even recommend a certain implementation and talk about the merits of a particular implementation. They file bugs directly into the same bug database that your internal people use. So, open source gives you that tighter coupling with your users. But—and I don’t think this is unique to Spring—I am finding this a lot; it is still built the way software has always been built.

Good people. Good process. Good QA. You have people assigned to do documentation. You have people assigned to do QA. Even though those are not the things that people would volunteer for, you’re not dealing with a bunch of volunteers. You’re dealing with paid employees.

Am I accurately summing it up?

Rod: I think you’ve got it. You raised one really interesting question there. You said that the community uses the same issue-tracker as we use. That is obviously true. We don’t have anything around the software in terms of management of bugs that isn’t publicly visible.

A big part of open source, to me, is free use of information. I know who wrote a piece of code, I know when it was last changed; I know what issue that person was trying to fix when they wrote that code. I can see all that information.
It also means that there is a great deal of security in adopting software that was produced by small companies. For example, at Interface 21, there are slightly under 50 people. We’re growing pretty rapidly, but that’s still a pretty small software company.

So, let’s suppose that our product was closed sourced. We would, in each deal, be dealing with escrow agreements, that kind of thing. Frankly, other people would not think of using our software. They wouldn’t necessarily know whether the internals of our software were any good. They could see it from the outside, but they wouldn’t know, for example, what’s our philosophy of bug fixes, what is the quality of the documentation internal to the code, or what is the quality of design.

I do believe that there is a very strong benefit in open source. I believe that all that information is freely available. You could look at the software, and I would strongly encourage [that for] customers making significant adoption decisions.
They’re probably never going to modify the code, but they should send some of their best guys to spend a day looking at the code. See what they think of it. See whether or not you think this code is maintainable, whether or not the language idioms reflect best practice. That’s a very interesting benefit.

Scott: One of the other things you talked about was support. That comes up a lot with open source, because the way to get support isn’t always as obvious as with closed source. If you buy something from Adobe or you buy from Microsoft or you buy from Symantec, it’s obvious where you go to get support.

In the open source world, it seems there are two models. One is, “OK, you pay MySQL, you buy their enterprise software and you get support.” It’s the same thing with RedHat and the same thing with Spring, I’m guessing – can you purchase a support agreement from Interface21?

The other option seems to be that you go to someone like Open Logic, who supports everything, and you just pay them for support across a lot of products. Now, you’re probably a little biased in your answer, but what do you think about the two different ways of getting support on a product?

Rod: There is nothing wrong with an aggregation model if you’re looking at, effectively, as a broker who is providing the ability to reach the bloke who wrote the software.

Where the aggregation model, I believe, is flawed, is in cases where essentially it’s not providing the ability to reach the people who wrote the software, and also where it is not providing the ability to pay for improvements to the software. So, what am I buying? So, let’s suppose we’ve agreed that we need to give something back. If we’re adopting free software for potentially the next 10 or 15 years, we need to give something back.

We care not only about how we get bugs fixed or how we get our production systems working, we also care that this software is still going to be mainstream in five or ten years or at least hasance of being mainstream because if it isn’t, we’re saddled with an expensive legacy. We also care, clearly, about the software evolving to accommodate new features that are coming forward in the industry over time. So that means: First, we need the ability to make critical fixes if necessary, to ensure that there’s no risk that our production systems are affected.

Second, we need to be concerned about where that investment is going to come from in terms of sustaining the product and moving it forward. So, we’re not just paying for an insurance policy today, we’re paying for the insurance company to be there tomorrow, and for the premium we’re paying to the insurance company tomorrow to be meaningful.

This is where some of the aggregation models, particularly what Open Logic calls it’s “Expert Community,” totally fall down. Because with the Open Logic model, not a cent of what you’re paying is going into writing software. You are not paying a cent towards writing software. That can never work in the long term. What you’re doing is paying, presumably- I haven’t looked at their price point, but I’m assuming their price point must be pretty low – a low price point for something that is not comparable with true enterprise support.

There are two problems with the model. One problem with the model is that it does not do anything for the health of the goose that lays the golden eggs. It says, “Well, I’m not going to feed the goose. I don’t care about feeding the goose; I’m just going to scrape up the golden eggs.” Obviously, the goose isn’t going to be there tomorrow, if that becomes the dominant approach.

The second problem is you just cannot get good quality support out there. Let’s suppose that I’ve got a piece of software that may have a lethal, critical problem in production. Now even though I think, for example, that the Spring Framework is a fantastic piece of software, the best pieces of software in the world have bugs in them. It’s impossible to write perfect software. It doesn’t matter whether it’s open source or closed source.

So, you’ve got a critical problem in production, and it’s really hard to resolve. You will see this happening with both commercial companies and closed source companies and open source companies. They all need to assign resources to work on a resolution of that problem until it’s fixed. When the developer in San Francisco clocks off work, he hands it over to the guy in Australia, who continues working on it until, probably, the guy in Romania starts work at the end of his day. And you need to be able to pay for that, and you cannot support that quality of support with essentially paying pocket money to volunteers.

Obviously, I have a vested interest. My company represents a different model. I hate it when people sell open source short. I hate the view that, “I only use this open source thing because it is going to be cheaper”. I feel so strongly about that because I used to be on the other side. I used to be the person making purchasing recommendations. In the financial industry companies where I worked, I would look at TCO. It made no sense to me to say, “Hey, I can save $100,000 here on a support contract.”

What I needed to look at was where, for example, one of the systems that I was involved in designing handles $5 trillion annually. If anything goes wrong with that system and we can’t fix it for a day, the cost to that company could be in the tens of millions of dollars. In fact, if it was down for two days in that example, the consequences to the UK economy could be quite catastrophic. So, I’m not going to be making the decision on, “Well, you know we can choose open source here and we can save $100,000 on a support contract.” I am more likely to be thinking, “OK, well if open source software seems to fit our requirements really, really well, what do I do if I have a mission critical outage? Who do I pick up the phone to call? Who can organize a team to work on this until it’s fixed?”

Scott: One follow-up: It seems that we are talking about the extremely large enterprise accounts that are handling enormous amounts of financial transactions or things like that. They have enormous vested interest in the quality and the future of the software that they are running. It seems like smaller organizations, whether they are choosing closed source or open source, kind of get to draft off of that.

Talk about the decision for the smaller organizations. For a 50-person company, how is the decision different than it is for an enormous financial exchange?

Rod: That’s an interesting point. For the smaller companies it depends on what you’re doing. For smaller companies who are not doing something that is as mission critical, it may be that the software effectively is free to them, and they don’t have the same requirements around mission-critical support.

Yet frankly, from my point of view as a chief executive of an open source company, if people who don’t need that kind of mission critical support don’t buy it because, for example, their company simply can’t afford it, good luck to them. I would love to see the 50-person company grow into a 500-person company with a website that has a huge amount of traffic for which they need to buy a support contract.

So, I think that open source expands the market and it brings the software to a new audience of people who don’t necessarily have the need or capacity to pay. Essentially, good luck to them! No one should give open source money out of the goodness of their hearts. They need to look at what they need as an organization and who can satisfy that.
So, for example, for every job where I have personally been involved, I would have been getting support contracts for the open source I was using because of the nature of those businesses.

However, let’s suppose I was working for a start-up, a dotcom that is trying to build up some kind of community site and is trying to get traffic. It doesn’t really matter if it goes down some of the time. The software that my company is developing is rapidly evolving. At that point I wouldn’t have a support contract. Let’s think about Twitter. Twitter is a great example here. Twitter clearly went through that. Then it became successful. Now they are encountering technology limitations, which are causing frustrating outage. Now you are getting into a different set of business needs. Now they are reaching the point where it is going to harm the growth of their business unless they improve the uptime of their infrastructure.

Scott: So, in other words, your goal as a start-up is to grow to the point where you need a support contract.

Rod: I have never put it in those words before but I think it’s an interesting way of thinking about it. In a way, you have a greater ability to pay the price that is appropriate to your business needs.

Scott: Right. At some point if you are a start-up and you’re not successful, then the open source that you are using is probably not as valuable to your business as if you are a start-up and you become very successful and you grow. You become an ongoing concern. Well, now your infrastructure has value because your business has value and you need to treat it as such.

Rod: Yeah. I mean what I would suggest is that in the early days people should look at purchasing things like consulting and training around open source.

Scott: Right.

Rod: I think that makes a whole lot of sense because it is simply going to reduce their total costs by making their developers more efficient.

Scott: Right.

Rod: So, the needs profile of the end-user is significantly different.

Scott: Well, I really appreciate you taking the time to chat.

Rod: No, I think that was an interesting discussion. Thank you for following up.

Scott: Thanks very much.

Rod: Thank you, Scott.

Bookmark this:

Interviewee: John McCreesh – Open Office

John McCreesh

In this interview with John who is the Marketing Program Lead for Open Office we asked him about:

• What the process is for delegating work out to members of the OpenOffice team
• How OpenOffice handles end user requests for features
• The engineering steering committee for OpenOffice
• Large Company contributions to OpenOffice
• How the Q/A process is handled for OpenOffice
• How OpenOffice generates appropriate documentation for end users
• How much goes into keeping OpenOffice cross platform (Mac, Linux, Windows) vs. emphasis on creating new features.
• The Office 2007 Ribbon, innovation, and standards as it relates to OpenOffice

Scott Swigart: John, thanks for taking the time to chat. Could you take a minute and tell us a little about yourself?

John McCreesh: I have been doing work on a voluntary basis for OpenOffice.org for five or six years now. For the last year I have been leading the worldwide Marketing Project.
My day job is in big commercial IT shops. I got into open source during the .com era – I suddenly found this wonderful world outside! As I’ve got a technical background, I started hacking for projects and then after a time decided I could probably do less damage by marketing than hacking. So that’s where I am now.

Scott: OpenOffice.org is interesting to us because a lot of the open source projects that we look at are things like the Linux kernel or Apache. Those are almost by developers for developers, to some degree. The feature set is driven by the developers who work on it. OpenOffice.org is different in that it doesn’t target IT professionals so much. It targets end users, right? The users are people who are producing documents, essentially.

So I’m curious, with OpenOffice.org, what’s the mechanism for features to get proposed, to be slated to be worked on? What’s the process for people deciding they are going to work on a particular feature? How does that work? Because it seems that it would be different from something like Apache.

John: That’s right. I think the traditional open source model is very much about scratching the itch. If you are a developer and you are not happy with the editor that you are using then you go off and write your own. That model does not apply in OpenOffice.org where as you say, it is much more of an end-user tool.
I suspect a lot more of our developers would sit down and write a document with Emacs than they probably would with OpenOffice.org.
The thing is though, lots of people get a big kick out of developing for a project that their mom and dad use, or the kids use, or they can take around and hand out at the school and say, “Look, you can use it for free and I am one of the people who helped develop it.”
Another thing is that this is a huge piece of code. So, for developers that poses challenges, compared to a lot of traditional open source projects which are comparatively small in terms of code and have a small number of developers.
The development methodology of breaking things down into small modules makes it easy for more people to work in open-source. But, OpenOffice.org has grown up over 15 years, so there is some very old code there and there is some very new code there.
Getting developers in to find their way around and to make contributions can take a bit of time. But equally, if you’re technically inclined, to get some code in there, to build OpenOffice.org, and see your work coming up the other end, again, that’s a big achievement.

Scott: Right.

Sean: When we talked to Stormy Peters, one of the things we talked about is how products take in end user requests, But So with that in mind what is your the overall process with OpenOffice.org for taking in a user’s request likesuch as, “I would like the ability to handle text editing just a little differently here because my boss wants me to.” But they have no ability to write it. In other words: What’s the process for getting those things integrated?

John: Yeah, we sometimes get interesting conversations between developers who say, “Well if people want this, why don’t they just write it, or find someone that can write it?” I think one of the challenges for us is to bring together people who’ve got the ability to respond to the user’s requests, and to get the users to put the request together.

We have the standard open source toolkits. Anyone can raise an issue or raise a request for an enhancement on our website. It will get checked to make sure it is valid. Other people can go in and support it, add notes to it and so on. So there is that traditional open-source process: people vote for things. It is an entirely open process and you can see whether what you’re looking for is commanding support in the community.

But equally we have a fair number of developers who are paid to work on the project. So, Sun Microsystems’ developers take the OpenOffice.org code and release it as StarOffice. Like any commercial software house, they have big customers who say to them, “Hey, you know this particular feature of StarOffice is poor. Can you do something about it?” If they’ve got a support contract with Sun, then Sun will put some developers onto it and they will get into the OpenOffice.org code base that way.
Similarly, when Novell started looking at the code base and decided to take OpenOffice.org and make it a significant part of their SUSE offering, they started using it internally themselves. There were things that they thought, “Yeah, we need to do this. We need to add this feature. This is important to the kind of market we are going to sell in.” So again, they took some of their developers and put them to work on it.
So OpenOffice.org is a combination of the traditional software model where you have a software company who listens to its users, plus input from users around the world.

Sean: What percentage of the new features that go into OpenOffice.org do you think are driven by a developer request mostly from soup to nuts or vs. what is onethose that is are driven predominantly by an end-user request but then just implemented by a developer?

John: That’s a tricky one.

Sean: Do you think it is equal or is the product still driven by developer input more than end-user input?

John: Well, it’s hard to say. One of the things that people said to us at the launch when we released the Version 2 product was, “Yeah, you’ve got all the functionality that we want but can you make it run faster?”

Now, from a developer point of view, a lot of the hacker community was really motivated by this. The thing about making more efficient code and making it smaller and leaner and all the rest of it really rang bells with a lot of people. That was something that had tremendous developer appeal, but it also has real marketplace end-user appeal. So I don’t know whether you would say that was driven by end-users or driven by developers. It was a happy area where the two interests coincided.

Sean: OK.

Scott: So on an open source project there is usually a chief maintainer right? If you take a look at the Linux kernel, it’s Linus Torvald who ultimately gets to decide what’s in or not. Is it Sun Microsystems that’s the final arbitrator of what gets checked into the main source tree and what doesn’t?

John: We have an Engineering Steering Committee, or ESC. It’s the ultimate arbiter if there are disputes about what goes into the code or what doesn’t, and the general technical direction of products is decided there.
On a day-to-day basis, the project is too big for any one person to sit there and say, “That goes in and that goes out.”

Scott: Right.

John: So OpenOffice.org is divided into a number of different projects, where the project leads have the ultimate say as to what goes in or what doesn’t go into a particular build.

Scott: OK. But I’m guessing that because Novell and Sun are so involved in it that some of their people would be owners of these different subsystems?

John: Yes. And I think that’s perhaps going to continue more and more as more commercial companies sponsor developers.

Scott: And that’s not uncommon. The one thing that I have heard across the board with open source is that company involvement is not a bad thing. Open source owes a lot of its success to the fact that IBM, Sun, Novell, Intel, AMV; other corporations are paying developers full time to work on it. So that’s an integral part of the process.

John: Yeah, I saw some analysis years ago about the Linux kernel. At that stage most of the contributions were coming from people who were paid by corporations.

Scott: Yeah, I would guess OpenOffice.org isn’t any different. I mean, when you take a look at a product that’s this large, that is this complex, there is a decent barrier to entry—I would guess—in really being able to get in there and really understand the code base, really develop experience with it, really understand the architecture and just get to the point to where you can make good, clean contributions that are going to add features or fix bugs and do it the right way.
For a lot of these projects, they have grown to the point where there is a high barrier to get to the skill level you need to get that to be able to do that. So it makes sense that the people who are being paid to work on it are naturally going to have an advantage in just really be able to produce the quality of code that is going to ultimately make it in.

John: One of the changes we have tried to make with the architecture in the past couple of releases is to open up the product more for extensions.

Scott: Yeah.

John: People who aren’t as technically skilled, and will find it a real challenge to go through pages and pages of code (some of which might be commented in German) and try and make sense of it, are quite capable of providing functionality in the form of a plug-in.

Scott: That also seems to be a key factor in the success of a given open source product—that it has a good extensibility story. It has a good modularity story. Apache is a great example of that. To work on the Apache core there is a high bar. But to write modules—anybody can do it, essentially.

John: Firefox is another classic example of that. It’s used a lot. Again, the developer gets a real kick out of seeing their piece of code looking as if it is part of the core product. Once you have incorporated the extension into the menus and in the help system, etc. then as far as anyone looking at the product is concerned, you have written a piece of Firefox or a piece of OpenOffice.org code.

Scott: Right, right.

Sean: I have a question about the new features. How is the QA process handled? A lot of our conversations have taken us in interesting directions as we talk to people who submit stuff, not in any pejorative sense for an open source but it just is interesting how who are contributors as the process of checking in code, andgoing through the process of validating it differs from project to project in terms of the rigor or the processes that go about it. So how does that work for OpenOffice?

John: There are two schools of thought. In OpenOffice.org there is the “Community OpenOffice.org”. So if you go onto our website and download OpenOffice.org, that is what you get. There’s also a hacker’s version of OpenOffice.org, which doesn’t go through the Community QA process. This version feeds into some of the Linux distros, who will take this code and will do whatever QA they feel is appropriate around it.

So if you want an absolute leading edge of OpenOffice.org, you go to the hacker’s version. If you want something that has been through quite a structured QA process, you go for the community version, which is what we do.

Scott: The community version, is that…? Who is ultimately responsible for doing that QA? Are there QA teams within some of the corporate sponsors of OpenOffice? Is it more of the responsibility of the distros to do that QA as part of putting their distro together?

John: QA is one of the OpenOffice.org community projects, which runs one of the most widely geographically distributed QA processes I have ever seen, because it’s not just a matter of getting the American English master version correct.

Scott: Right.

John: The native language teams in all the various countries will go ahead and do a QA process on their own build. Ultimately that’s one of the things that controls how quickly we can release. We used to go for long periods of time between releases. A couple of years back we looked at that and decided it was putting developers off. If you have to wait a year before you can see your code emerging into the marketplace, you’re not really very interested. We’re now down to about four releases for the year.
We would like to do more but the QA process is a limiting factor.

Scott: Sure.

John: Similarly for the hacker’s version of the code: if Red Hat or Ubuntu decide to use that as their source, they have to put it through whatever QA process they feel is appropriate for their marketplace. We all have tradeoffs between features and a stable product.

Scott: Yeah, yeah. And things like documentation also—is there a documentation team as part of the project as well? I would guess that would run into the same localization challenges because the product is such a global product.

John: That’s right. We also have a Documentation Project that does user guides and manuals and how-tos and all that good stuff. But there must be about a dozen other independent sites on the web where people have set up help forums or wikis or whatever.

Scott: Gotcha.

Sean: How much of the documentation creation is deflected or is tuned based on gaps you find that users are asking for? How much documentation is written because someone feels a personal need to write that documentation, or because they know people have struggled with it—similar to a developer adding a feature just because they need it?.

And how much of it is driven top-down from OpenOffice, such as, “We see X amount of requests for this, we need to build a doc set out to cover that need?”

John: It’s a mixture of the two. The way that I got into the OpenOffice.org project in the first place was I wrote a piece of documentation – a ‘how-to’. I’d been playing with the database stuff, thinking “Hey, this is really cool.” But it wasn’t not obvious how you got into it or how you found it in the menus.

So, I asked a few questions on the lists, and wrote the how-to. That got a fair amount of circulation, so I realized there was obviously more to OpenOffice.org than meets the eye. And that was how I got into the project.

So, an awful lot of that goes on. You can never have too much documentations and how-tos for a product as big as this, because different people have different learning styles and different needs.

Sean: One last follow-up on that. Are there types of documentation or areas of the product where you find that maybe there’s a pocket missing content-wise because it’s more community-driven versus top-down?

And are there any defined areas where you feel there is a gap and you have to motivate the community to contribute in a particular area?

John: I think the problem we have is the old Google phenomenon, that there is now such a mass of stuff out there on the web, it’s tricky finding the stuff that may be current, knowing what versions it applies to, and then finding the information in the form that you require.

We usually do one big master user guide, which we publish on the web. It’s also available from one of the publish-on-demand organizations, so you can order paper copies of it from there.

But apart from that we don’t try and have a monopoly on documentation because there are just so many other people out there who want to do it.

Sean: But you would say that if there is a pain point of sorts, it is that, “OK, I’ve Googled how to do mail mergex, but I don’t know if it’s the appropriate steps for this version of the product.” You know what I mean?

Or if it is, it might be missing an extra step that could accelerate ithe process of getting the job done from a user perspective. It’s like you said, you could solve a lot of developer problems by Googling them. That’s almost like Step 1. But on the other hand, you’ve got to make sure it maps to your version of the API, your version of the development tools, those kinds of things.

So, that’s a fairly accurate reflection of the main pain point.

John: I think so. We can produce a master reference manual, but it is in reference manual style, and some people find that off-putting and they’d like it more conversational style.

Sean: So, one question that I have is in OpenOffice, one of the key features is its cross-platform nature. It runs on Windows, it runs on Mac, it runs on every Linux distribution. Do you have any sense of how much development effort goes into ensuring that it is supported and runs well on all of those different platforms compared to the effort that is put into building new features?

John: I think it’s back to the QA limitation, that the more that we officially support, then the more testing that you have to go through. It’s also a major technical challenge because, if you’re completely cross-platform, then in a sense you’re always slightly sub-optimum on any given platform.

Scott: Well, that’s another question. Does OpenOffice.org strive to have full fidelity across all platforms? Because I know that some open source products go the route of they’re really optimized for Linux, and they’ll run on Windows or Mac or things like that. But they’re not 100% the same experience. Do you know if the philosophy of OpenOffice.org is that the experience and the features be 100% and the same, even if it means that it’s not really optimized for any platform?

John: That’s effectively where we find ourselves.

Scott: OK.

John: This is why the Mac port has taken such a long time, because the Mac is such a specialized platform and Mac users want all their applications to look just exactly like a Mac application should do. So, it’s quite hard to do that and still maintain the cross-platform thing.

Scott: So, there’s a core that’s really generic, but then for some of the presentation layer stuff, there’s actually Windows-specific code, Mac-specific code, you know, Gnome- and KDE-specific code.

John: We provide for example a common file selector out of the box. Or you can switch it off and say, “Just use the native file selector for my platform.” But the more of this you offer, the more your documentation has to be platform-specific. The common look and feel that makes the documentation piece easy to do.

Scott: Right.

Sean: This is in a different direction, but there’s an argument that goes that certain products in the open source world tend to follow rather than innovate. Sometimes OpenOffice.org is held up by that, right?

It’s trying to get to some base level of parity with either the most recent or the current version of the obvious competitor, which is Microsoft Office. But adding features that leap ahead beyond the current evolution of the office product isn’t typical.

I mean, just this statement, what’s your general response to that, when you bump into that? Because I’m sure you must have, at least at one point or another.

John: The problem is that office productivity software is a mature market. It’s been around a long time and we’ve been through the WordPerfect and the SuperCalcs etc.. By now, people know what they want out of an office software product, and they also have an expectation about how it’s going to look and feel. You know, where they’ll find things on menus.

So, you need quite a strong driver to go out and break out and produce something radically different. Chances are that unless it fundamentally gives people a better way of working, then they’re not going to accept it. Why would they learn this whole new thing if they only want to write a memo? They already know how to do that.

Sean: I guess in a product like OpenOffice, what would be the path to innovation? If you’re saying, let’s imagine a world where both are free, right? And we’re trying to equate it based on our level of productivity benefit to the end user and those kinds of things alone.
Where do you think OpenOffice.org or another product that’s a productivity suite, can have a leverage point there, I guess?.

John: Let me tell you where I think this happened. When we launched Openoffice.Org 2, we closed all the functionality gaps between OpenOffice.org and Microsoft Office. So for the average user, what they do 90% of the time is absolutely no different on either package. We recognized that and Microsoft recognized that.

So, our response was, well, the one thing that people are telling us is that, “OK, maybe we’ll move to your software, but every time we move we’ve got all these data conversion errors, or data conversion problems”. Or “I spend all my time writing these documents, but unless I’ve got your brand of software, I still can’t access my documents.”

So, we were hearing a demand from users that they wanted to own their intellectual property, the documents and spreadsheets that they’d created, independent of any software supplier. This took us off down this road that is ultimately the OpenDocument Format that you’re well aware of.

Sean: Yeah.

John: So users got an ISO standard for how office data should be stored. Once you’ve got that, then no software vendor can ever lock you into a product again. And it makes it easier to get your data and use it in corporate systems and all that good stuff.

So, that was our response to “how do we differentiate ourselves in the marketplace from Microsoft?” We knew that Microsoft would have grave philosophical difficulties going down that route.

At the same time, Microsoft looked at the same problem from their side and went, “What can we do to put a gap between ourselves and OpenOffice.org?” And their response was not actually to add new features or change the core of the product, but to change its look and feel.

This is where we got the Ribbon and all this other stuff. Sure, it looks different. Their response was pretty much a consumer-driven thing. “Let’s make it look sexier in the marketplace. Let’s get it looking so sexy so that people see it at home and then they go back to work and say, ‘Hey, we’ve got to have this Ribbon thing at work.’” It doesn’t actually add anything to their productivity, but it looks cool.

So, Microsoft has gone off in one way, which is a consumer-sexy-look-and-feel thing, and we’ve gone down the more technical route. What people are telling us on the domestic side is they want to know that their grandchildren will be able to pull up Granddad’s diary from 50 years ago and still be able to access it in whatever office software they’ll be using then. On the commercial side, public administrations worry that 20 years from now, someone’s going to come slap on with a Freedom of Information Act requirement to dig out some documents they’ve filed today. Are they going to have the word processor that wrote that document 20 years ago? Absolutely no chance!

So, the ODF development was our response to that demand for freeing the data from the application. So, was that innovative? I think that was hugely innovative. I think it was far more innovative than Microsoft’s response – . they got a few bells and whistles and their menus look a bit different.

Scott: Something like the Ribbon, that’s an interesting point, because how does OpenOffice.org make a decision about whether or not OpenOffice.org will adopt that look and feel or not?

Because I think that’s been a debate. OK, Microsoft significantly changed their look and feel, should OpenOffice.org do the same or not? What’s the thinking on that? And, I guess, how do you make decisions like that?

Sean: And –before you answer that—one of the things that I’m pretty cognizant of is that part of the change for the Ribbon was obviously just change itself. And then the other part of the change was driven by some studies that said, “Well, let’s put the first thing a user wants to use first in the Ribbon, etc.”

There are a lot of arguments about the decisions that were made, right? One person’s choice might not be another’s. But that was the driving influence too. But And to Scott’s point, how would something like that come to fruition in the OpenOffice.org environment? If it was deemed necessary, obviously…

John: OK. The Ribbon thing is one of the best things that happened for us for a long, long time, because over the years, we were sick of hearing arguments that said, “OpenOffice.org menus are slightly different from Microsoft Office’s, therefore, if we’re going to have to migrate people, it’ll cost us billions of dollars to retrain everybody to know that option isn’t here, it’s somewhere else.”

Scott: And now they’ve given you even a bigger change on their own end, right? So, yeah, there you go.

John: It’s a much simpler migration path from an end user perspective to go from current Microsoft Office to OpenOffice.org than it is to go to completely new Ribbon-style interface.
Will it catch on in the marketplace? I don’t know. I think it’s a big gamble on Microsoft’s part. They’ve tried a couple of times in the past to push the market in ways the market has eventually revolted against. They have had marketing failures in the past.

And it also takes an awful long time to get people off legacy versions of Office. There’s never very much more than low double-digit numbers of people using the latest version. So when latest version is such a big step change, it’s a big gamble for them.

Would we ever go the same route? Well, if five years from now, if everybody in the world has decided that Ribbon is the way to go and that’s what they want in a product, I suspect we will have to do something similar or come up with something better and convince the world that’s there’s a better way to do it. But at the moment the jury is very much out.

Scott: So in other words something like that, the prudent thing to do is to take a wait and see approach, in other words.

John: Exactly, just as Microsoft is doing around ODF. It’s doing its usual thing of trying to say, “Well, OK, if you want a standard, we’ll invent the Microsoft standard, and try to sell that to the world.” So that was a fairly predictable response.

Scott: Now, I thought I did read something saying that Microsoft was going to allow something like a “Save As ODF.” I thought recently there’d been a change in their thinking on that. I mean that’s outside of the scope of this conversation. But it seems like I bumped into that, but maybe I, maybe not, I don’t know.

John: What they have done is they’ve offered some support to an open source project to set up converters – ODF plug-ins for Office. Part of their recent agreement with Novell—and I think there’s something with Linspire as well—had some clause in it about working to get more file compatibility.
So, they’re not stupid. They’re keeping in touch with the technology, finding out how it works, so if they do find the market is forcing them down that way, you know it’ll be a comparatively easy thing for them to quietly drop into the next release of their product.

Scott: OK, I get it.

Sean: Well, one question on that too. So would you say that one of the strengths of open source is that you’re not going to push for a change as significant as the ribbon unless the user base asks for it? In other words, you won’t push a change like that top-down and if the users aren’t asking for it then a wait-and-see approach makes the most sense because nobody’s asking.

John: That’s right. It’s hard to think of anything where we’ve gone to change for change’s sake. But, equally, if one of our developers came up with a really cool new feature that no one had ever seen in an office suite before and, and we looked at it and thought, “Hey, that’s, that’s wonderful,” we’d do everything in our power to get it out into the product.

Scott: Well and I guess too you guys have the option of, like you said, there’s the hacker’s build, there’s the experimental build, and it seems to me like a lot of things could be tried out there, and if they caught on and were popular, it might make sense to move forward into the stable tested one.
Is that how it works? Is there a Darwinian effect with features where, somebody codes it up and it makes it into the hacker’s build, and then it either lives or dies there? It either catches on and it makes it into the, the community one or it doesn’t and it doesn’t.

John: Yep, that’s been a route in the past and we think the extensions route will be a much faster way in the future. So if someone brings out an extension, and we see that everybody and his dog is downloading it and blogging about it saying how wonderful it is, and if it makes sense to put that into the core product, then that would be a very good way of moving the product forward.
But, equally well, if it’s working well as an extension then why would you want the overhead of putting it in the core product, unless from a maintainability or efficiency or some other reason?

Sean: Well, what, what, one question I want to ask goes back to the original genesis of what we’re trying to accomplish in our investigation, “if the question was posed to you, “OpenOffice.org is predominantly an open source project. Microsoft Office is obviously a closed source project. From a development methodology standpoint, what would you say are the important characteristics that are advantageous on the OpenOffice.org side compared to Microsoft Office?”

Scott: What Sean is going for is what do you see as some of the biggest advantages of having OpenOffice.org developed as an open source product? What comes out of the open source process that is very advantageous?

John: OK, from a developer’s point of view clearly it means you know anyone can contribute to the project without being a Microsoft employee. From a general marketplace perspective, the appeal of open-source is it’s a transparent process. Anyone can request features, record bugs, whatever. If you’ve ever tried doing that with Microsoft you’ll know it’s not a transparent or efficient process. Anyone can come along to the OpenOffice.org conference and talk to developers and buy them a few beers and say, “Hey! Why don’t you come work on what I’m looking for?”

Then there’s the whole issue around openness of what’s in the code and what isn’t in the code. There’s been umpteen conspiracy theories over the years about Microsoft back doors etc. OK, most of those are just Internet conspiracy theories, but for a lot of governments in the world who are suspicious of big corporations, the fact that they’ve looked inside the code, seen what was there, get their own people looking at it, is very important.

And finally, software companies come and go. Even Microsoft will go someday. If you own the code or if you can get a copy of the code you’re guaranteed that you can run it just as long as you can compile it.

Scott: John, thanks for taking the time to chat.

John: Thank you.

Bookmark this:

Phil Costa

In this interview, Scott Swigart, Sean Campbell, and Richard Bowler interview Phil Costa who is the Director of Product Management for Flex and ColdFusion at Adobe, with responsibility for product definition and strategy of the Flex product line. Prior to joining the Flex team, he was product manager at both Macromedia and Allaire and led XML and Internet middleware research at Giga Information Group. Phil has a Master’s degree in English from Boston University and an undergraduate degree from Swarthmore College.

In this interview, Phil talks about Adobe’s decision to open-source the Flex SDK. In specific, Phil talked about:

• Background on the Adobe Flex SDK
• Why open-source the Flex SDK
• Risks in open-sourcing the Flex SDK
• Why the MPL license was chosen
• Advantages in open-sourcing the Flex SDK
• The risks of people forking the build

Scott Swigart: We were commissioned by Microsoft to do an investigation into the differences between how closed source and open source software are built, and how those changes in software development methodologies manifest themselves in the final products.

[Adobe] Flex was particularly fascinating to us, because it started out as a closed source product, and now the Flex SDK is being open sourced. We wanted to get some insight about that. That obviously made sense for Adobe to do, but it isn’t necessarily obvious to us what goes into a decision like that. Why does open-sourcing Flex make sense?

Phil Costa: It might help to give you a little bit of background on Flex. We announced about three weeks ago now that we were going to be open sourcing the Flex SDK. That actually hasn’t happened yet, but it has been a process we’ve been going through for the past two, two-and-a-half years. It has really accelerated in the last year.

The Flex product line started as a server-based product that we sold to a very small number of high-end customers, and we’ve been gradually opening it up, both from a license standpoint, in terms of making it free or parts of it free, and then also opening up from a source code standpoint.

We started publishing all the source code about ten months ago, and we’ve been gradually more and more open about the roadmap we’re taking with the product. Actually making the license open source was a logical next step which we’ve always envisioned taking.

Scott: What are the advantages of going with more of an open source model — what are the advantages of taking something that hasn’t been open source and open sourcing it?

Phil: There are a few different elements to it. First, there’s the nature of the product itself. The core part of the product is try hard to internally test, and get all the bugs out, because it’s a development framework. By its nature, it gets used in a million different ways. It’s very hard to actually set up tests for all those ways and to chase down all the bugs.

So having an open source model will actually help us by having more people looking at the code and suggest changes based on their particular use-case. In one respect, we view it as a way of magnifying the QA resources we have, and also magnifying some of the bug-fixing resources. So, that’s a very tactical thing, but that is a proven advantage of an open source project.

The second element refers more to the evolution of the product. We’ve always tried to be very customer-centric in terms of designing the product, working with early adopters, niche users, and new users, to determine what things we should be adding to the product, to make sure that it fits their needs, both on the learning-curve side as well as the power-user side.

We decided that having a group of people who can directly influence that — or at least feel more deeply invested in it — was a good way to continue to evolve the product. With a developer product, the people who develop the product are also the users of the product, and there’s a very efficient feedback loop.

The third piece is more PR and marketing focused. Because people have to make a substantial investment in Flex — in terms of spending a lot of time developing an application and then making their application dependent on the Flex framework — they’re looking for an open source project or a set of standards. In the rich-ended application space, there aren’t really any standards, per se. They’re mostly looking for open source or de facto standards.

It’s become increasingly apparent, as the market has grown up, that to be successful as a platform you need to be an open source project, so that people view the product as being bigger than one individual company, or in some cases one individual product team. In a lot of ways, that was another requirement that more and more customers were raising; they love Flex but they wanted it to be bigger than just Adobe.

Scott: What are some of the risks that you saw, if any, in open sourcing Flex? It was an evolutionary process, it’s a track that you’ve been on, but were there risks or concerns that were identified where people thought that this might not be a good idea?

Phil: I think some of them are unique to a product that is going from being a closed source product to an open source product, and some are more cultural.

On one hand, we looked closely at taking all of the current business relationships we have with people around Flex, not only from a revenue standpoint but also from a contract standpoint, and made sure that as we move to a different model around the Flex framework that we could either maintain the revenue streams, grow the revenue streams, or were willing to give them up to get other benefits from moving to an open source model.

Then, on an individual basis, we had to make sure that none of our customers were going to be left in the lurch because of the particular license we chose or the fact that we were going to open source at all. Because Flex is used not only by developers within a company for building an application for their web site, but also in a lot of commercial software products.

For example, SAP uses Flex, as do a lot of smaller ISVs like Dorado and other companies who have their own very specific processes and policies around the use of open source software that they can incorporate into their products. We needed to make sure that there were options available for them that would not cause all kinds of problems. That’s why we chose a dual license model, so that we could address our goals of becoming an open platform while also enabling companies that needed a traditional, closed-source license to also use Flex.

Richard Bowler: I’m guessing that’s why you guys picked the MPL license, because they could combine a proprietary module with the open source modules, and not be forced to open-source their product?

Phil: Exactly. We wanted one that was strong in the sense that it was viral around the core part of the product, but not so strong that no one would use it in a commercial product, because we wanted people to use Flex in commercial products.

We looked a little at LGPL. But our own history as a commercial vendor shows that even though the spirit of LGPL is that it can be combined with other things and it doesn’t affect them, in practice a large percentage of ISVs just won’t touch it.

Whereas things like MPL, ETL, EPL, those licenses were explicitly designed to be more commercial friendly while still carrying the basic spirit of open source.

Richard: I presume that in the past, other commercial products that are using Flex have paid for that privilege, and now anybody that wants to use it in a commercial product is going to be able to. Is that true?

Phil: That’s true. Originally, no one could use it without paying for it. But about ten months ago — last June — we made the core part of the product free, but it was still closed-source. We made the SDK, which was free and it included the source code, and there were specific cases in which you could modify and redistribute the source code, but it was all done under a traditional commercial license.

By making the move to open source, we’re making even more parts of the product free because not only can you use it for free but you can modify and redistribute the SDK for free. Of course, there are still lots of companies that will pay us to obtain support and escalations or for the additional Flex products we sell. If they’re making a big investment in Flex development, they want to make sure someone is on the other end of the phone to help them out if they run into issues.

Richard: It seems that part of the motivation for open sourcing and making it free, was it just to get wider market acceptance?

Phil: That was definitely part of it. There were lots of interesting open source projects that wanted to use Flex that didn’t feel like they could because of their practical or ideological limitations.

The other part was that, as the developer community has gotten bigger and more active, people are looking for ways to contribute back and to participate in Flex. And there wasn’t really a good way to do that, because it wasn’t being run as an open source project.

Richard: It’s an interesting paradigm, because Flex is a developer tool, so your users are the kinds of guys that are extending their own presence on the web by using your product. They are going to have very strong ideas about how your product should grow and what it should become. I imagine you’ve got a lot of feedback from your users already about how they want Flex to grow.

Is this also a way to remove some of the blockades to collaboration so that the user community can be a little more in control of how the product changes over time?

Phil: Absolutely. The way you use Flex is you write ActionScript, which is a JavaScript derivative, and MXML, which is an XML layout language for the UI. The Flex framework itself is entirely written in ActionScript, so people who get deeply involved in Flex are very deeply involved with the tools that actually created Flex. They’re the perfect people to provide feedback and to add things to it to support their needs.

What we’re trying to do is set up a process where they can do that, but it’s still run in a way that preserves the overall philosophy of the product design, and the goals of the product. The compatibility and stability of the framework is maintained over time.

Richard: I imagine that the gatekeeper for this project is ultimately going to be someone other than Adobe.

Phil: The initial plan is that all the committers will be the Flex framework product team. But over time, we plan to add external people who demonstrate skill, judgment, and commitment to the overall mission of the product. It is a big, complicated product, and we’re going to take the evolution of the project one step at a time.

Scott: One of the challenges that open source products face is that somebody decides they want a change in the product, but the people who control the source decide that they’re not going to incorporate that change, so somebody just forks their own build. And now you’ve got two of these out there, three of these out there, and so on. Is that something that’s a concern at all with this project?

Phil: It was one of the explicit concerns we talked about, and it’s still possible, certainly. I don’t expect that you will get a lot of strong forks. You might get weak forks where they take a particular area of the product and create an alternative implementation of a particular component or a particular subsystem.

Scott: It seems like your risk might be lower too, just because this is so tied into Flash and other things that are closed source proprietary Adobe. There’s only so far you can really go with your own fork.

Phil: It’s true that it’s a little bit different for those reasons. In the end, they can’t change the Flash Player API, but Flex does offer a lot of levels above the Flash Player that are design choices that we made.

I think more of what made us comfortable in the end with the question of forking was on the one hand, generally, the practice has been that people do not make major forks lightly. Having a community that can maintain and develop new projects, and taking a lot of resources and trying to split them just means that both projects have fewer resources.

And on the other hand, if you look at it optimistically, if you manage the project well and you have clear reasons why some things are being accepted and some things are not, then things that are not accepted in many cases can be moved over into a sub-project area or a related project area where they have a chance to evolve, even if they’re not becoming a replacement part of the project. In general, people submit to the group project’s will.

Richard: Are you worried about people writing extensions to it and keeping those extensions closed source and trying to pump up the value of code by just adding a neat widget?

Phil: That’s actually one of the things we encourage people to do. To give you an example, today we sell the core part of the framework, which includes the application model, a bunch of core classes, and then a library of components, and that’s the part we give away for free. On top of that, we have a pretty advanced charting and graphing package that we actually sell.

We’ve been working with a bunch of vendors, and there are already some of them out there who have built other component libraries and are selling them. So we encourage people to innovate around the core. The core we’ve made free, but we expect that more specialized visualization tools and so forth would be commercial products.

That was one of the other reasons we chose MPL — because of the nature of the framework, it gets blended in with your application. We wanted to be able to both have open source versions of that that are more community driven, but commercial versions that range all the way from individual components to full blown applications.

Scott: We appreciate you taking the time to chat.

Phil: Thank you.

Bookmark this: