Interviewee: David Campbell

In this interview with David Campbell we talked to him about:

• How the development methodology of SQL Server has changed.
• How the changes affect the ability of customers to give feedback.
• How virtualization makes it easier for users to test-drive SQL Server.
• As products and open-source projects grow, the developers have to realize that they are not the target user.
• David’s belief that the best product doesn’t necessarily have the most knobs.
• Building a community around a closed-source product like SQL Server.

Scott Swigart: David, thanks for taking the time to chat with us again. One of the things you mentioned in the first interview was the common engineering criteria. I’ve seen that mentioned in a number of different places, but I have never seen it explained. And I’m guessing that the common engineering criteria impact a product like SQL Server. It might be good to start off just by explaining what they are.

David Campbell: It started in the Windows Server system. We’ve been telling customers that one of the advantages of our stack is that it works better together. It does, but we started really challenging ourselves by asking, “OK, what are the real things we’re doing to actually make it be better together?” The way I think about it is, “What are the things that would add value for customers who require coordination or consistency across groups that otherwise wouldn’t happen organically?”

We started this a number of years back, when Paul Flessner was running the Windows Server group. We have a set of requirements, which are updated every year, and all the enterprise products within the Server and Tools Business (STB) have to adhere to those requirements. These are things like having a best practice analyzer, having a Management Pack for System Center before you RTM, having consistent user education, continuous publishing, a community engagement program, that kind of stuff.

Scott: Previously when we talked, you also mentioned that SQL Server went through a pretty big change in its development methodology between 2005 and 2008. Talk about that a little bit.

David: This is something that a small group started, and it ultimately took great effort across the division to pull off. It will never be “done,” but it has been an amazing, fascinating transformation. I’ve learned more in the last few years about culture and change management in a large organization than probably anything else.

I’ve been working on the product for a long time. Back in the good old days, we had something like 20 people who knew the whole server, end to end. Back then, as you worked on things, if you had an issue you just walked across the hall, sketched something out on somebody’s whiteboard, and went back and coded it up. Because things were so intimate, you could keep things moving at a rapid pace.

To somebody who understands the standard Microsoft development methodology, one of the things that’s very important is the daily build. We continuously build and test the product, and one major vital sign is the health of the daily build. Back when the product was small and there were 50 to 100 people working on it, you could do the daily build live, with everyone just checking stuff in and keeping it together.

But as the team grew and the size of the code base grew by several orders of magnitude, you just couldn’t have as many hands in the soup at the same time and get anything that tasted good.

In SQL 2005, we had sort of a standard large-team, waterfall model. If we were doing a feature that spanned several component groups, we would have one group get together and they’d do everything by the book. We had a process and everyone followed it. They’d write the spec. They’d develop a test plan. They’d review the test plan. They’d write the code. They’d test it. They’d check it in.

Then the next component team in the sequence, maybe it was the client data access team, would pick it up and they’d start working with it. And they’d go, “Hey, wait a minute. This interface isn’t quite right. There is something wrong here.” They’d go back to the first team and ask them to change it. Of course, the first team had moved on to something else, so if they had to go back and do major surgery, it was a hassle.

We had one feature in particular that, to be honest, we built two or three times during the course of SQL 2005 and we still didn’t have it right. So I said to the program manager at one point, “Look, either we take it out of the product or you get everyone, end to end, involved, get them in a room and just sit till you work it out and get the thing going.”

This involved the storage engine, the language parser, the query processor, the client data access libraries, the management tools. They had done it serially several times. They just never had it work correctly, end to end, when integrated.

They all got in a room. Of course, you know how that works. They were able to figure it all out and get it all done. We said, “OK, what can we learn from this?” We completely threw the development process up in the air and changed it around. How do we go “back to the future” and try to capture the intimacy of what we had in the “good ol’ days” now that the team and product are many times larger and more complex.

When we had a small team rebuilding the database engine for SQL Server 7.0, a number of people had the whole system in their heads. Furthermore, when we did SQL Server 7.0, we really didn’t need to think about the market that much. We were building a new relational database engine and introducing an OLAP system. The playbooks were already written for these technologies and the markets were already established. Now, we did it in a way that was different; we tried to make the product much easier to use so we could actually grow the market and bring the technology to a much larger base. For SQL Server 2000 and, to a lesser extent, SQL Server 2005, we were building out on the architectural base we had established with SQL Server 7.0.

The original model worked great for several releases, but when we got to SQL Server 2005, the product was much bigger, the team was much bigger, and the market expectations were much higher. Things just weren’t fitting together in terms of consistency across the product. So the other piece that we had to add for the SQL Server 2008 process was, instead of defining the release from the bottom up in terms of what we were going to do, we had to create a definition process that looked both at the market, top-down, and at what needed to happen to the technology base of the product and what people wanted to do, bottom-up, and bring them together.

The way we framed it to the team was, “Look. We want to have the PowerPoint presentation that we’re going to use in the keynote for the launch before we write the first line of code.” So we developed a set of themes for the release. With each theme we had this notion of scenarios that were end-to-end value propositions for customers. Underneath those scenarios, we had a number of what we called improvements, instead of features.

An improvement is what we would consider an engineering transaction. The idea is that we bring virtual teams together to work on an improvement. An improvement team is cross-discipline—development, testing, and program management—and it’s cross-component. For example, the database engine, data access, and management tools teams can come together to work on a single improvement. So if a particular improvement needs focus from four different components, they’ll all come together as a virtual team.

We assign an improvement team lead, who’s got a set of requirements they need to meet to integrate the result into the source base, but we’re not very prescriptive about how they run the team. If they want to use Scrum and have stand-up meetings and do a series of sprints, that’s fine. If they want to use a test-driven development methodology, etc., that’s up to them. We’ve got tools and a playbook that we encourage them to use, but we’re pretty liberal in terms of how they go about it. There’s a very strict contract in terms of what needs to be done before it is integrated into the main branch of the product source code, however.

These improvement teams work in their own separate branches in the code management system. Once they believe that the improvement is ready to ship to customers, then and only then will we integrate it into the product. What’s interesting, if you think about it from outside SQL Server, from the customer’s perspective the release looks much, much different than it used to look.

It used to be the case that—let’s say we were going to do 300 features. Everyone gets started on them. We put a bunch of stuff together and put it out for Beta 1. Everything would work about 30 to 50 percent. Then we’d keep going for Beta 2. We realized, “We’re not going to be able to finish 20 percent of these things, let’s cut them.” With Beta 2, things are working about 70 to 80 percent correctly. Then you have Beta 3, a bit better. And in the end, you’re still trimming stuff out, cutting it, and refining it to get it ship ready for customers.

We were constantly in the mode of having stuff that kind of worked in Beta and trying to get feedback from customers. We flipped it around with the new process. What customers see now in a particular drop is very high quality. They used to see the 120 percent of the ultimate features in Beta 1 in some measure. Now they only see 20 or 30 percent of the features early, but they all work. We also changed the name from Beta to Community Tech Preview, or CTP.

As we go through each successive CTP, more and more of these improvements show up. They’re baked, end to end, and they accumulate in number. What a customer is seeing is that the quality is actually very good the whole way through, so it’s an interesting change.

Scott: It seems as if the SQL Server team went through the same process the Visual Studio team did: Instead of putting code out there that is 30 percent functional and people can’t really get it to work end to end, they decided it’s better to wait until a given feature is 100 percent even if that means customers see it much later.

But there has to be a trade-off, because your window for getting customer feedback is smaller, and you have potentially invested a good deal more time in the original code before you get any feedback. Talk about that balance.

David: That’s a great question. One aspect of the previous releases is that we were frankly playing catch-up in an established market. We knew what we wanted to build and we knew how we wanted to differentiate in the market so we really didn’t need a lot of customer feedback on the feature set for SQL 7.0 because in many ways we were rearchitecting and reimplementing the existing product. This approach worked for SQL Server 7.0 and 2000, but one consequence is that we hadn’t built up muscle around how to engage customers earlier in the design process for completely new features. In some sense, we built stuff and threw it against the wall to see if it would stick. This worked well for awhile, but it made much less sense starting in SQL Server 2005.

Basically, you could say that the old process was quite inefficient, in the sense that we built a bunch of stuff, actually committed code, before we even knew what would stick in the market. Now we’re trying to do that through more concept testing, more up-front work, spending more time with customers to look at their real pain points. Are you going to build a set of features that’s going to be compelling? That’s one piece of the puzzle. Are you building something all up that will be valuable to your customers to you customers end to end considering they way they really work? That’s another pieces of the puzzle.

But then the question becomes, have you built each feature in a way that makes sense to the customer and is useful? So it’s less about the value proposition of having a particular capability and more about have you designed and built each thing correctly, on an improvement-by-improvement basis.

One fallacy around the Beta process, in my opinion, is that we would throw a Beta out there, and tens or hundreds of thousands of people would download it. A small fraction would really play with it, a smaller fraction would actually think about how they’d use a new feature, and a fraction of those people would actually give us any real feedback. It wasn’t very structured. Our TAP program tried to target specific features and scenarios but it mostly kicked in after we had written the code.

I believe it’s a bit different in the Developer division, because you’re close to developers. One of the challenges we have with SQL Server is that, for our real validation, you need to get closer to a production environment. And it’s very difficult to get the feedback from the production team.

Scott: Right. With SQL Server, a single dev banging away isn’t a real test. You could get somebody banging away at the feature, but that doesn’t really tell you whether it will scale, whether it will be reliable, whether it will be secure.

David: Right, exactly. So what happened with this release was, we had some teams that said, “Oh, crap. What are we going to do to get feedback for this really complex improvement?” For the database engine there’s an improvement called a Resource Governor. It allows you to constrain queries that consume specific resources so you can limit a particular connection to 10 percent of the CPU, and that kind of stuff.

The question is the mechanics of getting the architecture and user interface right. We think very hard architecturally about separating mechanism and policy. In this case, the mechanism would be things like how do we keep track of the various resources and who do we attribute the resource usage to. The policy would be how the user describes her intent in a way that allows the mechanism to enforce it. In the case of the Resource Governor, there’s a complex interaction between the policy and the mechanism, – not the least of which is that we needed to build new mechanisms to hold and enforce various policies. This is a great example, because what we would have done before was stare at the whiteboard, do a design, and throw something together. We’d ship it out in a Beta and hear, “No, that’s not what we need.” So it turns into a rock fetch. “No, this thing stinks. … This thing stinks. … You’re getting warmer …,” etc. It’s a pretty inefficient process all up.

In the case of the Resource Governor, we got together with some of our MVPs and key customers who had a need for the improvement and did more of an outside-in design process starting from use cases. What’s interesting is that this outside-in approach is completely natural for many people building applications, but for a development team that’s been creating low-level system software for 10-plus years it’s quite different.

The bottom line is that we definitely needed to figure out how to engage customers differently with our new development process, and there are opportunities for design councils, concept testing, early prototypes, and early builds. In fact, things like virtual machine technology allow an improvement team to create a build of their early thinking and test it with a small group in a focused way without having to integrate everything else that’s in various states of completion, like we used to do with the standard Beta process. They just create a VHD with their bits and have some customers try it. With this model they can iterate faster than was possible with the old Beta process. Ultimately, I think we’ll get better improvement feedback by finding the right 10 to 20 customers and working closely with them rather than putting it together in Beta and throwing it out to 100,000 people hoping they’ll give us valid feedback. I’m overstating things a little bit to make the point, but the general concept still holds, although not everyone here agrees with me on this one.

Sean Campbell: I’m curious about the point you just brought up, because Scott and I have been knee-deep in virtualization for a long time. The point you just made about letting people evaluate it on a VM, how has that impacted the overall development process for the SQL Server team? Has it given you the ability to basically give, let’s say, broken bits out earlier and get more people working with it earlier because they don’t have to install it, they don’t have to configure it? I’m sure in the past people would try to install it on top of whatever else they already had. You could tell them a hundred times, right? And they’re going to try to install it as a multiple instance on top of what they’ve already got.

David: It has, in exactly the ways you said. These applications are so complex, and the state that they put on the machine is so complex, and we work so hard to make sure uninstall really uninstalls all the stuff that it put there in the first place.

The other piece is that it’s hard for us to build and test the setup and the installation. With VHDs, we can basically handcraft them and then clone them, and they don’t pollute any other state on an existing machine. So it’s much easier to get together. The bulk of the savings is in the setup, for us to construct both the setup and the uninstall, to make sure we don’t pollute the machine state.

So I think from that perspective, it’s very, very helpful. I think we saw that, but we haven’t really perfected it as a process yet. I think we’ll probably go a little bit deeper on it in the next release.

Scott: In talking to a lot of people in the open source world, especially community-driven open source, like the Linux kernel and the Apache Web Server, I hear there is a pretty strong correlation between your ability to request a feature and your ability to actually code it. You will get on a mailing list and say, “Hey, it’d be great if the Web server did this.” And the response is basically, “Yeah, that would be cool. When will you have the code ready?”

What you have talked about is very different from that, in terms of interviewing customers, interviewing influencers, and putting together a team. Talk a little bit about how a specific feature happened from end to end, how the feature was initially conceptualized, how you validated it, the portions of the product it touches, etc.

David: Well, I’ll talk about it in the meta form, and then I’ll talk about a particular feature. It goes back to the thing I was saying about technology evolution. And it’s actually one of the challenges.

One of the challenges the open source community faces, and this will be a sweeping statement that I get flamed for, is that they don’t yet generally appreciate the difference between the consumer and the producer. By that I mean, as a technology matures, the gap between who’s using and who’s producing it becomes greater. It’s a challenge we face at Microsoft; we are no longer simply building products for people like us, the engineers. In one of the talks I give to the product development community at Microsoft, I stress, “Look. We’re no longer just building products by engineers, for engineers. The PC is moving toward a consumer electronics device for many users and we must build a product that interacts with them on their terms, not ours.” I then show screen shots of Task Tray bubbles talking about “Virtual Memory” and “Pagefiles,” etc. My friends and family shouldn’t have to know what virtual memory is.

We’ve seen the same thing in the database space. Twenty years ago, every DBA had to be a wizard and needed to know how to fiddle with hundreds of knobs, figure out how to manage raw disks, and learn all sorts of low-level details just to create and maintain a database. When we were building SQL Server 7.0, I used to troll the database newsgroups an awful lot. You’d see people asking perfectly reasonable questions and you’d see responses like “RTFM” or “You don’t belong here if you don’t know that.” So we started this campaign to fix what I called the “you dummy” questions. Basically, if an expert responded to a question with a “you dummy” tone, we saw it as an opportunity to address the issue by engineering—teach the product to do it rather than teaching hundreds of thousands of DBAs. We turned the “you dummy” on ourselves and did something about it.

I don’t have to be an expert on my automobile to keep the thing running, and that’s fine. If Toyota wanted to come to me, I wouldn’t have to talk to them about ignition advance, dwell time, and those things anymore. I’d talk more about my experience and what I wanted the car to do for me. I think that’s just an evolution, and you can certainly see it in the database space.

To get back to the specific feature, I think the Resource Governor is a good one, where there was a complex set of mechanisms inside our system that we needed to go off and engineer, but how we constructed those and how we presented them to users was the challenge, right? What is the policy? How does someone describe a policy around constraining resources in a particular query or session? And then how does someone bind different policies to different classes of users or applications or queries or times? What degrees or what dimensions do they need?

Another way to think about this, from the perspective of design, is that the software development community has not made the leap yet from expressing the product control surface as knobs on the underlying mechanism to capturing the user’s intent or objective and then acting to achieve that. A simple example is: Rather than having an administrator configure how many dirty database pages will trigger a database checkpoint, we can capture the user’s intent in terms of the trade-off between fast recovery in the event of a failure, which can be achieved by increased checkpoint frequency, or better run time throughput, which can be achieved by fewer checkpoints. Having a control that captures how long the administrator is willing to wait for recovery to complete in the event of a restart looks directly at the business intent rather than fiddling at the level of the mechanism. I mean, how many dirty log blocks can I have on a particular database and still have it recover in 60 seconds, anyway? Instead of publishing some equation in the documentation, capture the user’s intent and treat that as a constraint during run time.

So to go back to the Resource Governor—we go to the customer and ask, “OK. What sorts of things do you want to constrain? What dimensions are most important? Are they applications? Are they time of day? Are they this or that?” And then gather all that, design something that’s consistent from the user intent perspective, and then figure out how we build a mechanism underneath to marry up with it.

Certainly, there’s a lot of success in the open source community, using open source technology in consumer electronics. But I don’t think that they’ve gone end to end in terms of doing a great job of user-centered design.

Sean: Well, that’s one of the things I’ve talked to people about, too, and that I personally feel pretty passionate about, innovation in the usability area.

I am curious, from a closed source company perspective, about what processes you feel should be put in place in order for a development team to excel in terms of usability. You go out and talk to people about potential features to help make sure that a feature doesn’t turn into engineers building it just for engineers, and that you actually end up with something—like the Ribbon in Office—that honestly has been a success, if only because, after release, nobody complained about it.

It’s the proverbial tree that fell in the forest and nobody heard it, but everybody was freaked out about it. And I keep telling people, “Well, think about this. Microsoft changed the UI on the one application people use the most, on a daily basis, and yet nobody really complained once Office launched.” There must have been something that really went into the thinking, in terms of usability.

For you guys, especially with a product like SQL Server, where I’m sure you can lose yourself in B-tree discussions until the end of time, how do you ensure usability in the product?

David: It’s a great question and a great challenge. It’s funny, but one of the things I’ve been doing for awhile that’s had a good effect is to hold the mirror up to the product development community at Microsoft, to get them to see that we’re no longer building products for people like us.

I have a set of slides that I’ve been using for six years or so, where I have collected some crazy error messages and I have real-life scenarios. I’ve got a screen shot where, in Windows XP, if it extends to your page file, you get this balloon coming out of the system tray that says, “Your virtual memory is low. We’re extending the page file. Blah blah blah.” I came home one night and my wife hit this and she said, “Hey, I think my computer’s broken. It says something here.” So I started describing to her virtual memory and page files, and she said, “Shut up! Just fix it.”

It dawned on me that perhaps 99 percent of the people using PCs these days are not engineers, yet we’re still throwing these exceptions and doing these sorts of things as if they were. And we ridicule the users instead of ourselves. I go help my neighbors fix their machines, and they all start with, “Oh, I feel like such an idiot around computers.” And I just want to say, “No, you’re not the idiot. We are, because we’re not building computers for you yet.”

And so I have a set of rules, or things that I’ve captured, that I talk about, such as the “principle of least astonishment,” where a reasonable user action should do something reasonable with least astonishment. The example I use is another story from my wife. I don’t know how many years ago, when I first got her a PC, someone emailed her some pictures. She said, “How do I look at these pictures?” I said, “Just double-click on the filename right here.”

So, of course, up comes the picture in the picture viewer. And there are little buttons down on the bottom of the viewer: “Next picture,” “Previous picture.” What do you think happens if you click on “Next picture”? You don’t get the next picture in the set of attachments. You get the next picture of whatever the next file was in your IE cache or whatever it happens to open.

And she’s going, “What the hell is this?” She gets some random icon from the temp directory or something like that. So I started talking to her about temporary directories, the IE cache, and blah blah blah, and again she’s like, “Shut up! Just tell me why it’s broken.”

Sean: She’s saying, “I’m looking at photos. I want the next photo.”

David: Exactly. And you see that even in the database space. We’ve taken the market from tens of thousands of servers out to millions or tens of millions of servers. You can’t have every one of them require a highly skilled DBA. We just couldn’t do it economically. And so you have to get out of the mind-frame of an engineer and go adopt a perspective of what the customer needs and how the customer wants to express their intent.

That’s the other sort of piece I talk about: In terms of the degrees of control and the dimensions of control, get rid of all of those dimensions of control that don’t capture any user intent and figure out which dimensions of control are orthogonal with respect to the customer’s intent, and then figure out how you build your system, and express things that way. That’s been the most effective thing I’ve found to motivate the change.

The bottom line is that the software development community doesn’t teach “design,” certainly not with a capital D. Most developers believe that if they think for 10 minutes before coding up some function, then that’s design. It’s not. This isn’t a development methodology thing, but rather a cultural change that we need to go through across the entire industry.

Sean: That’s great. The chief creator of a project called Quicksilver for the Mac was giving a talk at Google, and I just saw the video of it the other day. His theme for the project is “act without doing,” which has an interesting similarity to what you are talking about. If you saw the app, you would probably see some interesting things in it.

To follow along that line, I have a question about Oracle. Oracle, through the years, treated the product as though the more complex the cockpit, the better it was. If you walked in and it looked like an old 727 cockpit with 50,000 switches, that was nirvana. And you were paid to know every position of every switch, right?

David: Yep.

Sean: The open source community has, to some degree, a similar bent, right? Not in everything, but in a fair amount of places. But even in a product like Ubuntu, which is seen as the ultimate of end-user experience—and it has made some great strides—there’s a massive list of steps just to join that machine to a Windows domain, for example, if that is something you wanted to do.

David: Yep, yep.

Sean: So, do you see any similarities with this line of thought in the way Microsoft helps you to administer a database and the way open source helps you with administration in projects like MySQL?

David: That’s another good question. Yes, Oracle—I think I mentioned it earlier, they marketed our ease of use against us, frankly. They said, “How can this thing be a real enterprise database product? It doesn’t have nearly as many knobs as ours does.” And when we left Digital—this is an interesting little story—we had a product, RDB, that had roughly as many knobs as Oracle did at that point.

And like engineers, we were saying, “Oh, shoot! Hardly anyone knows how to turn the knobs on this thing.” So we wrote more software, something we called RDB Expert, which was a physical database design and some other things, to recommend and actually turn some of the knobs for you.

I think the first step that you go through in this evolution is to put a facade on it, whether it’s a system that recommends turning the knobs or a GUI to cover up knobs in config files. But really, at that point, you haven’t got a good end-to-end design, because you haven’t, in my opinion, eliminated those points of interaction or control. You haven’t captured user intent.

It’s like having a TV that still has horizontal and vertical control, but you’ve put something on the TV that can watch it and turn the knobs for you. So I think the next step in the evolution is to have the system software itself become more adaptive.

Scott: That’s true. TVs used to have horizontal and vertical knobs on the back. They don’t have those knobs anymore, even though TVs are much more complex, and no one misses them.

David: And what you wind up with—here’s the key point, and this is something I didn’t understand a priori, but after the fact it really made a lot of sense. In SQL Server 7, when we did some of the memory work to have the system automatically adapt to the environment and adjust the amount of memory it consumed, we changed the system so that you could adjust the amount of memory that it was using dynamically. And the mechanism was dynamic as well, so we had an automated policy that was a closed loop, but for those people who wanted to get under the covers, they could change it themselves.

But the neat thing about the way we reimplemented it top to bottom, and not just put a facade on it, was that the underlying mechanism itself was dynamic. By that I mean you could change the amount of memory that SQL Server used while it was running and not have to shut it down and restart it to get it to adopt a new value.

It’s back to the separation of mechanism and policy. We made the mechanism dynamic, we made the policy automated, and, where it made sense to try to capture user intent, we allowed user intent to be captured and actually influence the policy. And that takes architecture, top to bottom. I think the evolution is: You recognize it’s an issue, you put a facade on it, whether it’s a GUI or whether it’s some other code, and then you have to step back and architect the thing, top to bottom, to do it correctly.

Scott: One of the attitudes I ran into back in my C programming days was, “Look, programming was hard for me. It’s hard for a reason. It should be hard. If it isn’t hard, anybody could do it, and that’s the last thing in the world you’d want.” I run into that attitude whenever I look at systems administration and other technical issues, where people say, “Look, this is complicated. It should be hard. You should have to be a little bit of a rocket scientist, otherwise you have no business doing it.”

So one of the complaints is, “Well, yeah, Microsoft’s made it so that any idiot can set up a Web server, therefore every idiot does, and they don’t keep it patched, and it’s vulnerable.” Same thing with the database: “Sure, people could install the database and set it up, but they won’t create their indexes correctly, and they won’t keep it patched.”

So there are certain people who would say you should stop more people at the front door and not let them in. Because if you do let them in, they get farther down the path, thinking that they know what they’re doing, and then they run into a disaster. What would be your response to that kind of attitude?

David: I think it all comes down to, or goes back to, design in the sense that if you have the means within the technology to bring it out to a broader market and let people do the job, then I think it’s incumbent on you to try to figure out a way to do that. It’s a great example and a great point around physical design of the database—how do I design the index set? It’s very difficult.

One of the things we did here is, we designed the Database Tuning Advisor. This was work with Microsoft research where we’ll look at a workload and actually propose indexes based upon the workload, and do physical design that way.

And the way we test that is kind of interesting. We bring databases in, we measure their response against a real user workload. Then we strip all the indexes and we use the Tuning Advisor to rebuild the index set. More often than not, the Tuning Advisor does a better job than the guy who gets paid a lot of money to do it by hand. Of course we can’t see the reporting jobs that are running at the end of the month if they haven’t been part of the workload, but it does a pretty darn good job. So I think it’s a matter of how far can you take the technology.

Another sort of thought experiment here is if you go back to the TV front. Go back to 20 or 30 years ago. Every small town had two or three TV repairmen. The technology is more complex now than it was, but it’s perhaps 50 or 100 times more reliable. So you don’t see the TV repairman in every small town anymore. The TVs just go and go and go until you decide you want a slick new plasma one. I think that’s the way software is going to move as well.

Sean: Obviously, one of the strengths of open source is the community. I think that is one contest they tend to win hands-down, if for no other reason than that open-source projects were driving community long before closed source companies were. But at the same time, I know Microsoft has been making some significant efforts in this area too.

You mentioned in the previous conversation about CodePlex. I knew about CodePlex, but I have not really looked at it with an SQL Server focus before. A brief glance at it showed me some of the activity that is SQL Server related. Talk to me a little bit about what is happening there, how that activity is affecting the product team, etc.

David: I agree with you. I think I mentioned earlier that tapping into the energy of the community is super valuable. I think of a lot of these things in terms of closed loops and how quickly can you run the loop, how fast can you get the feedback, and how fast can you incorporate it. Tapping into the community is a super way to do that. There are lots of examples of closing a loop, like Watson, to have the machine send back error information and actually close the loop and automate things to enable continuous improvement.

As I think about community, absolutely I give the open source guys credit for tapping into that phenomenon first, but I don’t think it requires open source in terms of the development or distribution model to engage the community. I think about community engagement in terms of layers. With the product itself there’s a way of tapping into the community by harnessing their feedback to improve the product over time. I think with the documentation content and general knowledge around how to use the product, there’s a deeper way of tapping into the community. As we publish content for the product, can we allow people to modify it, link to it?

Some of our content could provide a spine for community activity. For example, we could build a collaborative site around the product error messages and allow the community to link their responses into each error message to help one another out when they hit a particular issue. Today much of this happens in forums and newsgroups, but it’s not done in a way that closes the loop effectively. If we did it in the way I just mentioned, it would be much easier for the product development team to review and mine the activity to improve the product going forward.

I think the next level beyond documentation would be extensions and tools. Things that you could build around the product where the core of the product is still in a closed source model but you could open up and distribute add-ons, add-ins, and other sorts of tools and actually build the community around that.

Sean: This has been a great conversation, David. Thanks for taking the time to chat.

David: Thank you.

Bookmark this:

Interviewee: Justin Erenkrantz

In this second part of a two part interview with Justin Erenkrantz we talked to him about:

• How the Apache project ensures good collaboration.
• The Apache Foundation’s philosphy of having no single person as the leader.
• Apache’s security committee.
• The process of removing someone from a position of responsibility within the Apache Foundation.
• What would make someone want to be part of the Apache Foundation’s group of projects.

Scott Swigart:  In this part of the interview, we wanted to dig into some of the tenets, if you would call it that, of the Apache way. And the first of those of course is collaborative software development.

So talk a little bit, if you would, about how Apache does collaborative software development. I’m sure some things are very traditional and similar to the way that other open source projects might do it, and there are probably things that also might just be a little bit unique to Apache. So how do you try to insure good collaboration?

Justin Erenkrantz:  So the main center point of all of the collaborative software development that we do in Apache is the mailing list. That’s where pretty much everything happens. As one of the guys mentioned before, the maxim has been: "If it didn’t happen on the mailing list, it didn’t happen." And that generally tends to be true.

Basically, if you follow the mailing list for a particular project, then we’re expecting that you should know what’s going on within the project. Within that, those are pretty much all public lists. Everybody can subscribe, even people who are committers, people who are just users, people who just work on another project that may consume the Web server or maybe PHP modules or something like that.

It’s pretty much an open forum. Anybody can voice their ideas. Generally though, just the way things work, most of the traffic tends to be from the core developers who are active at that time. The traffic patterns of the list change, and you get an idea of how much discussion. You generally see peaks and valleys for the mailing list discussion. Things get really heated or things are just chugging along and there’s not much traffic on the list.

So that’s where all the discussion should happen. And then of course there is the source code repository. And in Apache we always had the thought of having shared repository. A lot of projects now are starting to ‑‑ Git and Mercurial and all of these distributed version control systems to be centralized version control systems.

And that’s something that, within Apache, that goes against what our thoughts are, because we want to all be agreeing on, "This is the Apache version."

Scott:  Sure, no problem.

Justin:  So there’s no leader within the ASF. There’s no person, if you look at say, Linux and you say, "This is Linux’ tree," or this is Andrew’s tree or this is Alan’s tree. Instead there is just the Apache tree. So there’s really no concept of, "This is Justin’s tree," or someone else.

Scott:  Is the feeling then that essentially there should just be intensive discussion before something gets checked in? So when you’ve reached a point of consensus, it should get checked in rather than the way other projects work where different things get checked into different people’s trees. And then when it’s time to build a release, you have to pull stuff from these different sources to figure out what’s going to be in the release and what isn’t.

Justin:  Right. So, generally, what tends to happen is there are two states a tree can be in within Apache. One of them is "commit then review." And then there is "review then commit." And you’ll see some projects differ on particular trees.

For example for the HTTP Server, the trunk ‑‑ which is the main development ‑‑ is usually always under the "commit then review," which basically means that anybody who has commit access can feel free to go and make changes and they basically have the benefit of the doubt that the change is going to be good. And there’s generally an implied threshold: that if you’re going to make a really big change, go discuss it on the list. But if it’s a minor change or adding a little feature, that’s probably not going to be controversial, go ahead and commit that to trunk.

But for our stable releases, generally things that have already been released and we’re doing maintenance on those, are under the "review then commit" model. So that’s going to be RTC, and that means that any change to those trees has to be pre‑approved. That means you need to get three binding votes from other committers to say, "Yes, this is good change and no one has vetoed it." Technically you would use a file called STATUS, just a plain text file that some projects will use, that basically tracks all of the things that are under discussion to be back‑ported or added into this tree.

Scott:  Got you. Like any open source project, and this one is democratic, there is a vote to commit. Well, there is a vote if it is a release product. If it’s not a release product and people later decide it wasn’t a good thing to do, it can be reverted out.

And you mentioned before that part of your governance is that out of these 60‑something maintainers, one of them can essentially veto a change if they want to. So talk a little bit about how conflict resolution usually works. For example, when you have people who are—and I understand it doesn’t happen often—adamant one way and another person who’s adamant a different way. How do you see it play out that those eventually get resolved and things move forward?

Justin:  Generally what happens is, like before, the veto just tends to be a last resort. So let’s just say that someone makes a change to the trunk and I don’t like it. I might just say, "You know, we should talk about this change," or, "Here’s the problem with this." Generally, the person who has committed that says, "Oh, yeah. Here’s why I did it this way," and comes up with an explanation, and then through a process on the mailing list, figures out and resolves those conflicts.

That’s what you tend to see happen. And it’s all tends to be, for the most part, "I’m sorry, I forgot about that particular corner case or that." And everything tends to get resolved very naturally.

The veto tends to be when someone says, "No. I have to do it this way," and someone else says, "No, that’s wrong." That’s basically when the community is at risk of breaking down. But generally it doesn’t get to that point. Everybody is, "We’re all going to go in this direction; this is the right direction for us to go in. We want to add this feature. Let’s work through whatever issue you may have about this particular commit."

Scott:  One last thing before we move on to the next point. Other than the fact that everybody has their own private tree, is there anything else about the collaborative nature that you think is somewhat unique to Apache?

Justin:  We tend to do a roll call before the release. So at this point there’s been a review of everything. But then let’s just say that I want to release the next Apache 2.4 or whatever. Then basically what I will do as a release manager is say, "OK, I’m marking this as 2.4," and I produce all the artifacts. I produce the tarballs, [inaudible]generated files, whatever. And then I send it to the list and say, "Hey, is everybody happy with that?" And then that goes to a voting process.

There’s review at all stages, but there’s also a review at the point where you do a release, and you have to get, at least, three people to approve a release. One thing that’s different is that it’s not possible to veto a release. It’s strictly majority rule on the release.

Scott:  So in other words, there is veto capability on the individual check‑ins. But, as you said, when it comes down to doing a release it is a majority rule vote.

Justin:  Yes. You’ll tend to see someone voting ‘no’ on a release if it doesn’t work on Linux or something, and generally you’ll see it get stalled and it then gets fixed. But there have been a couple of cases when we did the release even though we knew that it didn’t work on a particular platform, and so we made a release note. But the veto does not apply to releases.

Scott:  Interesting. So moving on to licensing, one of the key things with Apache is the commercial‑friendly standard license. Talk a little bit about what that means.

Justin:  Basically within Apache, we like to have a big tent where everybody can come in and play with us. We think that the community that we developed within Apache is going to be the motivation for you to stay involved. For example, within the HTTP Server community, you have all these experts and Web servers, and if you’re part of this community, you get the benefits from them. And so there’s an incentive to play within the community, so that I don’t have to hire five guys and do a whole team; I can leverage the other people within the community.

But in turn, all those people who are part of the community say, "Whatever you want to do with the code is fine. We’re not going to get hung up if you make it a commercial product or an open source project. We created it and it served our needs, and if it serves your needs, that’s great."

Scott:  What springs to mind are things like GPL. Am I seeing it right in that Apache is more commercial‑friendly than GPL, V2 or V3?

Justin:  There are companies built around GPL licensed software. But what we tend to see are two classes of GPL products: In one, there is a real community, maybe within Linux, and they’re all happy to make all their changes available to everybody, and that’s a very good community. The other community you tend to see has a single stakeholder that has a prevailing interest in the GPL product, and they basically have an unfair share.

You can see this with some of the GPL projects that require copyright assignment. In order to participate, you license your changes in the GPL and you have to give a copyright assignment to the principal stakeholder. Now they are then free to release the commercial closed source based on your work because they have the copyright or whatever legal mechanism. There is an imbalance there when you look at those two.

Generally when you think about the GPL, you’re divided, with broad strokes, into those two groups. This is a real community but the other groups are aware and want to be clear about which one has a dominant role, and that’s one thing within Apache we don’t like to see. As our projects go through incubation and get added to the Foundation, one of the things we do is make sure that the community is diverse. In fact, there is not a single dominant stakeholder that can direct the project in any untoward way.

Scott:  Right. I can make changes to it, distribute it as part of a commercial product, and I would not be required to contribute those changes back to Apache. But under a GPL license, any modifications made require you to make the source code available. You cannot have closed source proprietary extensions or modifications of it. Any modifications you make, you have to open source and it has to be under the same license. So that’s the key differentiator?

Justin:  Yeah, our philosophy is that the community is what’s going to bring you and keep you there, and that’s why you’re going to stick around. If you released a commercial product around one of our projects it’s going to be to your benefit, to basically keep your commercial project as close to whatever we’re releasing.

Scott:  Right.

Justin:  You can pick up all the bug fixes and whatever improvements; you get those as a free rider. But in a sense, you are contributing whatever changes you’re making voluntarily back into the greater community.

Scott:  Yeah, that makes sense. Do you have examples of companies that have used different Apache projects because of the commercial‑friendly licensing, where they probably wouldn’t have it if the license weren’t so commercial‑friendly? Is that a topic that comes up?

Justin:  Absolutely. You see companies like IBM that release their versions of the Apache Web server or Geronimo under different names, but in the core, they are Apache projects. We’ll see that even with smaller companies such as Covalent that does commercial support. Basically, they added in a couple of extra things that provide support to their users.

One thing that the Apache community really does not focus on providing is 7/24 support. Covalent goes in with their business model and provides the support and training around these particular Apache projects. You will see businesses like JBoss using Tomcat. So you see all of these commercial companies using things that are Apache projects under the covers.

Scott:  Right, so that freedom has led people to be a lot more creative about how they structure their business. They have a lot more options in how they participate with the different Apache projects, how they contribute back and how they structure their own products. What is the relationship between the Apache Software Foundation and the Free Software Foundation? Is there any or are those fairly separate endeavors?

Justin:  There is no formal relationship. I’ve never had a conversation with Richard Stallman, but I’ve had conversations with Bradley Kuhn who used to be, at that time, the Executive Director of the Free Software Foundation. So there’s an informal get‑together of foundations to compare notes, and that’s generally a very good thing. How do we keep our ears open to what Mozilla’s doing? If they’re doing this new technique, then we can give them a call and ask, "What are you doing? We’d like to follow on it."

One thing we’ve been doing with the Eclipse is a joint conference. There’s going to be a conference in Asia that’s now scheduled for 2008. So it’s a way for us to get the communities talking to each other.

Scott:  Sure. So basically, if I can summarize, you guys get together around joint events and joint things where it makes sense. You share information because you’re all part of the open source community. Philosophically you may agree to disagree in terms of the details of licensing, commercial friendly, and that kind of stuff.

Justin:  Well and you ought to be using more…projects have different circumstances. Apache‑‑we have a very vocal membership and we have this and you compare that to let’s say the Mozilla which has a completely different governance structure. But if you look at Brian Behlendorf, he’s been on the Mozilla board for a very long time and he was one of the founders of Apache.

Scott:  Gotcha.

Justin:  So you have this intermingling of the communities. So someone like Brian Behlendorf who was brought in through the Mozilla and says here’s how we did things within Apache, and here’s his expertise and his experience that he got, he can share that with the other people within Mozilla.

Scott:  Gotcha, gotcha.

Sean:  Let me refer to one of the other tenants of the Apache way. I’m curious about this just because I was thinking about the conversation we were having. To state the obvious, you’re focused on producing software. I notice that you have a security committee that—if I’m reading it right and for lack of a better phrase—provides a service to all of the projects that are part of the foundation. And it looks like those projects can turn to the security committee and ask security related questions or possibly look for guidance from them, regardless of whether they’re Tomcat or some other piece of the foundation? Is that accurate or is that not accurate?

Justin:  It’s somewhat so. We have a security team, which I believe is currently a Board committee. But basically what they’re responsible for doing is ensuring our security at Apache.org mailing address gets responded to. And these are generally people who are very security savvy.

But there tends to be some people from Tomcat, from the HTTP Web server, from a higher profile project on this internal mailing list. So let’s say that, to give you an example, let’s say there was a security vulnerability in Derby and they could parse to those reps and say, "Hey, we have a security vulnerability. What do we do?" And so there’s expertise and, "OK. Here’s what you do. Here is your administrative contact. Make sure your mailing list… Go talk to…" Kind of a shared resource. But we’re not getting the focus on producing the fixes for the project but it’ll be "OK, here’s the responsible disclosure policy and an attribution policy." So that’s generally what their role is.

Sean:   Are they providing fairly prescriptive guidance but just not down to the ‘I’m going to change you’re code’ level because they don’t know the individual projects at that level? Would that group essentially be the center for discussions around a security development lifecycle for the Apache Software Foundation? And an attempt to pull those best practices together?

Justin:  Yeah. I think basically our project concern…we have something. What’s the process? What do we do? And that’s as an advisory role. OK, here’s the process and the procedures to follow.

Sean:  But it’s purely advisory, right? I mean one of the projects where they feel that their code is "secure enough", or they’ve looked at it long enough or they feel that they’ve handled it.  Then the advisory committee comes back and says, "Well, we really think you could take a look at this again." That’s where the communication would stop and it would be up to the individual project whether they want to take that under advisement or not.

Justin:  Yeah. I think so. I think record security, can you maybe at that point write back to your original reporter and say, "We looked at it and we don’t feel there’s a security vulnerability here." That may be…that has happened where we look at things and we say, "No. This is not an issue." But generally, really the security team is more of a reactive. So they’re not proactively performing security analysis on our code or anything like that.

Scott:  I just want to clarify. It sounds like they have a little bit of an all‑up policy for somebody sending an email to that address; somebody reports what they perceive as vulnerability or reports some kind of issue. They do a little air traffic control. They route it to the project.

Justin:  Exactly, exactly.

Scott:  There’s a general process that the different projects would follow. Basically that sort of happens and that’s one of the things that the security group advises the other projects on. Well this is generally a ‘way we do it’ sort of thing.

Sean:  Let’s go to a different piece of the Apache way, the emphasis on a technical‑based interaction. One of the things that I find fascinating about open‑source projects is the way that they exorcise community members that maybe aren’t following those rules.

[laughter]

Sean:  Because we got some interesting responses when we talked to people about it. It’s like, fine. We understand that everybody is an adult. We understand that everybody will try to handle themselves in an appropriate manner. But if anybody’s worked on a software project they know that not everybody does, right? So…

[laughter]

Sean:  Considering that you can learn a lot from a story…if you’ve got a story or two about either fully pulling the ejection handle on somebody that would be interesting to hear? Or a scenario where it just took serious counseling to get somebody pointed in the right direction.

I would be curious to see how you guys handle that. You obviously have procedures in place. But at times you have to go beyond those with some amount of intervention and I’m just curious how that played out.

Justin:  Yeah. So there’s one case that comes into mind but I’m trying to reserve the right to figure out how much of this has been disclosed.

Sean:  Yeah, Sure

Justin:  So I’ll tell the story and then leave people’s names out of it but I have to go back and see how much of this has been told. So recently within one of our larger, well‑known projects, there was a bout, to use the word, between two committers. And they basically ended up vetoing each other on everything. It’s like no, no, no and tempers got flared. And it got into a very unhealthy situation. They’re two very strong‑willed individuals.

And basically what happened is the PMC, so it was the PMC responsible for this particular activity, basically had to step in and say, "OK; we need to come up with some policy or come up with some new rules to get everybody back to ground zero. So it wasn’t a matter of ejecting anybody. It was never really an option that…basically what happened was they said, "Here are the ground rules. Here is…if you’re going to go do this you have to follow this set of rules. If you’re going to go do that, you’ve got to go follow this set of rules." Basically the community agreed to say we’re going to go and we’re going to voluntarily adopt these rules. But as a settlement process; lots of flames and a lot of innocent people getting…

Sean:  Right.

Justin:  …accused of things and that process. The other one that’s in our history was a project called Avalon. And this is one that’s definitely well known so this won’t be any issue about this one. Avalon was a container framework. And what happened was two individuals just did not get along. And they were ending up in what we would call a commit war, where they would basically be reverting each other’s changes as soon as they came in. And it is just this whole really poisonous environment and basically in that case, the community wasn’t able to deal with it. And so basically what happened there was they fractured.

And so that one of the individuals went off and he took his code and you know, we wished him luck and said have a nice life and he went off and then it was kind of some other people came along and they did a project called Excalibur, which was basically the remnants of this whole Avalon project. You will just see if you look at the mailing list traffic, you will just see this giant peak and then this sudden nothingness because the project got shut down because no one could play well with each other.

Sean:  Right.

Justin:  By the way what it is interesting is some of the veterans of Avalon, they really got involved in the greater Apache community, one of our new directors this year. He was in the middle of all of this, but during that whole experience, he was one of the people trying to keep things level and stuck around and this year he got elected to the board of directors.

Sean:  Since Apache is a large foundation, I’m curious about a different point. If you were a closed source company and you feel you are short on testers or short of security experts, you simply go to HR and put out a requisition and hopefully some good folks come back.

So has the foundation had to answer requests from projects where they say, "OK, look, we think we are geniuses on nine out of 10 of the things we need to do, but this one thing we really need people to help." How does the foundation help with staffing up a project in this type of case?

Justin:  It is more bottom‑up than that I think in the sense of the culture that we have. You will see some overlap between the HTTP Server committers and the Tomcat committers because you will see that ‑‑ sometimes the Tomcat committers came over and they say, "Hey, we need some help with HTTP." Well lucky us, we have some of the world’s foremost experts in HTTP server, and that basically got them within the communities.

So I think our community’s diverse enough to, "Hey I am looking for a person who knows SQL." OK, I am going to go on the Derby mailing list and say, "Hey, I need some help with SQL" or for something for build systems, I’ll go to Ant community. And there will be some of the people who are the foremost experts in that. That’s actually one thing of having such a large diverse community is that you can pretty much find someone who understands something about something somewhere within the Foundation.

Sean:   I figure that’s one of the advantages. You’ve got a massive talent base but at the same time it is segmented into the project, so the Foundation can help orient a little bit of that knowledge of where the talent base is.

Justin:  Yeah and as I said if you look out the social graph and it is a weird mix of people who are committers on Cocoon, maybe committers on Mina, then maybe on Gump and so you will see that developers themselves, the committers aren’t necessarily staying in their silo, there are some who do, but there are also just as many who will go to other communities and work on other projects.

Sean:   Well I have couple more questions but Scott can go ahead. I want to give you an opportunity to jump back in.

Scott:  So talk a little bit about standards because one of the other tenets or pieces of philosophy is faithful implementation of standards. Talk a little bit about what that means for Apache?

Justin:  Right. So this was initially when we started off there was HTTP and there was the IETF standards and that was when you have editor of the HTTP standard is one of the people behind the code base, there is the knowledge is going both ways in a sense is that we are that able to influence the standards process. But at the same point we also have some of us involved with the standard process and feeding those changes back and to the development and supporting those standards.

But since then, the initial, you see our participation within some of the key web server specifications, then probably most importantly our participation in the Java Community Process and that is, as you know, we have so many Java projects and that so many of our projects are implementing some JSR specification and our involvement within the JCP has been to ensure that we can implement the specifications and we have projects have representation on these expert groups that device these standards.

Scott:  Right. So it isn’t just like the standard shows up and then you figure out how to implement it. There is this two‑way street where you are shaping the standard because you guys have such a big, real world implementation. And meanwhile, the standard is telling what you guys do because they have their own stakeholders, but they are considering your recommendations and you want to conform to what they eventually approve.

Justin:  Correct.

Scott:  Yeah, I don’t know. I don’t have anything else specific. Sean, do you?

Sean:  No, not right now. I think this led us into some new stuff and from our end, we really enjoyed chatting about it. Justin, do you have anything you’d want to add or things you think we should address overall based on the theme of where we were going?

Justin:  No, I mean you did a good job of asking me the questions.

Scott:  I guess there is one final question. When somebody is starting or has an open source project, they can pick the license they want  they can do what they want for their community, things like that. What makes people want to be, in your mind, an Apache project? What is the draw I guess?

Justin:  Well, I think from my perspective, the draw is we handle a lot of the mundane governing structures and all this and the infrastructure and the licenses. And that is all essentially managed and I think you see a lot of open source projects like, "Oh we need to go get a foundation."

And that’s a lot of overhead, there is a lot of overhead to create a corporation, handle donations and handle essential infrastructure and that is what our goal at the Foundation at the broadest level is to provide support. So that these people who are working on all these different projects all they have to worry about is doing code. They don’t have to worry about, "Oh I need to go and buy a new server, how we are doing to deal with this donation or this tax policy." We try to deal with all of that. So I think there is a critical mass that works in our favor.

Scott:  Right, right and let them focus on the piece of it that they really enjoy, which is whatever this project is that they have come up with. They have a passion, like you said, for not having to worry about all the housekeeping stuff.

Bookmark this:

Interviewee: Justin Erenkrantz

In this interview with Justin Erenkrantz we talked to him about:

• The Apache Foundation.
• Project Management Committees and the Apache Foundation.
• Some of the reasons Jason feels Apache has been so successful.
• What other open source projects might look to Apache for in terms of inspiration due to Apache’s longevity.
• The Apache Incubator and it’s role as part of the Apache Software Foundation.
• What is the on horizon for the Apache Web Server.

Justin:  So my name is Justin Erenkrantz. I’m currently president of the Apache Software Foundation (ASF), also on the board of directors. And I’ve been a contributor to the Apache HTTP Server, the Apache Portable Runtime, and Subversion, and some other projects for quite a while now.

Scott:  Talk a little bit about Apache and talk about how it’s built.

Justin:  The Foundation as a whole has over 50 different projects. There’s the Web server, Tomcat, SpamAssassin, Geronimo. There’s a whole variety of projects. So there’s an overall Foundation, with committers for each project. They’re relatively isolated. The fact that I have access and I work on HTTP Server doesn’t mean that I have access to, say, Maven.

Each project kind of gets its own merit. Our culture is a meritocracy, and so we expect people to show up on the public mailing lists and start contributing, and eventually they’ll be recognized. Eventually they’ll get a vote, and this vote will allow them to be able to commit (code). Across the Foundation, we have about almost 1,600 committers that can commit to some of the 50‑something projects. And with that, you get to make changes.

You also get something called a veto, which is something that we can probably talk about a little bit later. That’s one of the core governance structures that we have. There’s also a thing called the Project Management Committees, and those are the groups that are responsible for each one of those projects.

Above that is the board of directors. And in comparison with some of the other open source organizations, our board at Apache doesn’t get involved with technical details. We’re not going to get in and say, "Oh, you need to change this variable name." That’s not at all what we do. We’re just there to make sure the organization is running, make sure that everybody’s happy and getting along. You won’t see a director really getting involved in a technical discussion, unless they’re a part of that project to begin with.

Scott:  Does the board provide more of a steering functionality, then?

Justin:  No, not even that. We’re almost completely hands‑off. There are times when personality clashes happen in each project, and try to mediate situations. Again, we’re not going to make technical decisions.
There are some things that are centralized. So one of my responsibilities, as president, is I’m responsible for the day‑to‑day operations of the Foundation. We have a Subversion server. We have issue tracking. We have websites. All of that is centrally managed by our infrastructure, too.

Scott:  One of the things you mentioned was votes and vetoes as the way the project is governed. So expand on that a little bit. How does that come into play?

Justin:  For example, there’s the HTTP Server project. Within that, it’s governed by a Project Management Committee, a PMC. Looking at your past interviews, other people have mentioned this. This kind of structure has replicated itself on how we have it at Apache.

These are groups of committers who are responsible for the project. On HTTP Server, there are maybe about 60 to 70 people on the PMC. Every single one of those people has what we call a binding vote. The votes are used in two main ways.

The first way is for release. Within Apache, across any of the projects — this is one of the hard and fast rules — there must be three binding votes before it can be released. That means we must find three people out of 60 people on HTTP Server to say, "Yep, this is a good release. We’re going to put the Apache brand on it. It’s going to be the ‘Apache HTTP Server’." That’s the release part of the vote.

The other part is a veto. If even one of those 60 people say, "This change is bad, I’m going to veto it," that means the change doesn’t make it in.

One of the ASF’s founders, Roy Fielding, refers to it as a kind of a shot gun. It’s kind of like, "OK, I’m done discussing this with you. You’re not listening to reason. Veto. Stop. We’re not going any further."

And it’s usually a last resort. Vetos are uncommon. They’re not something that happens every week or every month. Generally, if there does happen to be a lot of vetoes, it means that people aren’t willing to compromise. So, that may be something where maybe the Board of Directors might say, "Hey, you know, we might keep a close eye on it. Is there anything you need to talk about, do you need any help to resolve this…"

But, generally vetoes are relatively rare but they do give a power to the 60 people to say, "You know what, there’s not going to be any change that I disagree with, there’s not going to be anything where I say ‘Oh my God, I can’t live with this change being made’." So that’s an enormous power and it’s given to the members of the PMC.

Sean:  We found these nuclear options in place across many projects, but they don’t get used very often. It that involvement in open source takes considerable time and you do it because you believe in moving the project forward. It seems like going into this, people know that the only way to make progress is by consensus. No one person is just going to get their way. And if they really want to, they have another option, which is to go fork the source. But that has extremely high barriers also.

Justin:  Yeah, absolutely. But as I said, generally the vetoes tend to be very rare. It’s almost like mutually assured destruction. That’s the point. One thing that has happened a couple of times over the 10 or 11 years of the HTTP Server Project, is where there were vetoes on both sides. "We’ve got to do it this way. No, we’ve got to do it that way". There was this huge flame war for a couple of weeks and then they finally said, "You know what, we’re not going to agree". They even had a telephone call. They were doing everything, it was just this big mess. And eventually they said, "You know what, we’re just going to leave it to a vote. We can’t agree, we need to move forwards, this is blocking us. OK, we’ll go ahead and resolve this, whatever way the vote may turn out".

Scott:  Talk a little about the history of Apache. How did it get its start and what are some of the major evolutions it’s had getting to the present point?

Justin:  Apache has its roots back in the early days of the World Wide Web. The story begins with NCSA Web server from Illinois Urbana‑Champaign. They were running the NCSA Web server. Eventually, a lot of people left to go to Netscape. The NCSA code eventually got abandoned, more or less. I think there were nine people who found each other on Usenet and said, "Hey, I have a patch for NCSA. OK. Why don’t we start trading patches?"

They got together and they started to exchange patches and start coming up with a new version of this NCSA service. They started taking it in this new direction. They started saying "OK. Maybe we should get another group going." So they founded something called the Apache Group. It was an informal thing. They did that for about four or five years, starting in ’93 or ’92 (started in Feb ’95) Eventually they got to the point where other people said, "Hey, we like what you are doing."

By this point Apache had already gone through, [inaudible] a Web server, and made it up to version 1.2. They created The Apache Software Foundation in 1999, and started doing things besides just a Web server.

That was the start of the Apache Software Foundation. The early initial project was the Web server, and that is still what a lot of people think Apache is. Now you have close to 60 projects.

Scott:  Focusing on the Apache Web Server, what are some of the secrets of its success?

Justin:  For the web server, more than anything, it’s been the way we designed and supported all the standards. And it’s free. That was the tagline that Roy Fielding had on his website for a long time, "Apache, the best web server money can’t buy."

One thing that really speaks well of our community has been the lack of forks. The community embraces anybody who shows up. The project has evolved and widened, from just the Web server itself, where people have wanted to do new things. People wanted to do an FTP server, a mail server. It can do all those things today. The community has been characterized by being willing to be open to just about anything.

Scott:  It seems like open source projects that are modular do better because working on the core of an open source project might have a high bar. If it’s modular, you can write modules without going through the scrutiny of submitting code to the core. Modules give you a way to get your feet wet and participate. At the same time you need a really healthy community too. It is the personalities and way the governance structure is set up around it. It has to be really healthy as well. Just to say it back in my own words, those two things seem like they came together with Apache…

Justin:  Yeah, and if you look at one of the key evolution points between the original NCSA server and Apache, it’s when (early Apache developer) Robert Thau modularized the whole thing one weekend. By and large, most of what he did 10, 12, 13 years ago is still present in the code base and technical architecture. By modularizing, he did a really good job of cleaning up the earlier NCSA code base.

Scott:  So what is version 2 all about?

Justin:  Version 2 was all about threading and portability. With Apache 1.3 they added Win32, Netware, and OS/2 support.

Version 2 started out with a number of internal forks. One of them looked at Netscape’s portability runtime. Other developers did their own portability library, implementing the same function on three platforms and hiding the implementation details."

There ended up being a licensing dispute with the Netscape/Mozilla guys that prohibited the Apache guys from using the NSPR runtime. That spawned the Apache Portable Runtime project. That’s a lot of different projects now, but if you look at why it happened, it had to do with the licensing issue. If you look at the Foundation now, I think one of the things we are well known for is the terms of our licensing.  It’s a key differentiator from, say, the Free Software Foundation.

Scott:  Right, right.

Sean:  Apache’s been around for a really long time, and it’s obviously seen as one of the more successful open source projects, to say the least. What do you think other open source projects look to Apache for in terms of inspiration when they’re starting up?

Justin:  I think by and large, what you see most people copying are the governance structures and the licensing. Those are two things projects have been copying. I think you can see that in Eclipse: they almost use some of the exact same terminology.

Scott:  You still there?

Sean:  One of the things you mentioned earlier that has always been intriguing to me, was portability. I can see that it’s really important for Apache to be portable between different Linux flavors, and maybe even be portable to embedded devices and things like that. How important is it, from a practical standpoint, that Apache runs on more than just Linux?

Justin:  Extremely important. We have contributors who are only interested in supporting a NetWare or Windows or OS/2, even BeOS in the past. It’s where we’ve gotten some of the diversity of the community. It’s a hook to get people into the community. "Here’s a little something I know about, I know my operating system, and I’ll contribute this patch. Hey, there’s something else that may not be platform‑specific."

Scott:  Does Apache take the standpoint that it should run equally well across operating systems? OpenOffice, for example, wants to be pretty much the same OpenOffice regardless of where it’s running. Does Apache run differently depending on…

Justin:  Absolutely differently. Basically, our approach is in whatever platforms people want to maintain, that’s what gets supported. By and large, on the Apache HTTP Server, we have one guy who does the Win32, and it’s been his baby for many, many years. There are other people who contributed a little bit to the Win32, but he’s this one person who had been the individual who is responsible for it.

It’s not a dictate that, "Oh we have to support that." If someone is interested in supporting an OS, great! We’re not going to stop them, but it’s not going to be a mission statement, that we have to support all these platforms equally.

Actually, if you do look at our HTTP server mission statement, it says, "Apache HTTP Server Project is an effort to develop and maintain an open source HTTP server for modern operating systems including Unix and Windows NT." So, it’s in our mission statement, but the only reason it’s there is because we have the contributors to provide that support.

Scott:  One other thing that varies from project to project is where the code comes from. If you take a look at MySQL, pretty much everybody working on it works for the MySQL company. If you look at other things like the Linux kernel, a lot of that comes from corporate developers: IBM, Red Hat and a lot of people. Do you have a sense for where the Apache code comes from? How much of it is from corporate‑sponsored developers versus the proverbial guy‑in‑his‑garage?

Justin:  I think it comes from a wide number of sources. What you will see is that contributors remain the same even when they move from job to job. That’s definitely been the case within the HTTP Server, that’s been the case for some of these older projects as well. One day they may be working for IBM, the next day they may be working for Red Hat, and then they may be working for some other company. They may be working for Google, maybe doing it on the side, that’s what you tend to see. Some of these contributors may have started out working at Sun or HP, then they move but they’re still working on it. They still contribute to the project.

Scott:  There are certain people who look at open source and they think it’s all written by people in their garages, contributing. Other people look at it and say, "It’s all written by people working for corporations." How important do you think big corporate sponsorship is to a project like Apache, and does that also create certain challenges for the project?

Justin:  It’s a balance. You see some people who are getting paid to work on it. They work on it all day during normal business hours.

Then you see people who are the exact opposite, who may be working as a system administrator or something else, and they only time that they can work on it is on the weekends. So you see the overlap.

One of the key things in Apache, another quote from Roy Fielding is, "If it didn’t happen on the mailing lists, it didn’t happen." All of the discussions, all of the decisions, have to be made on our published mailing list. That allows people who may be in different time zones, or different work schedules, to coordinate through this mailing list.

They can read it during the day when they’re at work, during the night when they’re at home, whatever works for them. That way, decisions aren’t made in a face‑to‑face meeting, or a call, or an IRC, all the decisions have to happen on a mailing list.

Scott:  So IBM, just to take a big company name, can’t get something into Apache just because they’re IBM and they want it. If one person out of sixty people vetoes it, it doesn’t really matter how badly a big company wanted certain code in, it’s not going in.

Justin:  That’s right. The other aspect of it, the thing Apache has been addressing the last couple of years, is how new projects come into Apache through something called the ASF Incubator. This is about how they operate as an Apache project. They have to get all of the legal paperwork in place, so we can say, "Yes, we can release this under the common Apache license." That’s how we are trying to get new projects, and that’s why you’re seeing growth in the number of our projects, because incubator keeps spinning out new projects.

It’s always a concern that in order to graduate from incubator and become a full‑fledged project, you have to have diversity. Basically, you can’t have any one company dominate the project.

The rule we use, that you see pop‑up again and again in Apache, is the rule of three. There must be at least three committers that are diverse. The discussion that is going on right now is, "What is the definition of diverse?" An example: "Well, I work for IBM, and I work on this project full‑time, but there’s another guy from a completely different division who isn’t getting paid to do this who’s also working on it." Should that be counted as a separate individual? That is a discussion now. Some of these companies are so big, it’s like the old joke of, "Oh, you’re from London!  You must know so-and-so."

Scott:  In open source projects what gets checked in is the source code. With Apache, it looks like there’s this thing called the Apache HTTP Test Project.  Is that essentially like a test suite for Apache?

Justin:  Yes. Yes it is.

Scott:  OK. And what’s that focused on? Is that mainly functional testing?

Justin:  Yeah, it’s basically a Perl‑driven test suite, originally from the Mod Perl guys. They had this whole Apache test tool kit that they used as a kind of smoke test. And we said, "Hey, we’d like to take that." And we extended it from there.

Generally, what you’ll see is you’ll see people will use that as a kind of smoke test before they do a release. We talked earlier before about that you need to have three plus one in order to do a release. But we haven’t said anything about how people make up their minds, and say, "Yes, release this." And so generally what people do is they run tests on their favorite platform.

One of the things that we did with 2.0, and still do to some extent is "eat our own dogfood."  In the early days in of the 2.0 series for the Apache HTTP Server, we would say, "OK, we have a release candidate. We’re going to put it up on Apache.org. We’re going to go run it for 72 hours and it can’t crash."

Basically that was kind of another way of doing the acceptance testing. Saying, "OK, we can run it on a site that gets this much traffic. It didn’t crash so it’s probably going to be OK for you."

Scott: Are there tests specifically looking for vulnerabilities like buffer overruns, or is that really outside of the scope…

Justin:  In the past when there’s been some type of buffer overflow, or some type of CDE vulnerability, generally you write and check in a test to make sure it won’t show back up in regression.  I think that basically depends on if we can come up with an easy reproducible test case.

But I think you won’t see some test cases there that are typically for the vulnerabilities.

Scott:  A while back we talked to Michael Howard, who’s a security guru at Microsoft.  There’s a lot of things they do, but one in particular was banning certain APIs like strcpy because they were just inherently vulnerable.

Justin:  Yeah. Basically we do some things and put them in Apache 2.0 with the APR path (code base). [sp]
If somebody actually tries to call these functions, it’s going to expand onto, "Why are you trying to do this?" There have been some cases where we put it in the file to say, "Don’t do this. Don’t call this." Or say, "Oh you’re going to call this? Well then we’re going redirect you to a safer version."

But, we don’t have the flexibility of say a Microsoft, and say, "Oh we want to have this new security API in the operating system." That’s not something that we have influence on.

We generally have to look into the constraints of the operating system and work with that.

Scott:  Sure.

Justin:  Over the past few years, I think there have only been one or two cases where there were remote root exploits, and that speaks well for us.

Scott:  When people are posting patches, are the security implications discussed on the mailing list?

Justin:  Oh, absolutely. You’ll get people saying, "Hey there’s something with this vulnerability, or this will break this or that."  So, yeah there’s this constant vigilance for the security.

Scott:  Are there a lot of security-related tests that are put in proactively? I hear about things like "fuzz testing" and other proactive ways to probe the surface area for vulnerabilities. Does that kind of thing happen….

Justin:  There are security product providers, Coverity is the one that pops to my mind. They’ll say, "Hey, we ran our tool on your code and here’s a report of vulnerabilities." We take a look at the reports and say, "Thank you very much." And then analyze them ourselves.

But it’s really triggered by what the committers are interested in. We’ll see committers who are very interested in conformance to the protocol specs. We’ll see people who are interested in security, people who are interested in performance, etc.

But we don’t tell the committers what to be interested in from the top‑down. It’s more like, "John is interested in security so he’s really focused on tying up all the security issues."

Scott:  I’ve bumped into companies like Coverity that use open source to market their tools because open-source provides a large, free code base they can throw at their tool.

Justin:  Absolutely. Yeah, I remember when we first looked at Coverity, the amount of false positives were generally high.  We’d look at the code and determine, "No, there isn’t a vulnerability there. What the tool is reporting can’t actually happen." There were maybe a handful of actual things that we said, "Yes this is an issue".

They may not have been as severe as what the tool was claiming, but we said, "OK, we’ll clean this up."

Scott: What do you see on the horizon for the Apache Web server?

Justin:  The IETF is forming a new working group to do an editorial revision of the HTTP stack. Their work might lead to the next generation of the HTTP protocol. And I think that is something that we will be very much involved with.

Scott:  What about in the incubator? 

Justin:  Every Board meeting we’re graduating things like ServiceMix which is an ESB and component suite based on the Java Business Interface. One that is going to be new, probably the next board meeting, is a standard C++ library, which we’re getting from RogueWave. There’s things like Abdera, which is an Atom feed. We’re seeing a lot of things like ActiveMQ which is event‑based messaging.

What you probably see a lot in Apache are low‑level infrastructure type things. You’re not going to see things like, say, OpenOffice. You’re going to see things that people can pick and choose to build larger applications. I think that’s what our niche really is.

Scott:  What’s your sense for what happens to an Apache Web server release between the time you’re done with it and it gets distributed by a Red Hat, Oracle, Solaris , etc?

Justin:  We have contributors from Red Hat. A lot of the people who are doing distros and ensuring that it gets in front of the users are involved in our community. Our philosophy has been why are you making this huge patch set for this particular product? Get it upstream, get it back to us, we want to take it. I think generally, for the most part, you will see that there isn’t a lot of variation when it gets into the distribution because these people have been working with us.

Sean:  Let me do a follow‑up on that. Some say the strength of a closed source project is that the company may be able to provide more of an integrated stack. You take a look at something like Microsoft ships and they might say, "You should use this because it’s an integrated stack, and there’s a single vendor you go to for support." Obviously there are some projects that are tightly coupled together, such as Suversion and Apache. What would you say to that from the open source side?

Justin:  Right.

Sean:  It seems like in open source, the communities are not isolated.  There’s a fair amount of core maintainer communication that’s going on.

Justin:  Yeah, I think you see that. I think that’s why you see a number of committers in multiple communities. As you get used to it, you start to follow the dependency chain, and you get into those communities and say, "Hey, I just broke this for you over here, but here’s the patch to fix it." You tend to see a lot of that happening.

Sean:  Are there things you see in other open source projects that look interesting and might influence Apache?

Justin:  I like what Ubuntu has been doing, where they say, "We’re doing a release every six months. (no matter what), we’re going to have a release." That’s a very hard thing to do. That requires some of the dynamics that Canonical has with their contributors. That’s something that I think they do really well.

Generally our philosophy has been, “we’re releasing when it’s ready”, and some think that’s a good philosophy. You don’t want to promise something, but then you think, "Well it’s been so long since the last version." There are all these changes that sit there and keep getting improved upon. But if you have the regular release cycles, I think that’s a good thing.

Scott:  Justin, we’re out of time, but this has been a great conversation.  Thanks for taking the time to chat with us.

Bookmark this:

Interviewee: Marc Frons

In this interview with Marc Frons we talked to him about:

• His background with the New York Times.
• Their focus on the LAMP stack and Solaris.
• What he likes about open source.
• The ethos of open source and the ethos of a news organization.
• How pervasive open source is in the New York Times business outside of the New York Times Digital properties.
• What the reporters use to get their job done.

Sean Campbell:  Marc can you lay out a bit of your background with the New York Times?

Marc Frons:  Sure. I’m the Chief Technology Officer of the New York Times’ Digital Operations group. I’ve been at the Times since June of 2006. My role is to oversee technology strategy and operations for all of our digital properties. Since July 2007, I have been directly supervising NYTimes.com’s technology and product development as part of my  responsibilities at the company in addition to looking at potential partnerships, acquisitions, technology strategy and tactics overall.

Sean:  Where are the places where the New York Times has made investments in open and closed source technologies?

Marc:  The history of the NewYorkTimes.com is interesting. It is one of the first newspaper websites and now the largest newspaper website on the planet, as far as I can tell. And for the past six years, it has been a proprietary platform and a very good proprietary platform. The folks long before I got here wrote their own application server, even wrote their own programming language. And it’s highly scalable, very secure, and incredibly fast and it has served us well for the past six years. At the same time it is somewhat inflexible in its current form. So we made the decision a little over a year‑and‑a‑half ago to begin to bring more open source technologies into the shop and develop some of our new products using open source technologies. And that was a conscious decision to leverage the knowledge of what other folks were doing in our industry, which would enable us to move faster and be more innovative. At the same time, that other platform isn’t going away and we are continuing to do some things on that as well.

Scott Swigart:  Since you essentially have the world’s largest news web site with the associated high expectations that brings, how do you approach something like that? What would be your advice to another organization looking at open source, things you can take for a given with open source, and things you need to take a closer look at in order to ensure it is going to be handled?

Marc:  We are making a bet, not a big bet, because we are being incremental. We are not just saying "OK, we are going to redo this entire platform that serves almost a billion pages a month and all sorts of other things on a new platform" and just flip the switch all in one big bang.

We have an architecture that allows us to experiment, hedge our bets and be incremental. And I would advise anyone to do the same when making any platform transition.  Do it a little bit at a time, try out a couple of new lower risk products, things that you can do quickly, things that don’t get a ton of traffic, and then you move up the food chain as you go along. Test your hypothesis; see if these things that you thought were true really are true.

Scott:  So just to make sure I understand that right, you may take some section that is three levels down in the hierarchy and move that over. When you look at a website, most people never really bother to understand what it is running on and so you keep reimplementing more sections until eventually the home page is being served up by the new technology?

Marc:  That’s pretty much our approach, because obviously your new technology is going to evolve along the way as well. You want to try it out on the smaller pieces of your site or smaller sites that are in your infrastructure and not do it all at once because you are going to evolve your open source platform as you along.

:  And so when you say open source and you are focusing on web, what immediately springs to mind are the LAMP stack, Linux, Apache…

Marc:  That’s pretty much what we have.

Scott:  OK.

Marc:  We have Sun Solaris in the house, OpenSolaris. But we will also be moving to Linux just for some things; we already have Linux for some things. We will be doing some more things on a pure LAMP stack.

Scott:  And some of the concerns around that I would assume are the quality of the software, support, servicing, documentation. So what are some of the things that you feel are really good versus things that are going to take extra diligence on your part?

Marc:  Quality is obviously important. We feel like these various pieces of the stack have reached a maturity, certainly on the level of the operating system, that allows us to use them more widely.  This is not something we have to worry about particularly much. On the database level, it has also reached a level of maturity that for our purposes anyway is adequate. On the web and application serving level, it is the same thing.

So if you look at the various layers of the stack, these are all things that we have increasing confidence that perhaps two or three years ago, you might say well these products have a ways to go. I mean others like Apache were more mature, but certainly even in the last couple of years, MySQL has made significant strides and Linux keeps getting better and the kind of support you can get for all of the various pieces of technology in your stack has certainly grown. The documentation that you get with it, the cost is a big factor obviously, but these are just attributes that make it possible to use open source software in the first place. For us, the major factor is when talented developers are excited about the tools they are using and the technologies they are using and are passionate about it, you get a lot of innovative development. You can’t overestimate the importance of that.

:  One question given that you’ve built somewhat of a home brewed solution. With that in mind what do you see about the typical advantages open source gives you to fork it, providing you complete access to the source, etc. Is that part of the equation for you or is it just that you feel the platform has matured to the right point?

Marc:  Access to the source is nice, but I really would rather not be rewriting the kernel.

Sean:  Right.

Marc:  What I like about open source is that you are leveraging the community to help you make improvements far more rapidly than any single company could make them for you with a vendor product.

Sean:  Talk to me a little bit about that. How do you see that playing out in terms of the rapid innovation? Do you see the New York Times eventually contributing more actively to some of these open source projects? The Apache folks would have to look at you guys and say, "It’s a billion web pages." Whatever you would have to say about Apache you think they would listen, etc.

Marc:  I think so. This is obviously a new thing for the New York Times. So there’s a lot of building we need to do internally.  But our goal, in general, is to think about ourselves as a platform‑‑not a technology platform so much as a news and information platform‑‑and adopt some of the same ethos of the open source community in software, when it comes to content, information, news, research and data. So as we move forward into 2008, our goal is to create APIs not only for our own internal consumption, but for the outside world so that you can access the New York Times search archive and mix and match that with other data and information and applications, and create compelling applications out of our content and our other tools.

:   One quick follow‑up to that and then I’ll turn it over to Scott. I’m just going to pull on a thread that seemed to be underlying the previous discussion, but I’m not sure if this is accurate or not. Would you say that, from your perspective, there is an almost symbiotic nature between the ethos of open source and the ethos of an organization that’s journalistic, in that you want to provide information? You want to provide an open sense of community. You want to have multiple eyes looking at something trying to analyze the righteousness of it.

Marc:  That’s right.  We’ve increasingly been moving to open NYTimes.com both journalistically and technologically. We have close to 100 blogs by our journalists and launch new ones almost every week. We allow our reads to post comments on articles and we’ll soon be launching additional community and social networking functionality on the site, built, of course, using open source technologies. Perhaps most significant, last September we ended our paid subscription product, TimesSelect, and made virtually all of our content free. And the response has been pretty amazing.

So it’s true, open source fits nicely with how we think about NYTimes.com as a whole.  If you think about the history of journalism, Journalism 1.0 maybe was the typical "who, what, when, where, why" third person journalist‑‑just the facts. Journalism 2.0, the new journalism, maybe is Tom Wolfe,  Hunter Thompson and others who themselves became the center of the story. And 3.0 is where the audience becomes a part of the story through blogs and other forms of online community, and where the Internet helps inform what you’re doing by enabling you to mash up other data, other news, other points of view. So I think that’s where the New York Times as an organization is going, and really needs to go as we become a part of not just an isolated node on the Internet but an integral part of this vast but still growing community.

Scott:  Tell me if I’m reading too much into what you’re saying, but I don’t think I am. It seems you’re saying if I had a team of developers and they were used to buying their software and they were used to a proprietary model of acquiring their tools, they may look at our business and our information in more of a proprietary way. But if you take developers who are kind of steeped in free flow of information – openness, taking from the community and contributing back – those people come up with ideas. And those ideas probably infuse a lot of different things they do, including ideas they come up with for how the New York Times could collect and contribute information to the planet.

Marc:   I haven’t really articulated that even to myself in quite that way, but I think you’re right. I think adopting an open source philosophy tends to attract a particular type of developer who perhaps‑‑and this is not true for everyone, obviously‑‑but who perhaps is more engaged and more aware of the possibilities of information sharing than someone who is very used to a single product, a single source approach, a more closed approach. A lot of that, too, has to do with personality and skill set and interest.

Scott:  Right, right, but people who tend to work in open source kind of gravitate that way because of their personality I would assume.

Marc:  Yes.

Scott:  People are kind of this sort…

Marc:  Let’s put it this way: it certainly couldn’t hurt, right?

:  [laughter]
Well, let me ask how maybe the digital business contrasts with some of the other businesses in the New York Times. You guys are using a lot of open source technology. Is that pervasive, do you feel, throughout the New York Times, or is it more like look, "We’ve got a lot of office workers. They all run Office all day long." Different businesses but not…

Marc:  It is not yet pervasive, and I doubt that it will be pervasive anytime soon.

Scott:  On the Web and for what you guys are doing, open source makes a lot of sense. Are there areas where you think it doesn’t make as much sense?

Marc:  When a proprietary product is so clear superior to anything you can get through open source, then naturally you’d go with the vendor solution. But I think for systems and products where software does not change very much‑‑where you have a vendor product that meets your needs, that allows you to run without a whole lot of technical infrastructure, a product that gets the job done,  doesn’t have to be particularly customized,  isn’t changing all the time, where business requirements are set, where you just sort of want it to work‑‑because your business really isn’t changing. You have a payroll system. You know you have 10,000 employees. You know they get a paycheck every two weeks.

Unless there’s an open source product that has been battle tested in the market and works and you can just download, install and license it, you’re probably going to go with an established vendor, right?

Scott:  Right, right.

Marc:  So the same is probably true for many of your other corporate systems.

Sean:  Well, let me ask a question on that, because maybe I’m right or wrong about this. I would guess the New York Times probably breaks down into a ton of different IT buckets- three broader ones at a minimum where it might be servicing the needs of the staff, for lack of a better phrase: the accounting teams, and the HR teams? Then you’ve got the web presence, which is huge. But then, I would also imagine you have the needs of the reports which is the heart of what people think the New York Times consists of in any case.

Marc:  Yes.

:  So with that in mind I’m curious. Maybe for the office worker, a closed source piece of software makes sense. Then you’ve got the Web, where you have a variety of reasons you went with open source. How does it play out for reporters who are having to report from almost anywhere on the planet? What type of technology stack enables them to do what they do?

Marc:  It’s a good question. And it’s actually very interesting that you mention that. Shortly after I assumed this role and reorganized the NYTimes software group, I created a group called Interactive News Technologies. Basically, it’s a small team of software engineers and technically-minded journalists who sit in the newsroom alongside the reporters, and are charged with building both tools and very rapid development applications for news stories and news events. And actually they are using Ruby on Rails for some of their stuff.

We’re also now talking about fronting the big print publishing systems with web forms and other tools in order to get structured data, and a lot of the other things that we want to do that we can’t do easily in big, closed systems. So we are beginning to use open source in that way to help our Newsroom.

Sean:  Well, one question on that then; so Ruby on Rails is obviously popular and the Basecamp guys have probably popularized it more than most. As in they are one of the canonical apps where they focus on how easy it is to use, it has good fidelity, etc. So just to drill into that one for a minute, why the choice of Ruby on Rails? Because you obviously laid out a good business problem of, you have team of reporters, and they probably have needs that are immediate. You don’t want to be slowed up in any way. You have to have a development team that is able to move pretty fast. And with all that in mind I’m curious why you made the choice you did.

Marc:  My philosophy is, you tend to go to a large extent with what your team is comfortable with and excited about using. So when I was at Dow Jones, we were a big Java shop and had Java J2EE stack with a mix of open source and vendor components. But we used Apache, and we were moving to Linux, MySQL, and Tomcat and away slowly from Oracle and WebSphere, as these open source pieces of the stack matured.  That is where my guys were comfortable with. Those were the technologies they liked. They proved to me they could make them work. I wasn’t about to tell them they had to use something they were opposed to using.

Our interactive news technology guys said, "Listen, we’d really like to try a Ruby platform as opposed to Django, as opposed to some other framework." And I said, "Well, go for it; show me that it will work and start to experiment, do some things on it." So my feeling was you have to let your people have the tools that they are comfortable with as long as you feel that they have a chance to succeed, and you have to charge them with succeeding.

Scott:  You said something earlier that I think ties into that and, again, let me say it in my own words and see if you’d agree. It is more important that you can build whatever you want with your choice of today’s modern tools. And so you want your developers to be passionate and excited, so you let them use the tools they are going to be passionate and excited about using. Because you are going to get more creativity, productivity, etc. if you are letting them have input into their toolset rather than if you are simply dictating a toolset such as, "Well we are a Java Shop, therefore everybody use Java."

Marc:  Yes, exactly. The typical corporate CIO wants to consolidate systems and technologies and have a monolithic structure because he thinks that is going to save him money. And for some things that is true, but corporate technology is almost always a cost center.  Whereas for the digital businesses, IT is integral to the growth of the business and you could argue that IT is the business — software development. You don’t have a business if you don’t have software developers. You don’t have one if you don’t have journalists either if you are in the New York Times, but that is why technology and journalism are inexorably intertwined in the digital world much more so than they have ever been before and why they will continue to be even closer, even more intertwined as we move forward.

Sean:  So what do you see in the open source community that excites or intrigues you the most and I guess at the same time what excites you in terms of the closed source side of the world. As a CTO, you are looking at technology trends as well as specific implementation issues. What seems to stand out?

Marc:   Let us put it this way, there are so many open source projects, there are so many things happening, there is so much innovation going on that it is hard to pick and choose. I mean I think obviously the whole LAMP revolution has been fascinating to watch. These kind of contained frameworks like Ruby, like Tapestry are quite fascinating in that they put tools in the hands of people who perhaps don’t have the skills of hardcore software engineers, who are busy writing code at the base level.

But what I think really excites me is the move to making a lot of these more sort of all‑in‑one frameworks more scalable because if you are in the New York Times, it has got to scale and if it doesn’t scale, we basically can’t use it for anything really important.

Sean:  Right, exactly. Yeah, a billion pages are simply a billion pages, right?

Marc:  Right. Now the truth of the matter is it is not exactly evenly distributed. I mean some parts of the site get a ton of traffic and others not so much. So if you want to do something on Ruby, you put that on a thing where you know it is not going get creamed. Just because it is the first time you have really done that and it is Ruby, so you have to make sure that it is going to work OK, but as you gain confidence in these technologies, then you begin to roll them out in more highly visible and highly trafficked places.

Scott:  I don’t know how core this is to our investigation, but it is an interesting question to me and you might have an interesting answer to it. This is sort of continuum, right? On one end of the continuum, you have closed source proprietary companies who want to be very secretive about their roadmap, very secretive about their features, not share their source code. And even that is diminishing because even they are realizing that there is business value in transparency.

And then on the other end, you have the free software foundation who feels like all code effectively should be public. Initially they were kind of happy with software as a service because there were people who were using open source and building on top of it. But they might even want to go after someone like Google and say, "Hey, you benefited from open source, share your code." Where do you think it makes sense between those two spectrums to kind of draw the line, because it has got to make sense. You can’t kill the goose that laid the golden egg in either direction.

Marc:  Absolutely. Obviously, people need to be compensated in some way, shape or form for their hard work.  But there are emerging models: Just because you are open, doesn’t mean you don’t charge for it. Look at MySQL, you can download it but then you pay for support. It may be cheaper than Oracle but it’s not really free.  The same is true for various flavors of Linux. So I think people are finding ways to make those models work.

Google, it is true, has used a lot of open source, they have also written a lot of their own proprietary stuff that they have not given back to the community. And I think they have done that probably because they needed to just because of the scale of their operation. It really wouldn’t even be germane to most other businesses, right?

Bookmark this:

Interviewees:Shawn Burke.

In this interview with Shawn Burke of Microsoft we asked him about:

• Shawn’s motivation for helping to make the framework source code available.
• Whether there are any limits on who gets access to the source code.
• Whether there was any internal resistance initially to providing the framework source.
• Why Microsoft doesn’t just provide a zip for the source.
• Whether Microsoft’s individual developers write code differently knowing that their actual source might be available for the world to see someday.
• Whether or not Microsoft is planning on making available other development artifacts such as test scripts, threat models, etc.
• Whether a closed source company or an open source company is better positioned to support external developers who need access to the source for a given project.

Scott: All right. So if you take a second and just briefly introduce yourself, that is probably a good place to start. Then we can drill into questions specifically around the source for the.NET framework 3.5.

Shawn: OK. I am Shawn Burke; I am a director here at the.NET development platform at Microsoft. I run a group that we internally call the agility team. We run small high value projects to try to help deliver value to customers in agile ways. Not necessarily using agile technology, but just being agile about how we deliver.

Scott: :So in other words, it sounds like you are looking for sort of low hanging fruit where you can do things that are a little bit outside of the norm but would not necessarily be an enormous effort, but would deliver a lot of value to the customers quickly. Am I characterizing it correctly?

Shawn: That is right. Our team works a little bit outside the bounds of the typical projects and we are not constrained by the same rules they are. We are able to do things that are more customer focused and a little less process focused way.

Scott: Just as a point of clarification, people have said Microsoft is “open sourcing the.NET framework.”That is not what your Microsofts intention is and not really what you are doing. You are making your source code available for reference. You are not really open sourcing it the way that, you know…

Shawn: Nope, some people in the community used characterization, that is definitely not what were doing.Were making the source available as a tool that helps people be more successful with the platform in its current state.

With this effort, my goal has been from day one, a simple scenario, which is you are debugging against an API, you are getting an exception from the API, and you cannot figure out why you are getting it. It is often very difficult to figure that out just from documentation. However, your ability to step into the API often makes it really clear what you are doing wrong.

Scott: Right. In the example that Scott Guthrie put up on his site, this is something I’ve run into,? At the point where you DataBind something, all kinds of stuff happens like down in the plumbing.

Shawn: Yes.

Scott: And the exception that you get, you know, you do not really have a lot of visibility and sort of what you did wrong in your code that caused the DataBinding exception. And so that’s like…

Shawn: Yes.

Scott: That’s like the key scenario that you went after.

Shawn: Yes. That is like the thing that you are unableto do currently you know, because I work here I have the benefit of being able to do that a lot. Quite frequently, the amount of time it saved me , “Oh, Ill just grab the symbols” and you do and start to debug with them and immediately you’re like “Oh! Because I forgot to do this or that.” It becomes so clear that I really wanted to enable that sort of service for customers because its such a powerful thing

Scott: Yes. And just to back up another thing that you said, too. I never read it. I was not indicating that Microsoft was somehow trying to create the impression that they open sourcing this. I know when I looked at it, it was clear to me that Microsoft was releasing the source for reference purposes, you were not releasing it so that people could…

Shawn: Yes. I get it. I did not mean to sound defensive there!

Scott: [laughs]
I’m sure it comes up a lot.

Shawn: Yes. That was the first thing we saw on a large website that is often not terribly pro Microsoft

Scott: [laughs]

Shawn: Fortunately many people came on there and said, “No, no, no.” They never said that, that is not what they are doing. I think most people get that now.

Scott: So what you are enabling is a sort of step into debugging experience where normally, as soon as you call into the API, you could not continue to drill down and step in. Now you can.

One of the things I was not clear on is who gets this? Is this available to all versions of Visual Studio and all customers or is there a sort a subset? Who is this available to I guess?

Shawn: If you are running on Windows, it is available to anyone who can use it or to look at it.

As far as the Visual Studio integration, you need to have VS Standard or above. You cannot do it with Express versions. You know the express versions of Visual Studio?

Scott: Sure.

Shawn: Express does not get it for some kind of fundamental reasons.The main reason there is it just simply because Express does not expose the UI that you need to set this up or have some of the other facilities internally to take advantage of it

Scott: Sure. Sure.

Scott: Yes. The only reason I ask is that some of the people in the Microsoft ecosystem have made their source available kind of under premium plans with their customers. However, that is not the approach that Microsoft is taking with this. This is not sort of a premium thing, this is just generally available.

Shawn: Yes, that is Code Center Premium (CCP), this works similar to that CCP , but there you need to have an account and a Smart Card actually. And you have to go through much more security than in this system. This is a public reference release of the code, so there is really no credentials required. There is none of that.

Scott: So when you sort of floated this idea internally, it sounds like it was generally well received. People kind of jumped on board and wanted to help make it happen. Just behind the scenes, what were some of the things that it took to enable this? It is more than just dropping a zip of the source out there to make this happen.

Shawn: Yeah. Well, yeah. It is good that you brought that up. This whole thing started quite a few years ago. When I was on the Windows Forms team, which I was for seven years, I always felt like we should be releasing the source for many reasons. Others had done it. I felt like it was good for customers. When I was the Development Manager on Windows Forms in early 2005, I did a blog post about this, saying I wanted to make this happen and talking about some of the challenges to doing so. It was kind of justI was just kind of brainstorming aloud about different ways and what some of the challenges were. The response was enormous.Hundreds of comments.

A funny story, I did that blog post, and then I went down to Cabo San Lucas with my girlfriend a couple of days later. When I turned my phone on halfway through the trip, just to see if it would work in Cabo, I had a flood of text messages that said, “Please call work.”

Sean Campbell: [laughter] All right. So you learned a lesson of do not blog anything but “I’m going to Cabo” on the blog before you go to Cabo, basically.

Shawn: Yeah. I think if you would really pull it down to basics, that is it. Well, actually, I had always done tons of blog posts that no one’s…

Scott: Meanwhile, you created this firestorm of all these people running around going, “What have you done?”

Shawn: Yeah, exactly. I like to joke that it created, basically, an international incident because it got picked up by eWEEK, Mary Jo Foley, CNET, and all the rest. Quite a few people picked it up. It generated a lot of interest. That was a good impetus for me that there was a lot of excitement in the community about this.

If you fast forward a couple of years, I did some more ground work on this thing in 2005 and 2006, got busy with new role I’m in now, and dropped it for six or eight months, and then picked it back upI think it was late last year, late 2006and started looking at how we would get this done.And Ive been shepherding it along since then.Getting this out there really is the realization of a dream that I have been chipping away at for quite some time.I really hope Developers find it as useful as I think they will!

You guys mentioned pushing it out as a zip. One of the real problems was that I did not want to have a situation where every team that did this put a zip file or MSI out every four months, so the customer had to go download, install, and set up his little studio to use. What if you have a hotfix or different version? You know what I mean. I did not want to have a situation that was going to be cumbersome for us to ship and for customers to consume.That would kind of defeat the purpose.

What really got me rolling on this was I started talking with the guy who developed this technology we use internally. Internally, you can get the source and demos for almost every product we build here at Microsoft off this thing called “Source Server.” Source Server has versioning built in, making sure things do not collide so you can have side by side versioning, and it all works very smoothly. The server side piece for source server is just a web server. There is no magic there so its very simply to deploy. There really was not any barrier to making to using that technology externally.

So what we able to develop was a system where teams could publish source very, very efficiently. After a first source publish, they can continue to do source publishing at near zero cost to them, which is great. On the customer side, you have a scenario where a customer has a few simple set up steps. Then they kind of auto magically get source and symbols for what we have chosen to make available at that time for free. No searching around on MSDN or having to deal with where to put the stuff on the machine.

Scott: It sounds like, too, one of things that you are saying this accomplishes is it sort of knows exactly what the client is running and building against. If they have installed the service pack or have not installed the service pack, they get the right source. They get source that matches up with their environment.

Shawn: Yes, if it is on the servers they can get it. One of the decisions I made in getting this done was that, the cheater thing I did was I said, “You know what, I ‘m not going to define policy on this. I’m going to get a system developed where individual product groups and teams can make decisions about how often they push source.” Meaning, does it make sense to push source for every new release we do? Maybe, I dont know. Does it make sense to push it for every service pack? Maybe, I do not know. I think those might be different for different teams. I think we are going to do learning in working with the community about what really makes sense there. However, the system is set up to handle it, in any case.

Scott: Like you said, generally people were receptive to this. Were there internal objections that had to be overcome? Where there people who had concerns that, “Hey, this is crazy. Why would we do this?”

Shawn: Well, if you go back into 2005, I think I had many senior people that I knew, vice president level people that were generally supportive of this, but very cautiously supportive. For me a little bit, one of the reasons it’s been a multi year effort is I had to push carefully and let some perspective change as the industry has changed and the environment changed over the last couple of years.

It has gotten significantly easier. Something happened, I think, in the greater consciousness of the industry over the last 12 months or so. A lot ofI do not want to say a lotresistance is too strong of a word. A lot of the real cautiousness and kind of by-default-no talking-into-yes stuff dissipated and switched to actually a lot of enthusiasm. We need to be careful about protecting the IP that we spent billions of dollars developing. However, I think people really see the value of this now. It has been great for us and for our customers for a subset of products. Now I have seen teams are signing up to push their code out really quite enthusiastically.

Scott: With the.NET framework, it is in a little bit of a unique position because, on one hand, Microsoft has invested a lot in it, and generated a lot of IP. However, on the other hand, it was never really a secret what was in the framework anyways. You could use reflector and other tools to disassemble what was in the framework and sort of figure out how it was doing what it was doing. However, it did not enable this “step into it while you are debugging” experience. It did not make it super easy to learn from the framework. This might have been a little unique in that the IP was not all that protected to begin with.

Shawn: Well, yeah. Yeah, it is funny. The case that I built for this, with managed code in particular, definitely had that aspect to it, that our level of secrecy around how this stuff works, as well. One of the problems with reflector, using reflector, the EULA does not really allow for it.

Scott: Ah, interesting.

Shawn: The EULA does say that you are not supposed to reverse engineer. That said, I felt like it was a good idea to give customers a way to do this within the license, which this is.

Scott: Hah! Yes, I guess I never really realized that that might be in the EULA. I think, especially among the more advanced developers and in the blogs, and in all that kind of stuff, it is such a common practice.

So, now that you have done this, you have taken this first step and you’re going to make source code available for certain subsystems of the framework, and that’s on a feature by feature basis, right?

So, Windows Forms will decide when they push their source code in, ASP.NET will decide when they push theirs in, am I understanding that right?

Shawn: Well, we have pretty much what you would consider, if you go back to.NET 2.0 and think about what made up the Framework at that point. We’ve got pretty much all of the big pieces there. We’ve got BCL, Windows Forms, ASP.NET, Data and XML.

So those guys are already taken care of. Therefore, we’re going out with essentially what is the original.NET framework. The other stuff will fall in. It is really a matter of me getting with a given team and through the process. The code preparation process is actually fairly labor intensive. It takes time to schedule that and manage it.

I had teams that were able to do that. A lot of teams have been obviously really busy getting VS 2008 and.NET framework 3.5 finished up. We will continue to push out more code as time goes on. We wanted to get the big ones first.

We also managed to get WPF in there as well, so its not just the older stuff.Lots more is in the pipe to come online as time goes on.

The choice was either we wait a couple of months to do this, and make everybody wait, or we get the big ones out there fast and then we push in the other stuff as we have time, and as it comes online.

Scott: So one of the things that is kind of interesting is, you’ve got these individual developers in Microsoft and they’re writing code. They have worked in Microsoft for a long time and they have a feeling that, you know, what happens in Microsoft stays in Microsoft, I guess, for lack of a better term.

The whole world is not going to see the line of code that they are typing right now. Now, with the source being released, how do you think that is affecting, if at all, how individual developers are doing their job? How product teams are looking at how they build features going forward, knowing that the source for it is going to be out there.

Shawn: Even product teams that are not the framework team, they are thinking, that’s great, they did that over there, so does this mean that someday my source is actually going to be out there? Yes. Yes. They will be thinking that. And they should be thinking that.

In fact, one of my to do list items is to do some broader communications across our Division about that exact point Software developers are pretty much the same around the world. Software developers, in general, always try to do the right thing. They are also always, in general, under a certain amount of time pressure. Anything that is going to encourage them to take a little extra time to do something right, in the kind of capital ‘R’ right, way, is a good thing.

Whether that’s testing, whether that’s static analysis like FxCop, or whether that’s security reviews or code reviews, or whether that’s the code is going to be visible to their peers or the public, I think those are all good incentives for software developers to think extra hard about how they do things.

So, you know, honestly, we went through the code, we’ve done these code reviews, and these code scrubs and we found a lot less stuff than we thought we would.Actually, we found almost nothing of any real concern, which was nice.

Sean: Are there other things that Microsoft is thinking about exposing around the development process?

There are elements like threat modeling, specs, test scripts, all kinds of things that Microsoft might be thinking about exposing. Are there any thoughts or plans along those ways about how far this maybe goes, eventually?

Shawn: That is a really good question. No is the answer. [laughter]

Sean: You’ve accomplished enough by ruining one Cabo vacation. Right? You can put that post up later, you know, on your next vacation.

Shawn: I think that stuff…that is a good point. If you start doing, a curve on the cost for us to do that versus what percentage of customers that level of access is going to be valuable for. For the threat models, and even for the specs, I think that is a harder case to make. I think those things are also harder to vet, or release, than the source is. They are also less useful to a broad set of customers.

Sean: Well, one of the things that crossed my mind was test scripts, and test routines that Microsoft creates because that would be helpful to others just from an educational standpoint. That does not really expose Microsoft to too much more overhead. I am not trying to pitch the idea here, But go ahead, Scott, what were you going to ask?

Scott: No, same thing. Just, Microsoft has given an inch so watch us try to take a mile.

Shawn: Oh, yeah, we know it. Absolutely.

Scott: A lot of times that’s what people want. They say, OK, there is this API. I want an example for how would use it. I would go look in the help. Well, that example is not what I really want. Microsoft I would assume writes really comprehensive test suites.

Shawn: I do not know if the test code is what you want, either. We call it auto PMEs. The PME stands for Property Method and Event. It is super granular. A lot of that code does not relate to any real scenario. It is just wide unit testing code. There are other types of tests as well. A lot of the stuff is that. The challenge of the testing stuff.

I actually think that the testing stuff is a good point, and an interesting thing and I have been thinking about, not really in terms of us releasing test stuff for previous products. I will get into one in a second. The question is going forward, is there a way for customers to leverage the kind of stuff we do in tests more broadly?

However, the reason, going backward, most of our test stuff is written using internal test harnesses and internal tools. There is this huge slippery slope there. You need this and you need that, and it would be really a lot of work.

Scott: Right, right, so they would have this test but they would not be able to hit F5 on it and actually watch it run anyways.

Shawn: Yeah, and the tests probably call into a bunch of other libraries that are internal that we would not ship externally anyway. The tests themselves might not make any sense. Most of the tests are stored in this big system called Mad Dog, which is kind of a giant database of test cases and test code, and I do not even know how you go about extracting code from that thing.

[laughter]

Sean: So Shawn, what do you think aboutthis is just something I was noodling on, trying to think of the right way to box it in given the time in terms of a questionso, Microsoft’s a closed source company, right. By definition if not necessarily for the sake of our interview. What do you think about closed source companies perhaps at times being better able to support the exposure of the source than maybe a company or a project that simply says ‘Well, just go check it out from subversion, from xyz open source project?’

Shawn: Right.

Sean: Because I look at the Visual Studio integration, and I look at robably how this will eventually weave into PSS, interacting with customers, and saying well, here’s an odd dynamic: is a closed source company perhaps better able to actually make the exposure of the source code, when they do it, a more positive experience that actually lets people leverage it more easily? I mean, absent the ability to fork, and some of the more foundational principles of Open Source, how an outside developer would use it day to day development and how a closed source company in this case Microsoft could support that.

Shawn: That is a great question. I think that the answer isone of the things that I’ve been trying to do, and that we’ve been trying to do, is look at the changes in the industry that open source has driven and really think hard about what the real value adds are there. Is the value add really that you can recompile your kernel, or is the value add that you can debug it and understand why it’s doing something?

And there are obviously cases where the answer is ‘both’ or ‘one or the other’. However, for I think the vast majority of folks, us being able to deliver the source in a way that’s easy for folks to consume and step through as a way to make them more successful on our platform is a really big win. I think that it fits well into our overall strategy of kind of keeping developers integrated in our overall processes, and it fits well into this another milestone of transparency.

We have kind of been trying to figure out ways to open up the envelope of transparency with Developer Division particularly and I think this fits into that portfolio really wellas well as the portfolio that the Shared Source Initiative folks are building. You have CodePlex on one end which is full open source, then you have kind of our proprietary source system on the other end, and there’s a nice continuum in between. I think that we do bring some assets to bear which are difficult for companies that are purely open source or service model driven.

Sean: OK, cool. Well, that is all I’ve got, and also to be sensitive to timeScott, do you have a closing question or two?

Scott: I think that’s a good note to leave it on. The thing that we always end with is: are there things that we didn’t cover that you think are really important for people to know? So, we’ll just kind of hand you the microphone at the end if there’s something you want to get out that we didn’t specifically ask about.

Shawn:Nope, I think we covered it all.Thanks guys!

Bookmark this:

Interviewee:Steve Morris

In this interview with Steve Morris the director of the Open Technology Center we asked him about:

• The role of the OTBC
• The original lean of the OTBC toward open source
• The dispersal of Open Source Tools in Large Organizations
• Licensing
• The percentage of companies at the OTBC that use open source technologies
• The business implications of Open Source

Scott: Steve, thanks for taking the time to chat.Could just take a second and tell us little bit about yourself and the OTBC?

Steve Morris: Sure. I’ve been executive director for about nine months. OTBC has been around for a couple of years. My background is very much a technology, marketing, and management background. That started at HP, and now I’m mainly working with local high-tech companies largely in the EDA and semiconductor test areas. This includes both software and hardware.

OTBC is an incubator for start-ups. It’s funded by the City of Beaverton, related to their economic development plan, so what they’re interested in is not really pushing open source or any particular dogma;they’re interested in jobs.

The feasibility study for this place was done as a software company incubator, and it got focused early in its life on open technology.

Since then, we’ve broadened it back to being technology and computer focused more generally, because focusing on open source was just too narrow to meet our goals. The logic there is basically that if your goal is to find lots of early stage companies with a lot of scalable potential for creating jobs, then it doesn’t make sense to limit yourself to a narrow niche.

Richard: Why did it originally lean toward open source? What were the arguments?

Steve: We were very much focusing on Oregon’s cluster technologies, of which open technology is one. We’re very open source friendly, and certainly encouraging companies to use open source in ways that are appropriate to their business model. But, we’re certainly not exclusive to that, just like a VC. You won’t find VCs being too narrow, because they want to consider a reasonable breadth of good business plans for growth, and we’re much the same.

One of my beliefs in the open source discussion is that you would be hard-pressed to find very many software development projects that did not use open source in some way.

Actually, I would challenge you to point out one that doesn’t use open source in some way, because software engineers like to use open source tools. At the same time, one question I see a lot of companies not asking, that they should be asking is, "How are we using open source today?” whether it’s tools, or just incorporating code into their product. Was that a conscious decision? Did they look at the business consequences? And should indeed they be using open source in a different way, either more or less than they’re using it today?"

Scott: What do you feel are some of the reasons why open source has so much gravity? Why does it draw people to it, especially around start ups? And what do you see as being some of the reasons why very early on, when they’re wondering if they have a viable company even, they should be concerned with the how and the why of open source, and how it might affect them as their business grows?

Steve: Well, first of all let me give you a nicely documented example that’s not exactly a start up, but it’s illustrative. Ben Berry, the CIO for the Oregon Department in Transportation, decided to do an audit a year or two ago. Using some automated tools, he did a search across all of his ODOT computer systems to count every open source tool that we could find– just literally to find out what was there.

They found, if I remember right, about 3,000 or 4,000instances in the ODOT computer systems of open source tools. ODOT at that point had absolutely no policy or strategy about open source. There was no conscious management behind this.

It turned out that this was because open source development tools were free. People at ODOT could download them and use them with the zero red tape. And when you work for state government, zero red tape is a pretty attractive thing.

And so in that one particular example, it was because they were good tools, they could do the job for a software developer, and they were free. So they didn’t have to ask anybody, and they already had the appropriate rights to install the software.

That’s an interesting example of one motivation for using open source development tools, which is different from using open source code in your product. Because the tools were free, they were an easy way to get the job done, while eliminating the need to go through red tape.

Scott: On the other side, you know a software start-up company’s going to have to decide whether it’s going to go with an open source development paradigm for its software product, or whether it’s going to go for a closed source. Some choose one, some choose the other. What are the main reasons companies do choose open source as a method for developing their main product?

Steve: Of course, some people choose a mix, and that’s almost an easier question to answer, because if I’m doing a mixed paradigm, probably what I’m doing is latching on to open source stuff that exists, so I don’t have to re-invent it. For instance, if I’m doing a web application, why would I want to re-invent MySQL? On the other hand, I wouldn’t necessarily want to go off and buy an expensive commercial product either, if MySQL would work.

Richard: But in a mixed case, their main code might be closed, so they’re still going along with the old private IP route, as opposed to making a profit on a product that’s entirely open source. Building an open source component is one thing, but making the choice to develop your main project in open source is another. Do you have a sense of the business models that are behind that — the reason people choose that route?

Steve: I certainly would not claim to be an expert on that, but I think there are a couple of different reasons. One is that there truly are people who believe that software ought to be free, or close to free,as in beer, and it really is a philosophical thing.

There are others who are going to use an open source paradigm, who really are thinking more pragmatically. They see something that already does a lot of what they need to do, and it’s open source, so they can use it, and it gets them where they need to go with less effort. People with that more pragmatic attitude are obviously looking at a monetization strategy that’s different then selling source code.

For example there are lots of ‘software as a service’ companies, which really is essentially a web based service. If you can put together most of your software by leveraging something that’s open source, that can get you to your software as a service play a lot faster.

Scott: One thing you said was that some companies start grabbing different open source components and tools, but they don’t necessarily put a lot of thought into it. And you alluded to the fact that it might have business implications down the road. Can you expand on that a little bit?

Steve: Take the ODOT example, where they had software engineers simply downloading it, and using tools with no license review,no policy, and no processes. One interesting question is whether they are going to download something and use it in a way that might violate the license.

My guess is, when you’re talking development tools, that’s probably a pretty rare problem. It’s a lot different, obviously, than downloading some code and putting it in your product, in which case you absolutely can have some license implications.

Scott: In the companies that you deal with, have you seen it happen that people find something that does what they need, and they bundle it with their product and then realize that the licensing terms are going to have some ramifications that they don’t necessarily want? I know that’s a concern, but I’m curious how often it really gets that far.

Steve: It gets that far a lot if you have a mixed model. If part of your product is open source, and part is proprietary, you have to be particularly careful from a licensing standpoint to build the right wall between the license for the proprietary stuff, and the license for the open source stuff. If you’re not careful, your open source license can infect your proprietary software.

Scott: I know that Microsoft cannot accept any community written code for anything that goes into any of their products,because they’re terrified that somebody will do a Google search, find a section of code and say, "these 10 lines of code were originally under the GPL license. Therefore, Windows has to be open source now." Maybe I’m exaggerating a bit, but it’s something like that.

Do you encounter companies that go the other direction? Do you encounter companies that just say, "We’re just using 100% the Microsoft stack: Windows, .NET, Microsoft SQL Server. Or, from your experience, is that pretty uncommon as well?

Steve: I certainly know a few companies that wouldfit that description, although I don’t think we have any that quite fit it at OTBC. My impression is that Microsoft represents such a huge market that you get a lot of companies out there that focus pretty exclusively on a Microsoft environment.

Scott: Do you find that companies put a lot of forethought into it? Is it because the company founder has a certain preference because of their experience, or that they pull in certain developers who lobby really hard for building on top of Linux, or building on top of Windows? Do you see it more often being a really conscious, up-front decision, or just something that kind of happens?

Steve: I see it being a pretty conscious decision most of the time, because it typically comes down to what your target market is, and what platforms you have to support. If your target market is largely Microsoft-driven, that guides you in that direction.

On the other hand, if your target market includes a fair amount of Linux, and Mac, or whatever else, then clearly you’re going to think about it differently. Most software companies are pretty pragmatic about who their target is. Obviously, their choice of platform and development approach can eliminate potential customers, and so they need to think about that fairly carefully.

Richard: You said that over at the OTBC, it’s not specifically or explicitly all open source anymore. What percentage of the companies that you have in residence there are working on purely open source products?

Steve: Well, including open source products with a software as a service business model, I had at in at least three or four,probably more.

Richard: Can you expand a little bit on what that model looks like?

Steve: One of the companies is called"GoSeeTell." They’re building what is essentially a social networking play having to do with travel. They’re taking an interesting approach where they target a particular sponsor. So, for example, they’re working with Travel Oregon, putting together a web site for Travel Oregon that leverages comments and suggestions from people who have actually been to the places.

But, at the same time, they’re doing one for a Fortune 500company, which is targeted for that company’s employees. It leverages the same database of user feedback as the other portal does, but it’s a portal that’s customized to some specifics that are of particular value to the company’s employees.

It’s very much in a social networking play in that it lets you look at suggestions from other people that have been to a place. And you go beyond that to entering your profile and having the system start to make recommendations: "People like you really enjoyed this place," and then you can hear some of their comments.

The infrastructure is all built on open source tools — things like databases, content management systems, and so forth.

Richard: So, they’re using open source behind the scenes to do their development, but their actual product is really a service offering through the Internet?

Steve: Right. The product itself is a service, and but some of the underlying software is open source.

Scott: In the companies you work with, do you see that as being a primary model of open source, where they’re building their own intellectual property and their own proprietary things, but they’re building it on top of open source because of the advantages that they see open source having?

Steve: It’s mainly what we’re seeing here now, at OTBC at least, in that the majority of the companies we have here are software as service pledged, and to one extent or another, most or all of them are leveraging open source, some pretty aggressively. I don’t know how representative our little sample is, but it certainly is very much the model that we’ve got going on.

Scott: In the Portland metro area, open source is pretty strong, but I would guess that it’s not unique to this geography. The reasons it makes sense here are geographically independent.

Steve: And when you’re building software as as service, again, that’s one place where it makes sense. If you can get pieces that are already built and that you don’t have to pay a royalty to use, then that’s great. And when what you’re offering is a service, you’re pretty safe,in terms of not violating anybody’s license.

Scott: In cases where they’re using a hybrid approach, where they’re not just going 100 percent open source for their infrastructure, what are some of the examples of where they’re choosing a closed source, proprietary component to be a piece of it, and what are some of the reasons you see behind that?

Steve: One particular company that used to be here definitely had a mixed strategy. They had some basic tools that were open source, and then they had some more advanced tools that were proprietary.

The business model was definitely based on proprietary software, and they’re using open source items as marketing hooks, getting people aware of the company and the capabilities. That’s certainly one model you see in the mixed mode — providing some value through an open source solution that will get your name out there, get users involved, and give them a reason to find out more about your other stuff, in order to up-sell them on your proprietary line.

Scott: So in that example, they offered an open source version of their product, but there were certain up-sell features that were for-pay that weren’t open source?

Steve: Exactly. In their case, they were offering a couple of fairly fully functioning modules which were very nicely expanded with capabilities in their proprietary software modules.

Scott: I’ve seen that model quite a bit, too. Some of the things we’ve run into, like SugarCRM, certainly have a model like that,where they give a certain amount for free, but then other features — fairly essential things — are for-pay. Other times the whole product is open source,but what you pay for is support. If somebody goes with a completely open source product, it seems like support, a lot of times, is what they end up charging for.

Steve: I think those are the two basic models.You’ve got to monetize it somehow. You know the Red Hat approach, right? You’re primarily paying for support and installation tools, which is not too different from the traditional model of having a basic product that’s really cheap. In this case, ‘really cheap’ is free, but that’s obviously a model that’s been out there for forever.

Scott: Of course, it’s not unique to open source.Take a look at Adobe. Everybody can download and run the Adobe Reader which work on every important platform. But for a long time, the tools to produce PDF files were something you had to pay for. Now PDF has become so ubiquitous that the format itself is pretty much open source. You see that a lot, where companies will give away a tool that lets you consume something, but the tool to actually produce it is something that you have to pay for.

Interesting to see how that’s evolved. Even companies that are building on top of Windows seem to be doing the same kind of things. I wonder if you’re seeing any of that, where companies are choosing a proprietary stack, but they’re still going with a product that’s modular and they can build a community around, and some of the modules or add-ons or things like that are community-built.

Steve: I haven’t seen a lot of that here or around here. I see what you’re saying, and I think there are certainly things like that out there. I just can’t think of too many examples off the top of my head of them.

Scott: For the companies that go with the route that you talked about, where it’s, "Give away one version, up-sell to another version," it seems that building a community around the free version is pretty critical. Do you see secrets to success in doing that, in getting your free version to achieve critical mass?

Steve: Getting the free version to get critical mass, getting some initial users, can certainly be an effective marketing strategy, whether or not you build a community.

To me, I guess there are two different things that you just talked about. One is having something out there that’s free. As you pointed out, Adobe Reader was free, and it wasn’t open source. It was very definitely a market development strategy, much like some open source plays are, which is different from putting it out there as open source specifically to try to develop a community that’s going to actually help develop and extend the product.

To me, those are two different goals. Maybe I just want to put out something that’s free, and maybe I want to call it open source so that I’m open source-friendly to appeal to the community. But I might not really be looking for a developer community. I might put the source out there, but I might not really care. I might mainly care about people using it.

Richard: Are any of your resident companies leveraging that sort of community? Are they getting development help outside,from a community that they’ve put together through open sourcing?

Steve: Right now, I think the answer to that is no.And again, it’s primarily because the software companies here are largely focused on the software as a service model. I think you’ll see that happening in the software as a service model, although I haven’t seen it here a lot yet.

You’ve got the same rationale, even if your software is available as a service. There’s potentially some leverage if you can open source it and get some additional functionality. It’s a little different, obviously,with software as a service, because this stuff is running on your servers, as opposed to the customers’. And so you care mainly about making sure that it’s going to run well.

I have seen some companies playing with models having software as a hosted service, where the software is open source, and as another option you can run it on your machine. That’s getting closer to a traditional open source model. You can go through the cost of hosting it yourself, or you can buy it as a service on the company’s server.

Scott: If I’m a company looking at releasing as software product, if I release a portion of it as open source, I will get a broader customer base potentially because there are certain people out there that really only want to run stuff that’s open source. Even though they may never look at the source code or do anything with it. Is that a fair assessment?

Steve: I think there’s some truth to that, although I think you can add that you can also position it as an insurance policy.Companies have various types of insurance that they hope they’ll never use.Having access to the source feels like a kind of insurance.

Scott: You know, one of the things that we’re looking for are some real world examples of companies that have been “saved by the source”. Our search hasn’t been extensive, but we haven’t really found any examples of "If I hadn’t had the source, I would have been totally hosed."

Steve: I’m not surprised to hear that because it’s a value proposition that you hope you never have to use, and that’s probably how most people think about it, although I’m sure there are some people that like to buy open source because they just can’t wait to go in and tweak it.

But I’m not sure that I can point you to a specific example of somebody who just really had to tweak something for their purposes.

Richard: So far, we have really been focused on open source consumers and mixed models in terms of using open source tools to put together interesting companies with different sorts of business models.Traditionally there’s been a certain level of antagonism between the open source and closed source community. And some of the things you’ve said imply that, on the open source side at least, that antagonism is changing into something else, some level of collaboration. Would you say that’s a fair assessment?

Steve: I’m not sure. It’s certainly a continuum;there are certainly people out there in the open source world that I would callfairly "purist" who truly believe that software should be free.

On the other hand, there are some more marketing-oriented pragmatists who look at open source as merely being one of the tools in the tool belt that you use either for a marketing purpose, or for a more efficient product-development purpose, or for an additional set of promises you can make to customers.

I think there are a lot more shades of grey out there. I suppose it is changing, but changing by virtue of more folks looking at it with a fresh view and being maybe a little more pragmatic as opposed to philosophical.

Richard: Well, if you think about a bell curve and you think of people who wouldn’t touch open source on one side and people who wouldn’t touch anything but open source on the other side, it seems to me that the center is broadening. That more and more of the majority of people area gnostic about open source versus closed source. Rather they’re looking for what works best for the business model or the support that they want to put together.

Steve: Yes, I absolutely agree. And I guess in my mind the thing that a lot of people gloss over when they talk about open source is that there are these two fundamentally different ways of using it. On the one hand, using tools that help you do your software development, like tools for tracking and reporting bugs, that kind of stuff–stuff that doesn’t go into your product, versus the stuff that does go into your product.

Software engineering tools are where you’d be hard pressed to find very many companies that don’t use open source to a certain extent,because there are just so many good tools out there for software developers.

Scott: I think big companies — the Suns, and the IBMs, and the Microsofts — depend on that. They depend on an ecosystem that’s going to fill in the gaps, and sometimes that ecosystem is a third-party vendor, and sometimes that ecosystem is free and open source software.

Steve: Exactly. Some of it’s third party, and some of it’s not. Some things develop simply because everybody needs them, there’s no good proprietary solution available, and there’s no particular competitive advantage would come out of building it. That’s where you get stuff like Apache. A bunch of people needed it, so instead of everyone inventing their own wheel they all decided to just cooperate.

Scott: Steve, thanks for taking the time to chat.

Bookmark this:

· Using an online “Lab” to engage the community in closed-source development.

· Does open-source tackle interdisciplinary problems well?

· What does an interdisciplinary team in action look like?

Scott:  Thanks for taking the time to chat. Would you mind introducing yourself?

Doug Look:  Sure. I’m Doug Look. My role is senior strategic designer for Autodesk Labs. That means, essentially, overseeing the design aspects of the solutions that we’re exploring at Autodesk Labs. Autodesk Labs is a place where we preview and share new technology. Our intent is to share things early enough in the process so that we can get a good read from our customers and from users as to what works, and what doesn’t work.

I have a particular interest and background in user centered innovation. Before I came back to work at Autodesk Labs, I had gone back to school at the Institute of Design here in Chicago, and I completed a Master of Design Methods program. It’s about design innovation methods and processes, and a lot of focus on going out to understand user needs, by using ethnographic and social sciences techniques.

The thing I’m hoping to bring to the labs, and back to Autodesk, is the notion of understanding user experience. You’ve probably heard that term used a lot. And we’ve got some really, really smart technology folks in the lab, and my role is to represent users and help us focus on doing things that make sense to our customers.

Before this line of work, my formal training and my experience had been as a practicing architect. I’m a second-generation architect and practiced in upstate New York, in Ithaca, for about 17 years. That’s my background.

Sean:  Thanks. Could you tell us a little bit more about Autodesk Labs? We’ve visited the site and taken a look at it, and it looks like what we sometimes refer to as a “Pirate radio station.” It’s a site that’s outside realm of the main corporate site.

That’s useful in terms of building a sense of community, or running certain kinds of product releases early to get feedback. So tell us a little bit about how that site got developed, and your particular role in leading it, day to day.

Doug:  I think you got it just right. I wasn’t around when Autodesk Labs was established, but I think it’s been around for just under a year. It is a way to engage customers and users. It allows us to put up trial balloons for new things that maybe the engineers cooked up.

Every so often, we get requests through entries in the blogs, or there’s someone from a discussion group requesting a certain feature. One example was somebody wanted an easy way to publish their model information from Autodesk Inventor to our Freewheel site, which is our zero client way of viewing both 3D and 2D model information. Folks at the labs took it on, and I think within three weeks or so, they had something up.

It’s really a test-bed. We have the ability to maybe have a little bit less process than other parts of the company, because this is not production quality kind of development. It’s really about offering as much as you can, getting it out there, with the hope of getting early and direct feedback from our users.

Sean:  So would you say the site predominantly informs the mainline products that you develop, and that things from the test bed may make it into the products?

Doug:  This is a place for us to test out new things. We’re looking to see if things are possible, to see if customers resonate with some of these ideas. I guess it’s fair to say that we have a little bit more leeway in terms of the things that we pursue, because if it works, great. But, hey, part of it is also finding out what doesn’t work.

Scott:  In terms of the site’s influence, what are some things you could point to that have been canted or targeted to a different direction based on the discussions or feedback on the Labs site?

Doug:  I think we’ve gotten some good feedback, some early feedback on a number of early concepts. One of the so-called graduates of the lab is something called Autodesk Impression. This is a way of taking CAD data and kind of putting different filters on it and rendering it. As I understand it, there was good, direct feedback, and the development team was able to take that into account. We felt confident enough that it is now a product.

Another product that we’ve pulled together is called ShareNow. It’s a plug-in, basically, for Autodesk Inventor, Revit, and AutoCAD. It came directly out of a request that was fielded through Labs. This lets you upload and share your 3D and 2D model data directly into a Web application where anyone can view and navigate the model.

I don’t know if you’ve had a chance to look at the site lately, but just the other day, we got an early version of Content Search posted. There’s also something called Project Showroom, which is an experiment to see how people feel about being able to view virtual spaces. Plans are rendered on the fly, and tested out with different products.

Those are examples of things that we’ve done.

Sean:   I’ve seen both of those. The Showroom one actually stood out to me as well.

Scott:  In terms of the lab, how does that process work? When somebody requests something, how do you prioritize what to work on and what to move forward with?

Doug:  I think it’s on a case by case basis. It’s a fairly small and fast team. We get feedback, not only from the discussion groups, but we’ll go out at customer events, and even arrange our own visits with folks. We just try to use it as a way to gather feedback. There’s no formal process that we crank it through. If it seems to make sense, it looks interesting and we have people that are available to do it, then we’ll go for it.

Scott:  OK. Which I guess makes sense, but it’s kind of based on a feeling of whether the idea really has legs and would really benefit people?

Doug:  Yeah, that’s it. I’d say that, based on our experience in the design industry, Autodesk has a good feel for what might benefit users. We’ve gained a wealth of rich knowledge from customers over the years (through customer surveys, beta testers, customer councils, AUGI, and the like), that’s helped provide a base of understanding to work from. We also want to take advantage of my background from the Institute of Design. I think one of the things that we’re hoping to do is get out in front a little bit, and do a little bit of generative research. Go out there and do some observations and interviews with folks, much like the way you guys are learning more about this open source notion by talking to people.

I think that, if we had a chance to do some of that research, analyze and synthesize, that would also help inform the direction in which we move, in terms of what we should be developing. One of my roles, one of my goals, is to keep focus on what users are really doing and trying to do, and focusing on user centered needs. Quite frankly, at this point, with the technology and with the experience of the developers that we have, a lot of things are very possible.

Scott:  How much of what you guys are doing was inspired, or triggered, in part by what you have seen open source projects do, in terms of developing in the open, making sure that people have a really clear line of sight to a project from pretty much the light bulb going off and the idea being conceived, all the way through development?

Doug:  I think that there are some interesting parallels. I think that what’s interesting to me about open source development is this notion that you’re making things completely transparent, that you’re going out and involving a large community of users, so you’re getting representative use from a lot of different areas.

If those are some of the guiding principles for open source, I think those are some of the principles that we’re trying to use in Autodesk Labs as well. It’s really to have that focus to be on being transparent and trying to get a full understanding and get input from our users and our customers.

Sean:  I trot around with an iPhone and a Mac, and I’m interested in what’s been called “peak experience.” Do you think open source can reach the level of peak experience, from a user standpoint, and usability standpoint? It feels like it’s easier for closed-source to have a rock-solid vision, but I’m not sure. I’m personally on the fence; I feel I’m still investigating this.

Doug:  It’s a tough problem, whether you’re open source or closed source, to come up and nail that incredible customer experience. And I think you touched upon it a little bit when you talked about having some kind of rock solid vision. I was thinking about that earlier today, and specifically thinking about Linux.

Torvalds had a rock solid vision, did a bunch of work, and made an open source development community. So it’s really possible to get things jump‑started.

Whether or not, you can turn it into the iPhone experience without someone constantly driving a singular vision and tying together maybe all the different constituencies somehow, I think that seems to be one of the challenges.

The other thing I was thinking about relative to this was… I’ve been reading this book, “Designing Interactions,” by Bill Moggridge, one of the founders of IDEO, and he has these really interesting interviews.

In one of them, he’s interviewing David Liddle, who worked on some of the original Xerox Star interface. And he explains development of technology and he breaks it into three phases; one is the enthusiast phase; the second is the professional phase; the third is the consumer phase.

I don’t know for sure, but I have a hunch that maybe the open source side is really great at the enthusiast stage but that what you really want are really quick and rapid iterations and lots of input from lots of sources and you’re maybe not as concerned with everything being all exactly worked out. You’re really just interested in pushing the ball forward as quickly as you can.

When you then need to move that forward into the professional phase ‑ when it needs to get more robust and you need to maybe, in some ways, to clamp things down a little bit. And then, at the consumer phase, that’s when you need to pull it all together and simplify it and make something beautiful as an experience, not just the way it looks.
So my hunch is that maybe some of these different development models are a bit more applicable to different parts of where a technology might reside in each cycle.

Sean:  OK Fair enough.

Scott:  One of the things that you said the lab lets you do is iterate and try things out and not go through as much process.

Doug:  Right.

Scott:  One of the things that we’ve found in talking to closed source companies is that, when you come out with a new version of a product, there are business reasons to make fairly substantial changes to the product. You want there to be enough of a delta between V2 and V1 of a product that people see a reason to upgrade and they see value in the new version. In closed source, your previous version is competition.

This pushes closed source, proprietary products, to trying to add a lot of value from one version to the next. Open source, on the other hand, can move much more incrementally. The people working on the Linux kernel, or working on a patch, they’re just out to get it right. They’re not out “selling” and upgrade of the web server or the kernel.

The challenge is that sometimes companies will think up a fairly grandiose idea, but when they actually ship it, it ends up being a flop and a lot of wasted effort. Do you feel like that’s one of the reasons for the lab, is to really track with what people are going to use and to keep your pulse on what people are going to find interesting and compelling? Or are there other kinds of benefits you get that I’m missing in my analysis?

Doug:  I think that one of the main reasons for Autodesk Labs is to get that early feedback. You guys are familiar with the software development process. By the time you are really at the stage where you’re giving early access to your product, in a normal product development cycle, it’s really almost too late to make any significant changes.

I think the theory, anyway, with a place like Labs, is that you get early leads and do quick prototyping, with the expectation that it isn’t production quality, but with the goal of really gaining user input and feedback. That’s really important.

I’ve noticed a difference, since my first tenure at Autodesk, that this is very important to the company. When I worked on some of these earlier development products at Autodesk, our team, didn’t know what we were doing was bleeding edge or anything, but we actually went out and we were doing early prototyping and sharing early versions of our ideas with our customers, even before things were coded up.

And I think, at the time anyway, that was a fairly unique process at Autodesk. And yet, today, if you look around, at least from what I see in the company, there’s a huge effort to go out and gain this early user feedback and use it to inform the process, early enough in the process.

Scott:  For a long time in business, it seems like the thinking was: “Absolutely don’t show the product until you have to. Be worried about what competitors will do if they see the direction that we’re going.”

Doug:  Right.

Scott:  ”Try to keep it under wraps.” And then, when you release a version, release it with a lot of fanfare. Now, the trend has moved to be very transparent, and have a really good dialog with your users, and get some of their ideas and enthusiasm to infect your development team so that they’ll build really cool stuff. What happened to the fear of the competitive threat? Is it there, but people have just recognized that they just can’t really worry about that? Or how was the thinking able to shift?

Doug:  My perspective of that is that I think that companies are finally starting to realize how really important it is to understand the customer. 10, 15, 20 years ago, the people that were using applications, using computers, were willing to put up with a lot more heartache, because they were enthusiasts and you didn’t have to really do as much to get people to like what they were purchasing or using.

Now on the user side, people are more mature in terms of their attitudes towards the use of technology, and they have quite a bit to say. It’s great that we’re changing this around — going out and really listening — and using that to inform our thinking.

Scott:  Is there a concern that competitors might look at what you’re doing and get there sooner? Or is that just the risks you have to take to do business?

Doug:  What tends to be happening is that, as companies are being forced to do things quicker and quicker in very fast cycles, the development times are way down. I don’t know, maybe because things are so quick cycled, there’s a little bit less concern about somebody else catching on and doing it. You can’t stop. You keep on developing and you keep learning and you keep moving the ball forward.

Sean:  You said earlier that open source projects often exist to initially solve a technology problem. And then there’s this life cycle of the products as they go through, and it eventually gets more refined and distributed to consumers.

Do you see any open source projects that you feel exhibit a good design aesthetic? I mean, I’m not of that field, so hopefully I’m using the appropriate phrase, but have you looked at something where you say, “Look at that. That was built in an open source model. I love the way they did design. They really seem to understand the usability aspect, and I would think that’s a good model of it.”

Doug: Since I figured you guys were doing all this research, I wanted to ask you that question. [laughs] But I guess you haven’t found the answer yet.

I guess I’m not aware of any of the open source efforts that have this slant, or ones where design seems to have billing, in the holistic sense. Maybe people haven’t reached the point that that’s an issue yet. I don’t see why it couldn’t work. If there was kind of a call to arms for designers to get out and solve at the interaction level or visual level or how the applications or platforms are working, you’d probably see kind of a community of designers get involved. Communities really have taken off at Autodesk. Labs is just one; there’s a manufacturing community, a civil community, and of course AUGI, our international user group.

Sean:  In open-source, it seems that credibility resides, to a large extent, in technical capability.

Do you think it’s, perhaps, nothing more than the habitual disconnect between the person designing, who has a different focus for their intellectual tasks than someone who’s simply trying to write code, and it’s kind of an impedance mismatch that never really fully settles itself out?

Doug:  Yeah, I think you’re right. Different people think in different ways, and in terms of the design side, those are different skill sets that maybe aren’t being called upon yet. This, again, doesn’t mean that that couldn’t happen.

Design is kind of part of this open source notion. It’s not really open source to the same extent, but you see every day in this little Web 2.0 world, people doing mashups and taking components that they’re finding from anywhere and kind of putting them together and making really interesting applications.

And I think, at least in the Web 2.0 world, design does seem to play a role in that. Not only as the underlying technology, but also the way things work, the way they feel, the experience that they’re giving to their users.

To me, that’s along the same lines as that open source on the design side, kind of this democratization of the process. Anyone can participate. I think that’s a really strong idea to, perhaps, get some people to talk about and challenge the design community to get involved in one of these things.

The other example, and it’s not a direct one-to-one example of software development, but there’s a group called Architecture for Humanity. Are you familiar with them?

Sean:  I’m not. I’m not.

Doug:  They gave a really interesting talk at one of the TED conferences, and actually ended up winning one of the prizes. They created an open source design network for solving architectural and environmental problems.

And what they’ve done is created a way of getting designers to input ideas into this network, and share out all these ideas with anybody who wants to participate and work.
One of the mechanisms that they used to do this was what’s called Creative Commons licensing.

Sean:  Yep.

Doug:  I just think that that’s another hint that there are certainly opportunities for the design community to get involved in the open source development world.

Scott:  And I’ll just make an observation on that. Open source seems to work when it’s used in a single discipline environment.

So if you want to contribute to Open Office, you pretty much have to be able to write code. If you want to contribute to the Linux kernel, you pretty much have to be able to write code. If you want to affect the user interface, you have to be a coder, rather than a pure designer, to do it. There isn’t a lot of ability…

Doug:  In the environment that exists today.

Scott:  In the environment that exists today. And what you talked about with architectural design was, again, fairly single disciplinary, right? It’s these architects sharing these designs.

And so you find this a lot, and I think one of the chasms that open source hasn’t yet figured out how to cross is to pull together an interdisciplinary team to solve a problem.

Doug:  And I think that’s where some of the really interesting things get solved, is when you’re working on teams that are made up of different individuals of different backgrounds and different perspectives.

That certainly happens on the open source development side, because you’ve got people from all over the place participating. But I’m not familiar enough with the way the process works. Do they ever get in the same room, or share a whiteboard and talk with each other? Or is it really just in the check‑in, check‑out of code?

Scott:  It depends on the product. Some products, like MySQL, really are open in the fact that the code is out there and somebody could fork it, but everybody who’s working on it works for the MySQL company.

Other things like Apache and the Linux kernel and a lot of other stuff, really is built over a mailing list, and they never do physically get together and whiteboard stuff.

Doug:  Maybe some of the limitations are there just because of the way that the environment, as we’ve been talking about it, has been framed out. I mean, what’s to say that if I wanted to get in and share some ideas and thoughts about the user interface with Linux, that I couldn’t just post some bitmap graphics to the nodes or a paper on some ideas that I wanted, not necessarily checking in and out source code and coding.

Scott:  I’m going out here on a limb, and if I’m off base I hope readers of this interview will comment and correct me, but it seems like open-source emulates (dare I say, copies) work that interdisciplinary teams have done. I don’t know that there would be an Open Office if there hadn’t first been a Microsoft Office.

Doug:  Right.

Scott:  The Linux kernel was obviously patterned after UNIX. Apache, on the other hand, was built from scratch as a web server. A lot of times open-source isn’t doing things that are interdisciplinary, or if it is doing something interdisciplinary, it’s because those interdisciplinary skills exist in certain unique individuals. You have a cryptography expert who can code. You have someone who can both envision a better process/thread scheduler, and can code it up as well. If you’re just a mathematical researcher, but you’re not a coder, I don’t know that you have a mechanism to get the fruits of your research into the product. You have to hope a coder will pick up your research and run with it.

Doug:  Right. You guys are writers, and write books and such. I guess the question that I have is for a process like creative writing, say take creative writing, could you imagine writing a novel by open source methods?

Scott:  Well, sure. Or definitely you can write an encyclopedia with open-source methods.

Doug:  I’m talking more about maybe something less technical.

Sean:   In other words, something that has a plot, a logical consistency, the characters don’t jump from one island to the other. So it’s not like the grammar school game of the story starts out on row one and by row five, essentially they’re slaying dragons on space ships all of the sudden.

Doug:  Yeah, these are just questions that I don’t know the answer to, but I think they’re really interesting ones to ask. I mean, whether or not the visual kind of design, where you’re working in concert with each other, can produce that kind of deep user experience, using an open-source model… I don’t know.

Sean:  Let me switch gears for a minute here. In terms closed source, talk about how Autodesk handles the process of weaving design in as part of a product. Walk us through the life cycle of a feature that required high design characteristics, and how that came about from inception through production.

Doug:  I can tell you a little bit about the initial way we developed the Autodesk Design Review product. Maybe that’s a good way of sharing how we went about doing that.
I was a senior product manager and I was working very closely with the senior product designer. And as a team, we went out to gather requirements. We had product management and product design go hand in hand, out to the field to talk to, interview, and observe customers.

So instead of starting with, “OK, we have this technology, what are we going to do with it?” we went out and talked to customers to see what they really needed. And during that process, we started getting hunches about the need for different kinds of tools to help people review and view their design data, and so we started doing prototypes.

The prototypes were not technology prototypes. They were not coded, they were graphical visualizations of the way these applications might work, or the way certain features might work. And in that process we would share this with these users and get a ranking of what’s important.

It was really interesting because every time we would go out we would think we knew that, “The customers are really going to love this red pen feature that we’re just crazy about,” and we’d have to have them rank it, and guess what? The red pen thing, they didn’t really think it was that important.

Sean: Out of their hundred dollars they spent zero on that feature.

Doug:  Right, exactly. And that was on a consistent basis, so it really comes down to not necessarily what the designers of the project think, it really is what do the customers think, what do the users think of this thing? Because we had done early prototyping and got it out there early, we didn’t have code yet. One of the things that we identified very early on, which we almost didn’t even pursue because people were saying, “Oh, no one cares about that,” was the issue of roundtripping redline mark-ups.

This is the ability to present an AutoCAD drawing, publish it out, have someone basically do a mark‑up on it digitally, and then ‑ and this is the part that we learned ‑ then getting those mark-ups back into the original CAD file. That was huge!

Before we had gone out, and people were telling us, “Oh, nobody needs redlines, they don’t do redlines.” The key thing we learned is that it doesn’t work because it doesn’t round trip. So that was a really great example of how we just didn’t know, we had to go out there and learn by interviewing and observing and really finding out what people really thought.

Scott:  On the closed source side, it is pretty easy to sort of put together an inter‑disciplinary team. Microsoft will sometimes even throw in a cultural anthropologists, and things like that. How do you resolve issues, though, if you’ve got designers who have envisioned something that might be a great user interface experience, but the coders look at it and say, “You have no idea how impossible that’s going to be to implement.”

Doug:  Well, we talked about having multi‑disciplinary teams. True multi‑disciplinary teams really involve the technologists as well. On these customer visits that I describe, we took care to do, we brought along coders and designers to come out into the field.

They just loved it, because then they were able to see firsthand the results of who they were coding for– the problems that were actually out in the field. We worked as a team, we really did. We had equal footing between the different disciplines, and I think that’s really important to have with all the constituencies represented.

Scott:  And I guess that probably fostered a certain amount of appreciation between the designers for how creative they could be, if it was going to be impossible to code, and the coders for, “OK, I can’t just say no to that,” because it really seemed to resonate with the client.

Doug:  Exactly right; appreciation, experience, knowledge, when you go out and you solve these things as a team. We would have our heated discussions, certainly, but what you have there is a situation that people were really promoting their constituency. I would be the one to be the voice of the customer.

Someone else would say, “Well, we’ve got to do this on the technology side.” Someone else would say, “I want it to look and feel like this on the design side.” And I think when things get out of whack is when any one of those things gets out of control and leads the whole effort.

Scott:  And how those things get arbitrated? Was there somebody who was something like a senior product manager who would take all the input and make a decision at the end of the day?

Doug:  The working relationship on the teams that I’ve been a part of has been very strong and with most things, it wasn’t that someone had to come in and arbitrate and say, “OK, we will do this.”

Most things we really did try to do as a consensus. It really wasn’t a vote. It just happened after in-depth discussion of the issues. That takes a certain level of maturity on the part of the participants who are willing to not just represent what they believe but also put themselves in the shoes of others, including the users and the people that are building it.

Sean:  Is there anything you want to add? We try to give everyone a chance to get in closing thoughts.

Doug:  I just think that this is a really interesting discussion and it’s made me think a lot more about some of the questions that you’ve been asking. I mean, what is the role of design and how does design work with technology, and to me, it’s a higher level question and answer than even the open vs. closed source model. It’s ways of working and ways of thinking about things. Thanks for the discussion. I found it very interesting.

Sean:  Thanks for chatting.

Bookmark this:

And one of the clear elements of their ascendancy is that they have a clear vision for usability and peak customer experiences. Notice I didn’t say “peak corporate experiences” but that is a topic for another post I’ve got planned…

Most people seem to point to this focus on peak customer experiences as being driven and shaped by one person – Steve Jobs.

So here is the question.

Do you need a Benevolent Dictator to create great out of the box experiences that extend for the lifetime of the product’s use by an end user?

And is this true for both OpenSource and Closed Source projects.

The folks at the humanized blog had a great post up about this that motivated me in part to put up a post on our blog.

But in addition this is a question that has come up quite frequently in our interviews and is the highlight topic of an upcoming one soon to arrive on the blog.

So what are your thoughts?

I’ve got my own two cents on this which is that you do need one regardless of whether it’s an OpenSource project or Closed Source.

But I’m curious what other folks think as well…

Bookmark this:

If you are looking for subjective opinion, I recommend looking through the interviews we’ve done here. At the risk of sounding like a Microsoft fan-boy, the Microsoft interviews (in my opinion) demonstrate a company where secure coding is “in the water”. Code goes through threat modeling, risky function calls have simply been banned, code goes through automated and human inspection, and vulnerabilities that do slip through feedback into the process to determine how to prevent them in the future.

I simply don’t get the same feeling from the open-source people we’ve talked to. When we’ve brought the subject up, the response is almost universally “many eyeballs,” and faith (without data) that “many eyeballs” is effective.

Am I completely off base? Do things like the Linux kernel and Apache go through rigorous security reviews? Is there proof that “many eyeballs” in open source is at least as good as something like the Security Development Lifecycle in Microsoft? If you’re in a position to know, let’s chat!

Bookmark this:

Update: It seems that this isn’t seen as happy news by all. There’s an article on eWeek that’s just too irrational and frothing to pass up, claiming that this is all a ploy by Microsoft to kill Mono. As Microsoft is officially supporting Novell’s efforts in porting Silverlight to Linux (on top of Mono), the evidence would indicate that Microsoft is doing this to support .NET developers, and not as some clever conspiracy to kill off Mono.

Bookmark this:

Interviewee: Robert Bray

In this interview with Bob, Architect for Geospatial Products at Autodesk, we asked him about:

• Introduction of open source within Autodesk
• Role of project steering committees
• Decision to create an open source product
• Open source impact on closed source product
• Developer and engineer adjustments to open source process
• Differences between feature design in open and closed source
• Test and documentation for Autodesk open source
• Community contribution to open source projects
• Business reasons for creating open source products

Sean: Bob, why don’t you give us some background on your role with Autodesk.

Robert: I’ve been in software development working for independent software vendors for about 17 years now. My current title is Architect for Geospatial Products. So, I’ve gone through the various ranks, from being a programmer in the trenches, in the cubes, to managing a product development team, to my current role as architect.

So, as I said, I’ve been through various things, mainly in closed source. Until roughly 18 months ago when we open sourced MapGuide. I think it was about six months before that that we really started heavily investigating that idea and started to put the ball in motion to actually make that happen. Basically, I’ve been involved in an open source project very actively for the last 18 months, and I won’t say that I was the key driver that made the decision that we were going to open source this, but I basically took it from that decision point and said, “OK, these are all the things we need to do…” and made that happen.

The product itself is MapGuide.The open source version is called MapGuide Open Source. The closed version is Autodesk MapGuide Enterprise. The codebases are the same. There’s one open source code stream for the whole product. There are a couple of minor functional differences between Enterprise and open source, but they are extremely minor, and we are trying to close as many of those as we can, actually. What we really have with the Enterprise version is basically a commercially tested and supported version. It’s much like the Red Hat model or the MySQL model, in that respect.

Sean: So, just out of curiosity then, if there are some functional differences, what led to those? I mean, they’re minor, like you said, but was some of that just by the nature of having to switch from closed source to open source? Is there some functionality that just couldn’t make it over that transform right away?

Robert: Basically, that is exactly it. We have a commercial desktop mapping product called AutoCAD Map 3D. We used some technology from that in the Enterprise version of MapGuide, and we do that for compatibility between the two products. So, those particular pieces, our coordinate system library, in particular, and our raster image handling library; those are things that we just could not open source at the time.

Sean:So, when Autodesk made the switch, did you model against an existing open source project in terms of how you wanted to handle QA or how you wanted to establish a community feel? You look at different products, like Subversion – I’ve heard they had one vote in the entire history of the project, right?

Robert: [laughs]

Sean: Literally. I watched their presentation at OSCON, and at least that’s the public face of things – that they had one vote. Or, for example, we talked to PostgresSQL yesterday, and they said, “It’s a very communal democracy” in terms of their decision process. They don’t really have a hierarchy. I know it varies across the projects, so did you look at the open source community and say, “Well, now that we’re going to open source this, we’d like to mimic this particular project” at least in some aspect?

Robert: Absolutely. So, when we made this decision, one of the first things I did was take a broad look across all of the projects. I was mainly focused on looking at the geospatial projects, because, if you will, our kin in the community is all of the open source projects – things like MapServer, GDAL, PostGIS… All those open source projects are very close in nature to us, so I certainly did look at those.

I also read a couple of books. The one I liked best was Karl Fogel’s book, “Producing Free Open Source Software.” Of course, he led the Subversion effort, but if you read that book you wouldn’t know there was only one vote in Subversion.

Sean: [laughs]

Robert: [laughs]

Sean: I was a little surprised, too. But, these were two guys from Google, so technically they’re reinforcing each other in their viewpoint, at least. It’s not just one guy saying it. They’re committers on the project…

Robert: I’m shocked.

Sean: Yeah, yeah. I was a little bit, too. But that’s part of the fun part of this investigation is some of the stuff we hear, and then we get to cycle back and say, “Really? So tell us a little more about that.”

Robert: So, we definitely took the approach of forming a community, and the project, and I say not from day one, but within six months, became a community driven effort. MapGuide has always been a product that, it’s almost like there’s a religious following of users out there. Even when it was a commercial product, it had some real hardcore, dedicated users.

So a handful of those users said, “Wow! This is going to be great!” They knew nothing about open source at the time either, but they kind of all jumped on board right away, started testing the open source version, and started getting active in that.

So, when we formed the community, basically, because we were modeling after MapServer and some of the other open source projects on the geospatial side and I really felt that was important we basically created a project steering committee for the product out of that group of people. So that project steering committee, right now, consists of three people from Autodesk and four people from outside in the community.

Sean: Are those people then the chief committers on the project?

Robert: Actually, that’s an interesting thing, because I would say there are a couple of people there that are active committers; the rest of them are users.

Sean: Got you. Because the interesting thing to me about this is that, obviously, a lot of these projects have grown up over time. PostgresSQL has a 20 year history, and to become a committer on their project, they told us, “Well, we wait two to three years to figure out if we can make you a committer, because we’ve been around for 20 years.”

But, if you birth a project out of the ether how do you determine who’s going to be a committer? Because, obviously, there’s probably a stampede of people at the start that say, “I’d like to be a committer, I want to be involved at the highest levels,” that kind of thing. How do you go through that winnowing out process?

Robert: There’s a URL I can send you, (http://mapguide.osgeo.org/psc.html) but basically what I did was draft a set of rules for the project steering committee to follow that include who can be a committer and how they become a committer.

We basically go through a similar process to, I think, Postgres – although we probably wouldn’t be two years in the road doing it. Basically, people start submitting some code from the open source community. They submit it to a developer, we’ll have that developer review it, and then do the actual commit. It’s almost in a patch style, right? Then, once we’re comfortable with them, we’ll vote to make them a committer – it’s a project steering committee vote.

Sean: Got you.

Robert: So, it’s kind of interesting, because, again, the project steering committee is not the core developers; again, it’s a core set of mainly users.

Sean: Once you transitioned over and said, “I want to make this an open source product, ” since you came mainly from a closed source background, what was the thing that most pleased you about being able to transition into an open source development model? What was the thing that you were, I guess, either most afraid of or most reluctant to see transition over? Because, obviously, the models are pretty different.

Robert: I think I was scared to death for quite a while…

Sean: [laughs]

Robert: …because it was completely unfamiliar territory to me. I’ll be honest, right? I’d never worked with an open source project before. I’ve used a few, but I’ve never really participated in an open source project. So, from my perspective, open source projects were these big free for all things, and I’m like, “Oh, my God. What are we getting into here?”

But, when I stepped back and actually looked at it, what I realized is that open source projects…yeah, they run a little differently than commercial projects, but they follow a fairly well-defined process; or at least the large projects do, that are successful. You can see a well established process there. It’s not a free for all, anybody can commit code, and a thing like that. So, that was my biggest fear going in; it was the step into the unknown, right?

Sean: What do you think was the biggest lesson learned from that? Now that you’ve got a couple of years at it, what would be the thing you might’ve done a little bit differently, given an opportunity? It’s apparent you wouldn’t have changed any decision to go open source, and that rings through clear. But, is there anything you would’ve changed during the process of going in that direction?

Robert: If anything, I would’ve been more open with the community right from the get go. I think I had a lot of reservations going into it, and so I didn’t really communicate as much as I should’ve right up front about what we were doing, how we were going about it, and all of that. So, I think I would’ve been much more open around communication right from the get go.

Sean: OK…

Robert: That’s the biggest lesson I think I learned.

Sean: …then what was kind of the biggest positive that you got? I didn’t want to leave that off the table either.

Robert: The biggest positive is that from a community perspective we are getting a lot more up front review; a lot more users using features before they go into, say, the commercial version or whatever, right? We’re getting a lot more early feedback on things that we’re thinking about doing or the ways we’re thinking about implementing new features, lots of early feedback that has a very, I would say, positive influence on the end result of the code that we actually write.

Sean:Well, I’m curious, too. That’s a functional change in the product, but have there been any change to the engineering process on the closed source side of the product? So, you guys open source it, that obviously rattles around and after a couple of years; I’m wondering if that has had any kind of change in the way you guys have developed internally, just having an open source mirror of the project, so to speak.

Robert: Well, it’s not an open source mirror. We only have one copy of the code, and that’s the public one. So, any change that we make to the code, whether it’s for the commercial release or for the open source release, typically gets vetted through the same process. We’ll go to request for comment. That’ll go out to the open source community, because it’s going into the open source version, one way or the other. So, we get early feedback on pretty much everything that goes in.

Of course some features that we may plan are for the enterprise version only, and those of course do not go through public review. Also we reserve the right to decide which features go into which enterprise release and we make no announcements about that because we are a public company right?

Sean: Right, right.

Robert: We don’t comment on that. We don’t do anything like that in the RFC process. But at the same time, the open source community certainly gets to vet all this stuff early. So, any code changes, and basically the whole product’s development process, are morphed into more of an open source development process for both the commercial and open source releases, because, again, we only have one copy of the code and we have one development process.The entire engineering team has basically transitioned to an open process, which is quite interesting.

Sean: Taking it down one level, as we talked to people sometimes as they’ve gone through this change, and the engineering team has a variety of differing reactions to it…various resistance points, or to put it maybe slightly more diplomatically, just things they have to learn as they go through the process. So, anything that would be worth talking about there in terms of lessons learned by the engineering team as they walked through this?

Robert: Yeah, I think there were a lot of lessons learned. Immediately it comes back around to being open about the kind of change we want to make and how you actually communicate with an open source community, because communication within the internal engineering team is quite different in my experience than communication with an open source community.

Sean: Now all of a sudden it’s a much more communal feel and development managers all of the sudden, I assume, had to make the switch from assuming everybody was on the payroll; there had to be a little bit more negotiation that affected the process of getting something developed, vetted, built, etc.

Robert: That’s definitely part of it. But also for our internal developers, I mean, they’re used to this kind of hierarchy of being able to walk down the hall and talk to somebody else. They’re used to being able to call a meeting and go in and start scribbling some stuff on a whiteboard.

With an open source development community that doesn’t really work anymore. You need to use the mailing lists a lot more. You need to use things like IRC for communications. So, it was really a complete change in the way you communicate.

Scott: So, that’s one of the things that I wanted to drill into because you’re obviously steeped in closed source development methodology. You’ve been in open source long enough to where you’re now immersed in that.
What do you see as the differences, kind of taking them side-by-side when you think about the aspects of the SDL, right? …how’re ideas envisioned, how they’re kind of tasked for somebody to code them, how they’re put through QA, how a product is released… If you don’t mind, I wouldn’t mind just kind of walking through how each of those is sort of fundamentally different in a closed source versus open source project.

Robert: Sure.

Scott: So, I guess starting with analysis and design and that sort of thing, how do you see analysis and design and requirements differing in open and closed source?

Robert: Sure. The requirements from a closed source perspective, I mean, we have project managers who basically go out and talk to our users. They’ll do focus groups. They’ll do surveys; all kinds of things like that just to basically pin down to a set of requirements for release of commercial software.

For open source software, you have a community of users who propose some ideas or a developer who proposes an idea. It gets kicked around on a mailing list for awhile. Eventually somebody either just picks it up and implements it, or the other option is they just walk in with something they’ve already written and say, “Hey, how about this? Isn’t this great?” So, the difference is really to me that it’s a little bit more organic, if you will, in the open source community. Whereas analysis and requirements and product definition is a much more rigid process in the closed source world, in the open source world it kind of happens organically.

Scott: So, where do you see those potentially? I’m guessing that both sides have their advantages. What do you see as the advantages of focus groups, meeting with customers, and project planning and what do you see as being the advantages of kind of more of the organic method of sort of showing up with a feature and saying, “What about this?”

Robert: If you show up with a feature and it’s already got a prototype or an implementation done – it’s a lot easier to talk about because you can actually see it and you can play with it. Whereas in the closed source side, you’re talking more in the abstract, if you will. So, the advantage in the open source side is really that you typically get a lot more users providing some feedback on it, at least if they’re really interested. Eventually you get to experience it before it actually goes in, which are both useful things.

On the closed source side, I mean, the advantage is that the release potentially comes out with a set of features that are much more…I won’t say well-defined because I think that’s the wrong term, but a release will typically come out that is much more cohesive as a whole. Instead of having a bunch of new things sprinkl