Interviewee: David Campbell

In this interview with David Campbell we talked to him about:

• How the development methodology of SQL Server has changed.
• How the changes affect the ability of customers to give feedback.
• How virtualization makes it easier for users to test-drive SQL Server.
• As products and open-source projects grow, the developers have to realize that they are not the target user.
• David’s belief that the best product doesn’t necessarily have the most knobs.
• Building a community around a closed-source product like SQL Server.

Scott Swigart: David, thanks for taking the time to chat with us again. One of the things you mentioned in the first interview was the common engineering criteria. I’ve seen that mentioned in a number of different places, but I have never seen it explained. And I’m guessing that the common engineering criteria impact a product like SQL Server. It might be good to start off just by explaining what they are.

David Campbell: It started in the Windows Server system. We’ve been telling customers that one of the advantages of our stack is that it works better together. It does, but we started really challenging ourselves by asking, “OK, what are the real things we’re doing to actually make it be better together?” The way I think about it is, “What are the things that would add value for customers who require coordination or consistency across groups that otherwise wouldn’t happen organically?”

We started this a number of years back, when Paul Flessner was running the Windows Server group. We have a set of requirements, which are updated every year, and all the enterprise products within the Server and Tools Business (STB) have to adhere to those requirements. These are things like having a best practice analyzer, having a Management Pack for System Center before you RTM, having consistent user education, continuous publishing, a community engagement program, that kind of stuff.

Scott: Previously when we talked, you also mentioned that SQL Server went through a pretty big change in its development methodology between 2005 and 2008. Talk about that a little bit.

David: This is something that a small group started, and it ultimately took great effort across the division to pull off. It will never be “done,” but it has been an amazing, fascinating transformation. I’ve learned more in the last few years about culture and change management in a large organization than probably anything else.

I’ve been working on the product for a long time. Back in the good old days, we had something like 20 people who knew the whole server, end to end. Back then, as you worked on things, if you had an issue you just walked across the hall, sketched something out on somebody’s whiteboard, and went back and coded it up. Because things were so intimate, you could keep things moving at a rapid pace.

To somebody who understands the standard Microsoft development methodology, one of the things that’s very important is the daily build. We continuously build and test the product, and one major vital sign is the health of the daily build. Back when the product was small and there were 50 to 100 people working on it, you could do the daily build live, with everyone just checking stuff in and keeping it together.

But as the team grew and the size of the code base grew by several orders of magnitude, you just couldn’t have as many hands in the soup at the same time and get anything that tasted good.

In SQL 2005, we had sort of a standard large-team, waterfall model. If we were doing a feature that spanned several component groups, we would have one group get together and they’d do everything by the book. We had a process and everyone followed it. They’d write the spec. They’d develop a test plan. They’d review the test plan. They’d write the code. They’d test it. They’d check it in.

Then the next component team in the sequence, maybe it was the client data access team, would pick it up and they’d start working with it. And they’d go, “Hey, wait a minute. This interface isn’t quite right. There is something wrong here.” They’d go back to the first team and ask them to change it. Of course, the first team had moved on to something else, so if they had to go back and do major surgery, it was a hassle.

We had one feature in particular that, to be honest, we built two or three times during the course of SQL 2005 and we still didn’t have it right. So I said to the program manager at one point, “Look, either we take it out of the product or you get everyone, end to end, involved, get them in a room and just sit till you work it out and get the thing going.”

This involved the storage engine, the language parser, the query processor, the client data access libraries, the management tools. They had done it serially several times. They just never had it work correctly, end to end, when integrated.

They all got in a room. Of course, you know how that works. They were able to figure it all out and get it all done. We said, “OK, what can we learn from this?” We completely threw the development process up in the air and changed it around. How do we go “back to the future” and try to capture the intimacy of what we had in the “good ol’ days” now that the team and product are many times larger and more complex.

When we had a small team rebuilding the database engine for SQL Server 7.0, a number of people had the whole system in their heads. Furthermore, when we did SQL Server 7.0, we really didn’t need to think about the market that much. We were building a new relational database engine and introducing an OLAP system. The playbooks were already written for these technologies and the markets were already established. Now, we did it in a way that was different; we tried to make the product much easier to use so we could actually grow the market and bring the technology to a much larger base. For SQL Server 2000 and, to a lesser extent, SQL Server 2005, we were building out on the architectural base we had established with SQL Server 7.0.

The original model worked great for several releases, but when we got to SQL Server 2005, the product was much bigger, the team was much bigger, and the market expectations were much higher. Things just weren’t fitting together in terms of consistency across the product. So the other piece that we had to add for the SQL Server 2008 process was, instead of defining the release from the bottom up in terms of what we were going to do, we had to create a definition process that looked both at the market, top-down, and at what needed to happen to the technology base of the product and what people wanted to do, bottom-up, and bring them together.

The way we framed it to the team was, “Look. We want to have the PowerPoint presentation that we’re going to use in the keynote for the launch before we write the first line of code.” So we developed a set of themes for the release. With each theme we had this notion of scenarios that were end-to-end value propositions for customers. Underneath those scenarios, we had a number of what we called improvements, instead of features.

An improvement is what we would consider an engineering transaction. The idea is that we bring virtual teams together to work on an improvement. An improvement team is cross-discipline—development, testing, and program management—and it’s cross-component. For example, the database engine, data access, and management tools teams can come together to work on a single improvement. So if a particular improvement needs focus from four different components, they’ll all come together as a virtual team.

We assign an improvement team lead, who’s got a set of requirements they need to meet to integrate the result into the source base, but we’re not very prescriptive about how they run the team. If they want to use Scrum and have stand-up meetings and do a series of sprints, that’s fine. If they want to use a test-driven development methodology, etc., that’s up to them. We’ve got tools and a playbook that we encourage them to use, but we’re pretty liberal in terms of how they go about it. There’s a very strict contract in terms of what needs to be done before it is integrated into the main branch of the product source code, however.

These improvement teams work in their own separate branches in the code management system. Once they believe that the improvement is ready to ship to customers, then and only then will we integrate it into the product. What’s interesting, if you think about it from outside SQL Server, from the customer’s perspective the release looks much, much different than it used to look.

It used to be the case that—let’s say we were going to do 300 features. Everyone gets started on them. We put a bunch of stuff together and put it out for Beta 1. Everything would work about 30 to 50 percent. Then we’d keep going for Beta 2. We realized, “We’re not going to be able to finish 20 percent of these things, let’s cut them.” With Beta 2, things are working about 70 to 80 percent correctly. Then you have Beta 3, a bit better. And in the end, you’re still trimming stuff out, cutting it, and refining it to get it ship ready for customers.

We were constantly in the mode of having stuff that kind of worked in Beta and trying to get feedback from customers. We flipped it around with the new process. What customers see now in a particular drop is very high quality. They used to see the 120 percent of the ultimate features in Beta 1 in some measure. Now they only see 20 or 30 percent of the features early, but they all work. We also changed the name from Beta to Community Tech Preview, or CTP.

As we go through each successive CTP, more and more of these improvements show up. They’re baked, end to end, and they accumulate in number. What a customer is seeing is that the quality is actually very good the whole way through, so it’s an interesting change.

Scott: It seems as if the SQL Server team went through the same process the Visual Studio team did: Instead of putting code out there that is 30 percent functional and people can’t really get it to work end to end, they decided it’s better to wait until a given feature is 100 percent even if that means customers see it much later.

But there has to be a trade-off, because your window for getting customer feedback is smaller, and you have potentially invested a good deal more time in the original code before you get any feedback. Talk about that balance.

David: That’s a great question. One aspect of the previous releases is that we were frankly playing catch-up in an established market. We knew what we wanted to build and we knew how we wanted to differentiate in the market so we really didn’t need a lot of customer feedback on the feature set for SQL 7.0 because in many ways we were rearchitecting and reimplementing the existing product. This approach worked for SQL Server 7.0 and 2000, but one consequence is that we hadn’t built up muscle around how to engage customers earlier in the design process for completely new features. In some sense, we built stuff and threw it against the wall to see if it would stick. This worked well for awhile, but it made much less sense starting in SQL Server 2005.

Basically, you could say that the old process was quite inefficient, in the sense that we built a bunch of stuff, actually committed code, before we even knew what would stick in the market. Now we’re trying to do that through more concept testing, more up-front work, spending more time with customers to look at their real pain points. Are you going to build a set of features that’s going to be compelling? That’s one piece of the puzzle. Are you building something all up that will be valuable to your customers to you customers end to end considering they way they really work? That’s another pieces of the puzzle.

But then the question becomes, have you built each feature in a way that makes sense to the customer and is useful? So it’s less about the value proposition of having a particular capability and more about have you designed and built each thing correctly, on an improvement-by-improvement basis.

One fallacy around the Beta process, in my opinion, is that we would throw a Beta out there, and tens or hundreds of thousands of people would download it. A small fraction would really play with it, a smaller fraction would actually think about how they’d use a new feature, and a fraction of those people would actually give us any real feedback. It wasn’t very structured. Our TAP program tried to target specific features and scenarios but it mostly kicked in after we had written the code.

I believe it’s a bit different in the Developer division, because you’re close to developers. One of the challenges we have with SQL Server is that, for our real validation, you need to get closer to a production environment. And it’s very difficult to get the feedback from the production team.

Scott: Right. With SQL Server, a single dev banging away isn’t a real test. You could get somebody banging away at the feature, but that doesn’t really tell you whether it will scale, whether it will be reliable, whether it will be secure.

David: Right, exactly. So what happened with this release was, we had some teams that said, “Oh, crap. What are we going to do to get feedback for this really complex improvement?” For the database engine there’s an improvement called a Resource Governor. It allows you to constrain queries that consume specific resources so you can limit a particular connection to 10 percent of the CPU, and that kind of stuff.

The question is the mechanics of getting the architecture and user interface right. We think very hard architecturally about separating mechanism and policy. In this case, the mechanism would be things like how do we keep track of the various resources and who do we attribute the resource usage to. The policy would be how the user describes her intent in a way that allows the mechanism to enforce it. In the case of the Resource Governor, there’s a complex interaction between the policy and the mechanism, – not the least of which is that we needed to build new mechanisms to hold and enforce various policies. This is a great example, because what we would have done before was stare at the whiteboard, do a design, and throw something together. We’d ship it out in a Beta and hear, “No, that’s not what we need.” So it turns into a rock fetch. “No, this thing stinks. … This thing stinks. … You’re getting warmer …,” etc. It’s a pretty inefficient process all up.

In the case of the Resource Governor, we got together with some of our MVPs and key customers who had a need for the improvement and did more of an outside-in design process starting from use cases. What’s interesting is that this outside-in approach is completely natural for many people building applications, but for a development team that’s been creating low-level system software for 10-plus years it’s quite different.

The bottom line is that we definitely needed to figure out how to engage customers differently with our new development process, and there are opportunities for design councils, concept testing, early prototypes, and early builds. In fact, things like virtual machine technology allow an improvement team to create a build of their early thinking and test it with a small group in a focused way without having to integrate everything else that’s in various states of completion, like we used to do with the standard Beta process. They just create a VHD with their bits and have some customers try it. With this model they can iterate faster than was possible with the old Beta process. Ultimately, I think we’ll get better improvement feedback by finding the right 10 to 20 customers and working closely with them rather than putting it together in Beta and throwing it out to 100,000 people hoping they’ll give us valid feedback. I’m overstating things a little bit to make the point, but the general concept still holds, although not everyone here agrees with me on this one.

Sean Campbell: I’m curious about the point you just brought up, because Scott and I have been knee-deep in virtualization for a long time. The point you just made about letting people evaluate it on a VM, how has that impacted the overall development process for the SQL Server team? Has it given you the ability to basically give, let’s say, broken bits out earlier and get more people working with it earlier because they don’t have to install it, they don’t have to configure it? I’m sure in the past people would try to install it on top of whatever else they already had. You could tell them a hundred times, right? And they’re going to try to install it as a multiple instance on top of what they’ve already got.

David: It has, in exactly the ways you said. These applications are so complex, and the state that they put on the machine is so complex, and we work so hard to make sure uninstall really uninstalls all the stuff that it put there in the first place.

The other piece is that it’s hard for us to build and test the setup and the installation. With VHDs, we can basically handcraft them and then clone them, and they don’t pollute any other state on an existing machine. So it’s much easier to get together. The bulk of the savings is in the setup, for us to construct both the setup and the uninstall, to make sure we don’t pollute the machine state.

So I think from that perspective, it’s very, very helpful. I think we saw that, but we haven’t really perfected it as a process yet. I think we’ll probably go a little bit deeper on it in the next release.

Scott: In talking to a lot of people in the open source world, especially community-driven open source, like the Linux kernel and the Apache Web Server, I hear there is a pretty strong correlation between your ability to request a feature and your ability to actually code it. You will get on a mailing list and say, “Hey, it’d be great if the Web server did this.” And the response is basically, “Yeah, that would be cool. When will you have the code ready?”

What you have talked about is very different from that, in terms of interviewing customers, interviewing influencers, and putting together a team. Talk a little bit about how a specific feature happened from end to end, how the feature was initially conceptualized, how you validated it, the portions of the product it touches, etc.

David: Well, I’ll talk about it in the meta form, and then I’ll talk about a particular feature. It goes back to the thing I was saying about technology evolution. And it’s actually one of the challenges.

One of the challenges the open source community faces, and this will be a sweeping statement that I get flamed for, is that they don’t yet generally appreciate the difference between the consumer and the producer. By that I mean, as a technology matures, the gap between who’s using and who’s producing it becomes greater. It’s a challenge we face at Microsoft; we are no longer simply building products for people like us, the engineers. In one of the talks I give to the product development community at Microsoft, I stress, “Look. We’re no longer just building products by engineers, for engineers. The PC is moving toward a consumer electronics device for many users and we must build a product that interacts with them on their terms, not ours.” I then show screen shots of Task Tray bubbles talking about “Virtual Memory” and “Pagefiles,” etc. My friends and family shouldn’t have to know what virtual memory is.

We’ve seen the same thing in the database space. Twenty years ago, every DBA had to be a wizard and needed to know how to fiddle with hundreds of knobs, figure out how to manage raw disks, and learn all sorts of low-level details just to create and maintain a database. When we were building SQL Server 7.0, I used to troll the database newsgroups an awful lot. You’d see people asking perfectly reasonable questions and you’d see responses like “RTFM” or “You don’t belong here if you don’t know that.” So we started this campaign to fix what I called the “you dummy” questions. Basically, if an expert responded to a question with a “you dummy” tone, we saw it as an opportunity to address the issue by engineering—teach the product to do it rather than teaching hundreds of thousands of DBAs. We turned the “you dummy” on ourselves and did something about it.

I don’t have to be an expert on my automobile to keep the thing running, and that’s fine. If Toyota wanted to come to me, I wouldn’t have to talk to them about ignition advance, dwell time, and those things anymore. I’d talk more about my experience and what I wanted the car to do for me. I think that’s just an evolution, and you can certainly see it in the database space.

To get back to the specific feature, I think the Resource Governor is a good one, where there was a complex set of mechanisms inside our system that we needed to go off and engineer, but how we constructed those and how we presented them to users was the challenge, right? What is the policy? How does someone describe a policy around constraining resources in a particular query or session? And then how does someone bind different policies to different classes of users or applications or queries or times? What degrees or what dimensions do they need?

Another way to think about this, from the perspective of design, is that the software development community has not made the leap yet from expressing the product control surface as knobs on the underlying mechanism to capturing the user’s intent or objective and then acting to achieve that. A simple example is: Rather than having an administrator configure how many dirty database pages will trigger a database checkpoint, we can capture the user’s intent in terms of the trade-off between fast recovery in the event of a failure, which can be achieved by increased checkpoint frequency, or better run time throughput, which can be achieved by fewer checkpoints. Having a control that captures how long the administrator is willing to wait for recovery to complete in the event of a restart looks directly at the business intent rather than fiddling at the level of the mechanism. I mean, how many dirty log blocks can I have on a particular database and still have it recover in 60 seconds, anyway? Instead of publishing some equation in the documentation, capture the user’s intent and treat that as a constraint during run time.

So to go back to the Resource Governor—we go to the customer and ask, “OK. What sorts of things do you want to constrain? What dimensions are most important? Are they applications? Are they time of day? Are they this or that?” And then gather all that, design something that’s consistent from the user intent perspective, and then figure out how we build a mechanism underneath to marry up with it.

Certainly, there’s a lot of success in the open source community, using open source technology in consumer electronics. But I don’t think that they’ve gone end to end in terms of doing a great job of user-centered design.

Sean: Well, that’s one of the things I’ve talked to people about, too, and that I personally feel pretty passionate about, innovation in the usability area.

I am curious, from a closed source company perspective, about what processes you feel should be put in place in order for a development team to excel in terms of usability. You go out and talk to people about potential features to help make sure that a feature doesn’t turn into engineers building it just for engineers, and that you actually end up with something—like the Ribbon in Office—that honestly has been a success, if only because, after release, nobody complained about it.

It’s the proverbial tree that fell in the forest and nobody heard it, but everybody was freaked out about it. And I keep telling people, “Well, think about this. Microsoft changed the UI on the one application people use the most, on a daily basis, and yet nobody really complained once Office launched.” There must have been something that really went into the thinking, in terms of usability.

For you guys, especially with a product like SQL Server, where I’m sure you can lose yourself in B-tree discussions until the end of time, how do you ensure usability in the product?

David: It’s a great question and a great challenge. It’s funny, but one of the things I’ve been doing for awhile that’s had a good effect is to hold the mirror up to the product development community at Microsoft, to get them to see that we’re no longer building products for people like us.

I have a set of slides that I’ve been using for six years or so, where I have collected some crazy error messages and I have real-life scenarios. I’ve got a screen shot where, in Windows XP, if it extends to your page file, you get this balloon coming out of the system tray that says, “Your virtual memory is low. We’re extending the page file. Blah blah blah.” I came home one night and my wife hit this and she said, “Hey, I think my computer’s broken. It says something here.” So I started describing to her virtual memory and page files, and she said, “Shut up! Just fix it.”

It dawned on me that perhaps 99 percent of the people using PCs these days are not engineers, yet we’re still throwing these exceptions and doing these sorts of things as if they were. And we ridicule the users instead of ourselves. I go help my neighbors fix their machines, and they all start with, “Oh, I feel like such an idiot around computers.” And I just want to say, “No, you’re not the idiot. We are, because we’re not building computers for you yet.”

And so I have a set of rules, or things that I’ve captured, that I talk about, such as the “principle of least astonishment,” where a reasonable user action should do something reasonable with least astonishment. The example I use is another story from my wife. I don’t know how many years ago, when I first got her a PC, someone emailed her some pictures. She said, “How do I look at these pictures?” I said, “Just double-click on the filename right here.”

So, of course, up comes the picture in the picture viewer. And there are little buttons down on the bottom of the viewer: “Next picture,” “Previous picture.” What do you think happens if you click on “Next picture”? You don’t get the next picture in the set of attachments. You get the next picture of whatever the next file was in your IE cache or whatever it happens to open.

And she’s going, “What the hell is this?” She gets some random icon from the temp directory or something like that. So I started talking to her about temporary directories, the IE cache, and blah blah blah, and again she’s like, “Shut up! Just tell me why it’s broken.”

Sean: She’s saying, “I’m looking at photos. I want the next photo.”

David: Exactly. And you see that even in the database space. We’ve taken the market from tens of thousands of servers out to millions or tens of millions of servers. You can’t have every one of them require a highly skilled DBA. We just couldn’t do it economically. And so you have to get out of the mind-frame of an engineer and go adopt a perspective of what the customer needs and how the customer wants to express their intent.

That’s the other sort of piece I talk about: In terms of the degrees of control and the dimensions of control, get rid of all of those dimensions of control that don’t capture any user intent and figure out which dimensions of control are orthogonal with respect to the customer’s intent, and then figure out how you build your system, and express things that way. That’s been the most effective thing I’ve found to motivate the change.

The bottom line is that the software development community doesn’t teach “design,” certainly not with a capital D. Most developers believe that if they think for 10 minutes before coding up some function, then that’s design. It’s not. This isn’t a development methodology thing, but rather a cultural change that we need to go through across the entire industry.

Sean: That’s great. The chief creator of a project called Quicksilver for the Mac was giving a talk at Google, and I just saw the video of it the other day. His theme for the project is “act without doing,” which has an interesting similarity to what you are talking about. If you saw the app, you would probably see some interesting things in it.

To follow along that line, I have a question about Oracle. Oracle, through the years, treated the product as though the more complex the cockpit, the better it was. If you walked in and it looked like an old 727 cockpit with 50,000 switches, that was nirvana. And you were paid to know every position of every switch, right?

David: Yep.

Sean: The open source community has, to some degree, a similar bent, right? Not in everything, but in a fair amount of places. But even in a product like Ubuntu, which is seen as the ultimate of end-user experience—and it has made some great strides—there’s a massive list of steps just to join that machine to a Windows domain, for example, if that is something you wanted to do.

David: Yep, yep.

Sean: So, do you see any similarities with this line of thought in the way Microsoft helps you to administer a database and the way open source helps you with administration in projects like MySQL?

David: That’s another good question. Yes, Oracle—I think I mentioned it earlier, they marketed our ease of use against us, frankly. They said, “How can this thing be a real enterprise database product? It doesn’t have nearly as many knobs as ours does.” And when we left Digital—this is an interesting little story—we had a product, RDB, that had roughly as many knobs as Oracle did at that point.

And like engineers, we were saying, “Oh, shoot! Hardly anyone knows how to turn the knobs on this thing.” So we wrote more software, something we called RDB Expert, which was a physical database design and some other things, to recommend and actually turn some of the knobs for you.

I think the first step that you go through in this evolution is to put a facade on it, whether it’s a system that recommends turning the knobs or a GUI to cover up knobs in config files. But really, at that point, you haven’t got a good end-to-end design, because you haven’t, in my opinion, eliminated those points of interaction or control. You haven’t captured user intent.

It’s like having a TV that still has horizontal and vertical control, but you’ve put something on the TV that can watch it and turn the knobs for you. So I think the next step in the evolution is to have the system software itself become more adaptive.

Scott: That’s true. TVs used to have horizontal and vertical knobs on the back. They don’t have those knobs anymore, even though TVs are much more complex, and no one misses them.

David: And what you wind up with—here’s the key point, and this is something I didn’t understand a priori, but after the fact it really made a lot of sense. In SQL Server 7, when we did some of the memory work to have the system automatically adapt to the environment and adjust the amount of memory it consumed, we changed the system so that you could adjust the amount of memory that it was using dynamically. And the mechanism was dynamic as well, so we had an automated policy that was a closed loop, but for those people who wanted to get under the covers, they could change it themselves.

But the neat thing about the way we reimplemented it top to bottom, and not just put a facade on it, was that the underlying mechanism itself was dynamic. By that I mean you could change the amount of memory that SQL Server used while it was running and not have to shut it down and restart it to get it to adopt a new value.

It’s back to the separation of mechanism and policy. We made the mechanism dynamic, we made the policy automated, and, where it made sense to try to capture user intent, we allowed user intent to be captured and actually influence the policy. And that takes architecture, top to bottom. I think the evolution is: You recognize it’s an issue, you put a facade on it, whether it’s a GUI or whether it’s some other code, and then you have to step back and architect the thing, top to bottom, to do it correctly.

Scott: One of the attitudes I ran into back in my C programming days was, “Look, programming was hard for me. It’s hard for a reason. It should be hard. If it isn’t hard, anybody could do it, and that’s the last thing in the world you’d want.” I run into that attitude whenever I look at systems administration and other technical issues, where people say, “Look, this is complicated. It should be hard. You should have to be a little bit of a rocket scientist, otherwise you have no business doing it.”

So one of the complaints is, “Well, yeah, Microsoft’s made it so that any idiot can set up a Web server, therefore every idiot does, and they don’t keep it patched, and it’s vulnerable.” Same thing with the database: “Sure, people could install the database and set it up, but they won’t create their indexes correctly, and they won’t keep it patched.”

So there are certain people who would say you should stop more people at the front door and not let them in. Because if you do let them in, they get farther down the path, thinking that they know what they’re doing, and then they run into a disaster. What would be your response to that kind of attitude?

David: I think it all comes down to, or goes back to, design in the sense that if you have the means within the technology to bring it out to a broader market and let people do the job, then I think it’s incumbent on you to try to figure out a way to do that. It’s a great example and a great point around physical design of the database—how do I design the index set? It’s very difficult.

One of the things we did here is, we designed the Database Tuning Advisor. This was work with Microsoft research where we’ll look at a workload and actually propose indexes based upon the workload, and do physical design that way.

And the way we test that is kind of interesting. We bring databases in, we measure their response against a real user workload. Then we strip all the indexes and we use the Tuning Advisor to rebuild the index set. More often than not, the Tuning Advisor does a better job than the guy who gets paid a lot of money to do it by hand. Of course we can’t see the reporting jobs that are running at the end of the month if they haven’t been part of the workload, but it does a pretty darn good job. So I think it’s a matter of how far can you take the technology.

Another sort of thought experiment here is if you go back to the TV front. Go back to 20 or 30 years ago. Every small town had two or three TV repairmen. The technology is more complex now than it was, but it’s perhaps 50 or 100 times more reliable. So you don’t see the TV repairman in every small town anymore. The TVs just go and go and go until you decide you want a slick new plasma one. I think that’s the way software is going to move as well.

Sean: Obviously, one of the strengths of open source is the community. I think that is one contest they tend to win hands-down, if for no other reason than that open-source projects were driving community long before closed source companies were. But at the same time, I know Microsoft has been making some significant efforts in this area too.

You mentioned in the previous conversation about CodePlex. I knew about CodePlex, but I have not really looked at it with an SQL Server focus before. A brief glance at it showed me some of the activity that is SQL Server related. Talk to me a little bit about what is happening there, how that activity is affecting the product team, etc.

David: I agree with you. I think I mentioned earlier that tapping into the energy of the community is super valuable. I think of a lot of these things in terms of closed loops and how quickly can you run the loop, how fast can you get the feedback, and how fast can you incorporate it. Tapping into the community is a super way to do that. There are lots of examples of closing a loop, like Watson, to have the machine send back error information and actually close the loop and automate things to enable continuous improvement.

As I think about community, absolutely I give the open source guys credit for tapping into that phenomenon first, but I don’t think it requires open source in terms of the development or distribution model to engage the community. I think about community engagement in terms of layers. With the product itself there’s a way of tapping into the community by harnessing their feedback to improve the product over time. I think with the documentation content and general knowledge around how to use the product, there’s a deeper way of tapping into the community. As we publish content for the product, can we allow people to modify it, link to it?

Some of our content could provide a spine for community activity. For example, we could build a collaborative site around the product error messages and allow the community to link their responses into each error message to help one another out when they hit a particular issue. Today much of this happens in forums and newsgroups, but it’s not done in a way that closes the loop effectively. If we did it in the way I just mentioned, it would be much easier for the product development team to review and mine the activity to improve the product going forward.

I think the next level beyond documentation would be extensions and tools. Things that you could build around the product where the core of the product is still in a closed source model but you could open up and distribute add-ons, add-ins, and other sorts of tools and actually build the community around that.

Sean: This has been a great conversation, David. Thanks for taking the time to chat.

David: Thank you.

Bookmark this:

Interviewee: Britten Martin

In this interview with Britten Martin we talked to him about:

• The role of support inside Microsoft.
• A walkthrough of how a support request is handled.
• The ways in which the outside community can interact with the internal support processes.
• How a product is supported differently in beta than when it’s finally released.
• How a bug eventually become a fix in the product.

Scott:  Britten if you could just introduce yourself for the record. That is usually a good place to start, and then I’ve got an initial question after that.

Britten:  Cool. So, my name is Britten Martin, I am a group manager within CSS which is Customer Service and Support. I’ve been with Microsoft since 1999 where I came in as a support engineer supporting Windows 2000. So I have been a support engineer, I have done some work with our outsource partners in a partner technical lead role, then I moved into frontline management and did that for about three years. In my current role I am a manager of managers, I have an organization of about 75 people and I am the global sponsor for System Center Support, which is your SMS and MOM products.

Scott:  So talk a little bit about what falls under the scope of support inside of Microsoft. I mean, engineering rather has its domain, marketing has its domain: what is support defined as and what are kind of the responsibilities roll up under that inside Microsoft?

Britten:  That is a good question. It really depends on which segment of support. It is a great clarifying question, Scott, because I am within Commercial Support which is the enterprise-based support, your back‑end, your servers, many of your big customers that buy dedicated support contracts, the government, many of the major companies around the world. We also have certainly have a consumer end which is your home user running Vista, Office, also Xbox, Zune. Any product that you can think of that is released by Microsoft has support. So really what it is, the thing that separates, I think, Microsoft from other companies in that, is that no matter what’s out there, you can pick up the phone, call 1‑800‑MICROSOFT, tell them what your product is, and you’ll get through to someone to support your issue. Although it certainly depends on the customer segment that you are in, who you are routed to, where you go and how much it costs.

Scott:  Sure.

Britten:  In Commerical Support, we support customers who have technical how-to questions about the products and certainly those who are getting error messages.† Also, if you need help setting it up, if you need more consultative type services, we do that as well.

Scott:  OK. Then one other thing I was somewhat curious about too is ‑‑ one thing I have always been curious about I guess – is how a problem works its way through the cycle.

Britten:  OK.

:  So, you start out when the phone’s ringing and you have one customer, two customers, and three customers: they seem to be having a similar issue. How does it work its way through the process from the technician helping them on the phone, to maybe later on deciding it needs a KB on it, to maybe a hot fix, to maybe being rolled up in a Service Pack?

Britten:  That is a good question too. OK. I will speak in today’s terms because major things have shifted over the last couple of years. There used to be a lot of process behind just what you were saying, which is, you know: "When do we do a KB? How many issues before we make a hotfix?" Nowadays with the advent of blogs, which is kind of, what you are doing, you know, what this interview is about, in support we have become a lot more proactive with getting immediate information out to our customers. I will give you an example from my group, within System Center: last week we had an issue with SCCM, the new version of SMS. Just like you described: one call, two calls, three calls. All of a sudden we are like, "Wait a minute, there’s something going on here!"

So we got some information together, dumped it out into our blog so it was out there. On the back‑end our support engineers, who are your front‑line resources, are working with the escalation team, our escalation engineers who do the debugging, who look at source code, who work with the product group. They are debugging the issue, and they get it down to, "OK, this is clearly a bug." At that point, that escalation engineer has a hotline per se to the product group, they pass along all of their research, they are reproducing the steps, reproduce the issue, send it on to them, all the documentation.

Then it is kind of in the product group’s hands to decide, "Is this something that we need to fix now. Do we need a hotfix, is it just something that we can put documentation out." You know kind of that form of decision making process. In this case it was a major issue, it was a deployment blocker, huge customer pain, so they released a hotfix. Then once again, depending on the urgency, is it a publically available hotfix or is it something that you have to call support to get a hold of? In this case, we knew this was a major issue so we made it available out on Microsoft.com, put a KB article out there, and linked the hotfix to it. Therefore, customers could immediately just go and download that hotfix.

Down the road certainly that’s going to be in the Service Pack 1 or similar, so you can pretty much figure for all of our major products, any bugs that we fix, or any hotfixes we release up until, there’s a cut‑off date, will make it into the Service Pack.

Scott:  And then talk to me a little bit about the timeframe, right, between the time where maybe the first few calls came in, until the time that hotfix and that KB article are out there.

Britten:  Well, once again it is very, very dependent upon the criticality and the widespread nature of it.

In this case from first call to KB/hotfix was about five days. In addition, some of those delays ‑‑ they had identified the code, they had written a hotfix within about two days, but there were some delays, you know, going through legal on official KB review, and getting everything up there and replicated. So from beginning to end, about five days.

Scott:  Yeah, that is fast. I do not think most people would ever think of Microsoft turning something around that quickly.

Britten:  Some times, I think Microsoft is seen as very slow to respond, but in reality, itís often not the case.† †If it is truly a deployment blocking issue or something that causes widespread customer pain you will see a quick turnaround on it and you will definitely see the documentation out there in a timely manner. I have been here eight years, I have seen issues where it was tough to figure out what the fix was, what the resolution was. Where it was something that would hide itself into multiple components so it was a complicated fix to write, it takes a little longer. However, if we can identify it, if we do a good job on the support side with identifying it, providing reproduction steps to the product group, you know, they can turn it around quickly.

Scott:  So one question then too, would be, what does the triage process look like if after reproducing the bug you quickly realize that it has security implications versus something that’s a deployment blocker, which might be the next layer and then further down which are just kind of usability or general problems in using the software?

Britten:  Yeah, I think, once again when you are running into security type issues what you get with that is even more resources thrown at the problem. We have security aliases internally, they have product group members, and they have certainly a high-level support presence as well, that if something comes out we can fire it up and quickly. There †is a whole process that is enabled. Let us say there is a new Sasser or Blaster or any of those. We have processes in place that will quickly call together members of support, product group. We have open lines 24 hours a day. It is kind of the war room type mentality.

It has product group members on it. It has support members. So people on the field can call in and say, "Look, I have got an issue with this bug or with this security hole and just want to make sure I have the up to date information". So they can call into this grid, they can get the information.

On the back‑end, it is hard to say how long it is going to take, depending on the fix but you are going to have even more resources available. You will see a fix as quickly as we can get it out there. After the Sasser/Blaster/Nimda pain, we really turned the corner not only on product development but also on the support side to ensure we could have a quick turnaround.

:  What do you think the level of community involvement is in the support process because obviously that is a huge piece of the open source model. Right down from many eyes looking forsecurity bugs to a very well populated mailing list where there is a high discussion level, traffic, and things like that. However, at the same time with the closed source support process while the team is staffed potentially adequately with full time employees, there are general concerns out there with a closed source model in terms of how much the community can get involved.

So what are the avenues the community can actually affect the support
process at Microsoft and/or see what is happening under the covers of the support process?

Britten:  Yeah, I will tell you that it has really been something we have been working on for the last few years. The product groups and the support teams have been trying to find ways to tap into that community model because there are many folks out there that do our support day in and day out with customers that have a lot of knowledge that could be helping the broader community.

So, one of the things that we have is the MVP program, the Microsoft Most Valuable Professional. One of the things that these folks do is to actively take part in the public newsgroups for our products. I can point to a couple of products ‑ Small Business Server is the poster child for that. It has a very strong community. Customers can go up to the Small Business Server forums and post problems and the MVPs will quickly get involved to help resolve the issue.

There is a lot of energy around building a stronger MVP community and empowering the MVPs to help solve customer issues. Many of the product groups have CPE, Customer Partner Experience, teams, that are really focused on tapping into that partner community.

Another example from my world System Center Essentials, or SCE. We actually launched support for that on a complete forum based model, where we had engineers from CSS monitoring the forum, and we worked to build strong partner participation. If the partner could not answer the question, or did not answer the question within 24 hours, someone from support would jump in. We are trying to grow more community presence around support.

Globally, there is more partner presence out there as well. A good example is SoftGrid or Softriciy, a recent acquisition. In Europe, before Microsoft took over, there was not really a ton of dedicated support. It was all done through the community. Therefore, we have been looking and trying to learn from them on how they involve the partners, and how we can keep that type of interaction going. I know there is a log of energy around this and it is a
huge initiative.

Sean:  Well, one question which is really back to the second part of the original question which, is that a lot of the product groups in Microsoft are being affected by the open source model.† For example recently we talked to Shawn Burke who has been instrumental in getting the source code† for the framework out the door. How much of push to learn from the open source community is affecting the support team in general? For example could you see a day where the status on bug fixes is more open and less a black box until the actual fix arrives out on the web?

Britten:  So thinking more about ‑‑ I am partner, I am out there. I find a bug or a hole, I put it out there, and really being more transparent if that is going to be fixed. Is that kind of…?

Sean:  Yeah, exactly. I realize it’s not going to be a plate glass window ‑ it never is. But at the same time I think a lot of people will look at support right now and say, once I send in my support request I don’t really have a lot visibility of how they are stacking it, ranking it, prioritizing it, and where it goes, until maybe I get an email that says you have been deferred. Right?

So I am just curious, I know a lot of the product groups specifically are trying to expose a fair amount of their development processes, so I was curious if maybe that was making its way over to the support side.

Britten:  You know it is a fair point. Let’s say you are running a beta version of the product and you are involved in the one of the official beta programs, such as TAP ñ our Technology Adoption Program.† †The beta version is fully supported and you can actually go and directly file a bug. In addition, I know that they try to make a lot of that process transparent, so you know if the bug is a duplicate. Is it being fixed? Is it being deferred to the Service Pack 1…whatever. So actually, I know on some levels we do that.

On the support side, I think we make good use of our blog. There is a huge blogging effort, because there is a lot of knowledge within our support engineers and within our internal databases. For one reason or another that has never been made public. So I think we try to do some of that as well. But I can see the point. I mean maybe someday and maybe that some day is not too far away where you can log in and say, here is a bug that I submitted. Alternatively, even better for Windows, here is the bin of things that are going to be fixed in the next two weeks, months, whatever.

Sean:  Exactly

Scott:  Well, and so you know not every support request anymore comes in through a phone call, right? There are things like the Connect site. There are things like Windows Error Reporting services. I guess if you would not mind talk for a minute about some of the different channels that, a problem can kind of come in through and… Go ahead.

Britten:  No, no, it is fine. So today, and if I use Americas Commerical Support as a kind of a measuring stick, somewhere around 85% of our support issues come in via the phone ‑ very traditional means of operation. We are really trying to push more of our online type offerings. One is online submission where you can go up and create your issue; you can attach data, submit it to a support engineer. That support engineer reviews that data and calls you back, hopefully with an answer.

The Connect site is a lot more beta focused at this point. I am not an expert there, but that’s where you get into many of the cool things where you can actually submit bugs and all that. We also have the managed forums, which we talked a little bit about. The managed forums are a completely different avenue. Most of those are staffed by Microsoft support engineers as well as product group representation and the partner community.

So you have an issue with Windows, you can go up to a specific Windows forum, post a question. You might get an answer fromsomeone in the product group, a partner, or a support engineer. Therefore, you have those as well.

On the consumer side, they do really cool things with automated responses. Therefore, you submit an issue and it checks your data, checks your error log to see if it is a known issue. Before you would hear back from the support engineer, you might get an automated response with, you know, try these two things. If this does not work respond back and the engineer will contact you, then as you said, you had the whole Watson or error reporting. The error reporting stuff is very cool, because regardless of bugs filed or anything else that goes on, all the product groups take all of their Watson or their error reporting data and they make sure that as they look to roll out service packs, with new versions of products especially, they want to handle those top 15, top 20, top 50 Watson issues. They want to make sure that they are knocking those off.

Therefore, it is an interesting mix of many different avenues for customers to submit data, and in some cases, a Watson log, the error reporting, customers might not always think they are really contributing to the next version of the product, when in essence, they truly are.

:   I realize, too, that you are right, that is largely a mechanism for when a product is in Beta. I guess, I have two questions, but one is a follow up on this thread initially. Talk a little bit, I guess; about how a product is supported differently while it is in beta, versus once, it has been released?

Britten:  That is a good question. We have a few different types of beta opportunities for customers.† There are more formal relationships, such as TAP (Technology Adoption Program) and RDP (Rapid Deployment Program).† With these formal programs, customers agree to deploy this product into some type of production environment. In return, they get access to submit bugs, they get access to product group members, and they get
access to dedicated support engineers.

From within support, let us say we have a new version of Windows, we might have three or four support engineers in the US dedicated to this beta support. These engineers get a hold of the product early, they get access to product specs, they visit the product group, and they may even get to go out on site to customers who are deploying this product so they get some real world hands on experience.

Now, that is for the structured beta program. If you’re not a customer in one of these programs, you likely can still download beta copies of the software. Even in these situations, there is support out there, through the forums that we talked about before. Once again, you probably get answers from those same beta engineers and some of the product group folks. You just do not have that one-to-one interaction in a lot more, you know.

Does that make sense?

Scott:  Yeah. Yes, very much so. One of the other things that ‑ I will kind of circle back to something you said earlier – If it needs code to solve the problem, you know, if it is not just configuration file that was messed up or any one of the myriad of things that could cause a problem. You know, some registry key got corrupted or whatever. However, it actually needs the coding hot fix, you could hand it over to the engineering, send it over to the product team and they triage it, compared to other stuff.

I mean, have you ever run into situations it seems like this would inevitably happen where they’ve got their engineers dedicated to building the next version of the product, and meanwhile, there’s an endless list of bugs in any complex software that they could be tasking engineering resources with. In addition, it seems that they kind of have to make that call about pulling people off engineering new features to work on a hotfix or something like that.

Britten:  Yeah.

Scott:  And how does that process of triage work? Because support is kind of on the front line with the customers, it seems like sometimes you guys might have to go back to them and say, "Um, you know the time frame that you’re looking at fixing this in probably isn’t really going to work, so think about it again."

Britten:  Right. In addition, I will tell you, it is really dependent upon the product group. You look at something like Windows. Windows has a whole team, and you hear it from time to time called Win SE. ,Windows Sustained Engineering. They have a whole team dedicated to current version, and whatever supported versions are out there, on the hotfix front.

With some of the smaller products, you may run into situations where they have to pull developer resources off the next version to fix current version or legacy version problems. However, I tell you, in my time here, I have never seen that as being a blocker in getting something fixed. If there is truly an issue that needs to be resolved in the current version of the product, especially if its security related, you can be assured that they will find someone to handle it.

However, as a general rule of thumb, all big product groups have sustained engineering, folks that are dedicated to current version problems, or legacy version, so it is not necessarily that huge resource problem that you would think.

Scott:  OK. Therefore, many times, they are not pulling people off features. They have people who are dedicated just to the maintenance and hot fixes.

Britten:  Right. That is your sustained engineering folks, for the most part. As I said, I think you would run into some smaller product that might not be in that same boat, but the major ones have sustained engineering.

Scott:  It seems, too, like support, it is a little bit of a difficult situation, because, I guarantee, if I call with any bug, it is a critical show‑stopper. The world will end if it does not get fixed. So how do you kind of manage customers, who are generally going to feel the importance of their bug is high, but Microsoft does not necessarily determine it so in the grand scheme of things?

Britten:  Yeah. I tell you, our people are experts at conflict management and really setting customer expectations, and certainly the empathy piece as well. I mean, you have to be empathetic with the situation, but you also need to be very transparent and upfront.. It is saying, "Look, here’s where the bug is at this point. It is something that may or may not get fixed. Here’s the timeline."

With many of our customers in the Commercial space, these customers also have account managers that are Microsoft employees that really serve as the liaison between support and the customer. Therefore, we do a lot of work with the account managers to help us on the customer maintenance side of things, when maybe the messaging isn’t as good as we’d like it to be.

However, frankly, all major problems end up getting fixed. They might not get fixed in the period the customer would like to see it, but, between a hotfix, the next service pack, and the next version of the product, all issues do end up getting resolved.

Once again, it is really just dependent upon how many customers are having the same problem. Is it something that other customers could run into? In addition, really prioritizing, like you said before. I mean, the product group, while they have a sustained engineering team, has a limited number of resources, and they need to prioritize what are the big-ticket items that need to be worked on.

However, that skill of conflict management is one of the key pieces for successful support engineers, because, every day, when we answer the phone, when we respond to an email, that is generally that networking administratorís worst day of the year. The servers are down. They are catching heat from their management.† Something is going on, they are being yelled at. Our engineers deal with this on a day in and day out basis and as a result, they get very good at handling the heat.

:  So one question I am curious about is, how do you deal with it when the support engineer knows that it is really a usability issue and not a bug, sure there is a sequence of phone calls and bug reports, but it is pretty clear that this isn’t a bug, it is more of an issue of usability. And it might be in the land of the undiscovered feature right? We were told, if I remember, some person in the office team told me that something like 90 percent of the feature requests for office, are already in the product, but nobody knows the feature is in there. So how do you people educate the product teams and what does that flow look like? How do you finally classify that, "OK this isn’t really a bug, it is an issue of usability?"

Britten:  That is a good question.

Sean:  Because obviously the product team, like everybody else is blind to what they build, right? Of course, it is useable. Because I built it, I mean every developer has fallen to that problem and we all have seen some pretty horrible UIs over the years. So how does the support team educate the product teams on this point?

Britten:  Yeah. In the past, if you back up five years, 10 years, support was as you said an island. We were out here, taking all the heat and there was limited interaction with the product group. About five years ago, things drastically changed.† we started doing what is called supportability in our world. We review what are the top issues, how long does it take us to solve those issues and how much do they cost the company for us to support these issues?

Therefore, we came up with a list of the top 10 issues. We then present that back to the product group. Now the interesting thing about that is the product group actually funds us to do support for their product. So, when they see that a problem in their coding is †costing them $800,000. they to take some notice , and say wow OK, what does right look like, what are we missing here?

Sean:  Well, just a quick follow up to that. What are some examples of things where you feel like you touched the product, and you made a real delta that the customers ended up bettering from, but at the same time you folks probably would never personally got credit for it

Britten:  That is right. I donít have a list in front of me but DNS is one that consistently scored high for Windows 2000 and even some into Windows Server 2003.In addition, there was so much push and so much interaction that there were some major changes that happened later in 2003 and into 2008 that have drastically changed the way DNS works.

Another one I can give you that clearly had all the supportability people driving this was terminal server licensing. Terminal server licensing, when you go back to 2000 and once again, kind of into 2003 with a complete and utter nightmare for something that should be very simple. Consistently that silly process ended up top two, top three on our supportability list for Windows Server. And some very small tweaks based on advice provided by support completely changed this, dropped out of the top 10, out of the top 20 in support issues and saved the company a lot of money.

Those are two examples of technologies I know we beat on. However, on all the products out there, SQL, Exchange, Windows, SMS, MOM, they had the supportability groups that review the top 10 issues. In fact, I was just in Redmond last week, looking for the SMS side and they do cool things with it. I mean they look at top 10 issues for previous version of the product, look at the top 10 issues for current version, and review whether or not they fixed the top 10 call generateors? If not, what do we need to do to make sure it is in the next version that we do? Support is a huge data mine that we are just starting to realize the power of all the data that we have.† We are a great voice of the customer.

Sean:  One quick follow up to that and then Iíll turn it back to Scott. I am curious because you mentioned that five years ago there was some type of a change, right? In addition, all of a sudden, you folks were a lot more integrated. What drove that? In addition, I realize with all changes ongoing, there is probably stuff you can say and stuff you cannot, but I am sure somebody reading the interview is going to be like, "Wait, wait, and ask him that." So what drove it at a high level?

Britten:  I think that is fair. I wish I had some great and gory backroom drama to share, but if it happened, I just donít know the details.† In my opinion, you know, part of it was just the fact that, wow, especially on the Enterprise side, Windows Server 2000 started to really turn the corner and become a huge enterprise related product. People were out there buying it and implementing it and someone on the CSS side as well as, I can guarantee you, the Windows product group was like "We need to tap into this.î

Around this time, I was a support engineer and I suddenly realized that the product group was really listening.† In all businesses you started hearing about some monthly conference calls where we were there to provide feedback on top issues.

It was a time when we were moving from being just a consumer, home user company to being a legitimate player in the enterprise. In addition, there was a lot of pressure at that point to make sure these products were just better, and I think someone realized "Wow, CSS has a lot of knowledge out there; we need to tap into it."

Scott:  Well, how much now do you guys get involved in product development from the standpoint of, you guys are looking at the product, while it is in development and saying, "You know what, if that error happens, there is no way we will able to trace it back and figure that out. We need instrumentation here, we are going to need these things, spit it out to a log file, we are going to need these kind of support features built into the product to be able to kind of quickly
and officially troubleshoot things."

Britten:  That is a huge piece of our beta involvement. I know we talked about it before. For each beta, we dedicate engineers from our support teams to the pre-released product. Well, one of the way they make an impact is by providing that ongoing feedback to the product group. They are not only working with beta customers, they are not only answering newsgroups, but they are also messing with the product itself.

Therefore, they filed their own bugs. They have supportability war room meetings where they say, look there needs to be some logging here, or this error message does not make sense, or if this component is blowing up, I need something to trace it back. That is generally our involvement.

We do spec review. So many of our engineers have been on betas before, they are hooked in with the product group contacts. As the next version of the product is being planned, our engineers get involved to review specs, and just generally throughout the beta process from the beginning of our involvement through RTM, we are providing that ongoing supportability feedback.

So not only are we always supporting our customers, but we also look to help the product group on the supportability side. It is like you said, I mean, they are focused on here is a new component, here is a new thing that customers can use, maybe not always focused on how would I support that if it breaks?† Letís be honest Ö if we donít provide our feedback to the product group on what we need, it just causes more pain for us once the product is released.

Scott:  One other one question is, so a new product is coming out and how do you get support folks up to speed on it? In the OpenSource model the development team in some cases is the support team and there is a tight integration between that team and the open source community so there isn’t really a need to educate the "support team" in this case. So how do the support engineers come up to speed on a given product?

Britten:  This is another benefit of our involvement on the beta side.† †We get access to the source code. We get engineers playing with the product. We get that constant interaction with new builds of the product as it comes out. As a result, months, in some cases years, before the product actually ships, we have experts on this product.

Now, our experts certainly might be great technical writers, but we have a group called Global Technical Readiness. These folks are really program project managers who will say, "All right, we need Vista training for our support engineers." Therefore, they get their list of top issues. They get a list of new components, features, all of that. In addition, they start mining for data using these beta resources to come up with, "Here’s what engineers need to know. Here are the troubleshooting pieces; here are the top issues we have seen with our beta customers. Make sure engineers know this and that."

Therefore, over the course of this beta cycle, we are also in the background writing our own training. So these engineers write the training and as RTM approaches, depending on the training strategy, they might train all their engineers or a subset of the engineers. However, the knowledge is based off spec; it is based off beta engineer experiences, and beta engineer customer interaction.

In addition, the product group, in recent years, has really stepped up to help with training. They will review our training. They will provide insights. They will attend beta deliveries of the training and help the trainer get comfortable with the material and the product.

In some cases, the product group will also do a session of live meetings on, "Hey, here’s how we intended you to support the product. Here are the supportability features that we thought would be helpful." Therefore, you get a lot of interaction with the product group as well. From the beginning of the product to the end, not only are we helping customers on the beta side, but also our selfish win is to get the training that we need.

Scott:  So, let me ask the flip side of that, right? So, you mentioned Configuration Manager. You’ve got System Center Configuration Manager 2007 coming out. Everybody wants to learn it…

Britten:  Sure.

Scott:  …because everybody is motivated. Have do you rather incent people to keep wanting to support the previous version when there is something new, shiny, and interesting.

Britten:  At least on the Commercial stage, I do not know that I have ever really run into that problem.

Scott:  OK.

Britten:  Just simply because, in the enterprise customer segment, what you generally see is not the consumer model where a product releases, and everybody goes and gets the new version. However, it is a gradual uptake, usually. So, say Configuration Manager. Right now, and I can give you the specifics on this one, it is 10% of our total SMS volume. Therefore, engineers get a couple of SCCM cases, but they get the majority of SMS 2003.

In our minds, that is beautiful. Because what happens is, slowly over time they can build their SCCM knowledge while still providing that long held knowledge on SMS. They can help those customers resolve their issues. Therefore, it is a slow mix that generally happens. I do not know, it is ‑‑ they are tech junkies. They love the new stuff, but I guess maybe we all know that we have to pay our dues with the old stuff as well. [laughing] I do not know.

Scott:  OK. That makes sense.

Britten:  That’s a good question but it never really happens quite that way on the Enterprise side of support.

Sean:  What it sounds like for you folks, is that from an enterprise standpoint, you can point to pretty clear statistics on customers who are significant accounts. Alternatively, you could point to customers who, for lack of a better phrase, probably spend a fair amount of revenue, right. In addition, it is probably easy to also look at the segment of the market and say, "We’re not going to give up 15% of SMS just because SMS, the latest version is out" ‑‑or in this case, obviously the rebranded version.

Britten:  Right.

Sean:  Because you folks can point to real metrics in terms of your percentage of the number of people who are putting in support requests and things like that.

Britten:  Right.

Sean: Thanks for taking the time to talk with us Britten.

Britten: No problem.

Bookmark this:

Interviewees:Shawn Burke.

In this interview with Shawn Burke of Microsoft we asked him about:

• Shawn’s motivation for helping to make the framework source code available.
• Whether there are any limits on who gets access to the source code.
• Whether there was any internal resistance initially to providing the framework source.
• Why Microsoft doesn’t just provide a zip for the source.
• Whether Microsoft’s individual developers write code differently knowing that their actual source might be available for the world to see someday.
• Whether or not Microsoft is planning on making available other development artifacts such as test scripts, threat models, etc.
• Whether a closed source company or an open source company is better positioned to support external developers who need access to the source for a given project.

Scott: All right. So if you take a second and just briefly introduce yourself, that is probably a good place to start. Then we can drill into questions specifically around the source for the.NET framework 3.5.

Shawn: OK. I am Shawn Burke; I am a director here at the.NET development platform at Microsoft. I run a group that we internally call the agility team. We run small high value projects to try to help deliver value to customers in agile ways. Not necessarily using agile technology, but just being agile about how we deliver.

Scott: :So in other words, it sounds like you are looking for sort of low hanging fruit where you can do things that are a little bit outside of the norm but would not necessarily be an enormous effort, but would deliver a lot of value to the customers quickly. Am I characterizing it correctly?

Shawn: That is right. Our team works a little bit outside the bounds of the typical projects and we are not constrained by the same rules they are. We are able to do things that are more customer focused and a little less process focused way.

Scott: Just as a point of clarification, people have said Microsoft is “open sourcing the.NET framework.”That is not what your Microsofts intention is and not really what you are doing. You are making your source code available for reference. You are not really open sourcing it the way that, you know…

Shawn: Nope, some people in the community used characterization, that is definitely not what were doing.Were making the source available as a tool that helps people be more successful with the platform in its current state.

With this effort, my goal has been from day one, a simple scenario, which is you are debugging against an API, you are getting an exception from the API, and you cannot figure out why you are getting it. It is often very difficult to figure that out just from documentation. However, your ability to step into the API often makes it really clear what you are doing wrong.

Scott: Right. In the example that Scott Guthrie put up on his site, this is something I’ve run into,? At the point where you DataBind something, all kinds of stuff happens like down in the plumbing.

Shawn: Yes.

Scott: And the exception that you get, you know, you do not really have a lot of visibility and sort of what you did wrong in your code that caused the DataBinding exception. And so that’s like…

Shawn: Yes.

Scott: That’s like the key scenario that you went after.

Shawn: Yes. That is like the thing that you are unableto do currently you know, because I work here I have the benefit of being able to do that a lot. Quite frequently, the amount of time it saved me , “Oh, Ill just grab the symbols” and you do and start to debug with them and immediately you’re like “Oh! Because I forgot to do this or that.” It becomes so clear that I really wanted to enable that sort of service for customers because its such a powerful thing

Scott: Yes. And just to back up another thing that you said, too. I never read it. I was not indicating that Microsoft was somehow trying to create the impression that they open sourcing this. I know when I looked at it, it was clear to me that Microsoft was releasing the source for reference purposes, you were not releasing it so that people could…

Shawn: Yes. I get it. I did not mean to sound defensive there!

Scott: [laughs]
I’m sure it comes up a lot.

Shawn: Yes. That was the first thing we saw on a large website that is often not terribly pro Microsoft

Scott: [laughs]

Shawn: Fortunately many people came on there and said, “No, no, no.” They never said that, that is not what they are doing. I think most people get that now.

Scott: So what you are enabling is a sort of step into debugging experience where normally, as soon as you call into the API, you could not continue to drill down and step in. Now you can.

One of the things I was not clear on is who gets this? Is this available to all versions of Visual Studio and all customers or is there a sort a subset? Who is this available to I guess?

Shawn: If you are running on Windows, it is available to anyone who can use it or to look at it.

As far as the Visual Studio integration, you need to have VS Standard or above. You cannot do it with Express versions. You know the express versions of Visual Studio?

Scott: Sure.

Shawn: Express does not get it for some kind of fundamental reasons.The main reason there is it just simply because Express does not expose the UI that you need to set this up or have some of the other facilities internally to take advantage of it

Scott: Sure. Sure.

Scott: Yes. The only reason I ask is that some of the people in the Microsoft ecosystem have made their source available kind of under premium plans with their customers. However, that is not the approach that Microsoft is taking with this. This is not sort of a premium thing, this is just generally available.

Shawn: Yes, that is Code Center Premium (CCP), this works similar to that CCP , but there you need to have an account and a Smart Card actually. And you have to go through much more security than in this system. This is a public reference release of the code, so there is really no credentials required. There is none of that.

Scott: So when you sort of floated this idea internally, it sounds like it was generally well received. People kind of jumped on board and wanted to help make it happen. Just behind the scenes, what were some of the things that it took to enable this? It is more than just dropping a zip of the source out there to make this happen.

Shawn: Yeah. Well, yeah. It is good that you brought that up. This whole thing started quite a few years ago. When I was on the Windows Forms team, which I was for seven years, I always felt like we should be releasing the source for many reasons. Others had done it. I felt like it was good for customers. When I was the Development Manager on Windows Forms in early 2005, I did a blog post about this, saying I wanted to make this happen and talking about some of the challenges to doing so. It was kind of justI was just kind of brainstorming aloud about different ways and what some of the challenges were. The response was enormous.Hundreds of comments.

A funny story, I did that blog post, and then I went down to Cabo San Lucas with my girlfriend a couple of days later. When I turned my phone on halfway through the trip, just to see if it would work in Cabo, I had a flood of text messages that said, “Please call work.”

Sean Campbell: [laughter] All right. So you learned a lesson of do not blog anything but “I’m going to Cabo” on the blog before you go to Cabo, basically.

Shawn: Yeah. I think if you would really pull it down to basics, that is it. Well, actually, I had always done tons of blog posts that no one’s…

Scott: Meanwhile, you created this firestorm of all these people running around going, “What have you done?”

Shawn: Yeah, exactly. I like to joke that it created, basically, an international incident because it got picked up by eWEEK, Mary Jo Foley, CNET, and all the rest. Quite a few people picked it up. It generated a lot of interest. That was a good impetus for me that there was a lot of excitement in the community about this.

If you fast forward a couple of years, I did some more ground work on this thing in 2005 and 2006, got busy with new role I’m in now, and dropped it for six or eight months, and then picked it back upI think it was late last year, late 2006and started looking at how we would get this done.And Ive been shepherding it along since then.Getting this out there really is the realization of a dream that I have been chipping away at for quite some time.I really hope Developers find it as useful as I think they will!

You guys mentioned pushing it out as a zip. One of the real problems was that I did not want to have a situation where every team that did this put a zip file or MSI out every four months, so the customer had to go download, install, and set up his little studio to use. What if you have a hotfix or different version? You know what I mean. I did not want to have a situation that was going to be cumbersome for us to ship and for customers to consume.That would kind of defeat the purpose.

What really got me rolling on this was I started talking with the guy who developed this technology we use internally. Internally, you can get the source and demos for almost every product we build here at Microsoft off this thing called “Source Server.” Source Server has versioning built in, making sure things do not collide so you can have side by side versioning, and it all works very smoothly. The server side piece for source server is just a web server. There is no magic there so its very simply to deploy. There really was not any barrier to making to using that technology externally.

So what we able to develop was a system where teams could publish source very, very efficiently. After a first source publish, they can continue to do source publishing at near zero cost to them, which is great. On the customer side, you have a scenario where a customer has a few simple set up steps. Then they kind of auto magically get source and symbols for what we have chosen to make available at that time for free. No searching around on MSDN or having to deal with where to put the stuff on the machine.

Scott: It sounds like, too, one of things that you are saying this accomplishes is it sort of knows exactly what the client is running and building against. If they have installed the service pack or have not installed the service pack, they get the right source. They get source that matches up with their environment.

Shawn: Yes, if it is on the servers they can get it. One of the decisions I made in getting this done was that, the cheater thing I did was I said, “You know what, I ‘m not going to define policy on this. I’m going to get a system developed where individual product groups and teams can make decisions about how often they push source.” Meaning, does it make sense to push source for every new release we do? Maybe, I dont know. Does it make sense to push it for every service pack? Maybe, I do not know. I think those might be different for different teams. I think we are going to do learning in working with the community about what really makes sense there. However, the system is set up to handle it, in any case.

Scott: Like you said, generally people were receptive to this. Were there internal objections that had to be overcome? Where there people who had concerns that, “Hey, this is crazy. Why would we do this?”

Shawn: Well, if you go back into 2005, I think I had many senior people that I knew, vice president level people that were generally supportive of this, but very cautiously supportive. For me a little bit, one of the reasons it’s been a multi year effort is I had to push carefully and let some perspective change as the industry has changed and the environment changed over the last couple of years.

It has gotten significantly easier. Something happened, I think, in the greater consciousness of the industry over the last 12 months or so. A lot ofI do not want to say a lotresistance is too strong of a word. A lot of the real cautiousness and kind of by-default-no talking-into-yes stuff dissipated and switched to actually a lot of enthusiasm. We need to be careful about protecting the IP that we spent billions of dollars developing. However, I think people really see the value of this now. It has been great for us and for our customers for a subset of products. Now I have seen teams are signing up to push their code out really quite enthusiastically.

Scott: With the.NET framework, it is in a little bit of a unique position because, on one hand, Microsoft has invested a lot in it, and generated a lot of IP. However, on the other hand, it was never really a secret what was in the framework anyways. You could use reflector and other tools to disassemble what was in the framework and sort of figure out how it was doing what it was doing. However, it did not enable this “step into it while you are debugging” experience. It did not make it super easy to learn from the framework. This might have been a little unique in that the IP was not all that protected to begin with.

Shawn: Well, yeah. Yeah, it is funny. The case that I built for this, with managed code in particular, definitely had that aspect to it, that our level of secrecy around how this stuff works, as well. One of the problems with reflector, using reflector, the EULA does not really allow for it.

Scott: Ah, interesting.

Shawn: The EULA does say that you are not supposed to reverse engineer. That said, I felt like it was a good idea to give customers a way to do this within the license, which this is.

Scott: Hah! Yes, I guess I never really realized that that might be in the EULA. I think, especially among the more advanced developers and in the blogs, and in all that kind of stuff, it is such a common practice.

So, now that you have done this, you have taken this first step and you’re going to make source code available for certain subsystems of the framework, and that’s on a feature by feature basis, right?

So, Windows Forms will decide when they push their source code in, ASP.NET will decide when they push theirs in, am I understanding that right?

Shawn: Well, we have pretty much what you would consider, if you go back to.NET 2.0 and think about what made up the Framework at that point. We’ve got pretty much all of the big pieces there. We’ve got BCL, Windows Forms, ASP.NET, Data and XML.

So those guys are already taken care of. Therefore, we’re going out with essentially what is the original.NET framework. The other stuff will fall in. It is really a matter of me getting with a given team and through the process. The code preparation process is actually fairly labor intensive. It takes time to schedule that and manage it.

I had teams that were able to do that. A lot of teams have been obviously really busy getting VS 2008 and.NET framework 3.5 finished up. We will continue to push out more code as time goes on. We wanted to get the big ones first.

We also managed to get WPF in there as well, so its not just the older stuff.Lots more is in the pipe to come online as time goes on.

The choice was either we wait a couple of months to do this, and make everybody wait, or we get the big ones out there fast and then we push in the other stuff as we have time, and as it comes online.

Scott: So one of the things that is kind of interesting is, you’ve got these individual developers in Microsoft and they’re writing code. They have worked in Microsoft for a long time and they have a feeling that, you know, what happens in Microsoft stays in Microsoft, I guess, for lack of a better term.

The whole world is not going to see the line of code that they are typing right now. Now, with the source being released, how do you think that is affecting, if at all, how individual developers are doing their job? How product teams are looking at how they build features going forward, knowing that the source for it is going to be out there.

Shawn: Even product teams that are not the framework team, they are thinking, that’s great, they did that over there, so does this mean that someday my source is actually going to be out there? Yes. Yes. They will be thinking that. And they should be thinking that.

In fact, one of my to do list items is to do some broader communications across our Division about that exact point Software developers are pretty much the same around the world. Software developers, in general, always try to do the right thing. They are also always, in general, under a certain amount of time pressure. Anything that is going to encourage them to take a little extra time to do something right, in the kind of capital ‘R’ right, way, is a good thing.

Whether that’s testing, whether that’s static analysis like FxCop, or whether that’s security reviews or code reviews, or whether that’s the code is going to be visible to their peers or the public, I think those are all good incentives for software developers to think extra hard about how they do things.

So, you know, honestly, we went through the code, we’ve done these code reviews, and these code scrubs and we found a lot less stuff than we thought we would.Actually, we found almost nothing of any real concern, which was nice.

Sean: Are there other things that Microsoft is thinking about exposing around the development process?

There are elements like threat modeling, specs, test scripts, all kinds of things that Microsoft might be thinking about exposing. Are there any thoughts or plans along those ways about how far this maybe goes, eventually?

Shawn: That is a really good question. No is the answer. [laughter]

Sean: You’ve accomplished enough by ruining one Cabo vacation. Right? You can put that post up later, you know, on your next vacation.

Shawn: I think that stuff…that is a good point. If you start doing, a curve on the cost for us to do that versus what percentage of customers that level of access is going to be valuable for. For the threat models, and even for the specs, I think that is a harder case to make. I think those things are also harder to vet, or release, than the source is. They are also less useful to a broad set of customers.

Sean: Well, one of the things that crossed my mind was test scripts, and test routines that Microsoft creates because that would be helpful to others just from an educational standpoint. That does not really expose Microsoft to too much more overhead. I am not trying to pitch the idea here, But go ahead, Scott, what were you going to ask?

Scott: No, same thing. Just, Microsoft has given an inch so watch us try to take a mile.

Shawn: Oh, yeah, we know it. Absolutely.

Scott: A lot of times that’s what people want. They say, OK, there is this API. I want an example for how would use it. I would go look in the help. Well, that example is not what I really want. Microsoft I would assume writes really comprehensive test suites.

Shawn: I do not know if the test code is what you want, either. We call it auto PMEs. The PME stands for Property Method and Event. It is super granular. A lot of that code does not relate to any real scenario. It is just wide unit testing code. There are other types of tests as well. A lot of the stuff is that. The challenge of the testing stuff.

I actually think that the testing stuff is a good point, and an interesting thing and I have been thinking about, not really in terms of us releasing test stuff for previous products. I will get into one in a second. The question is going forward, is there a way for customers to leverage the kind of stuff we do in tests more broadly?

However, the reason, going backward, most of our test stuff is written using internal test harnesses and internal tools. There is this huge slippery slope there. You need this and you need that, and it would be really a lot of work.

Scott: Right, right, so they would have this test but they would not be able to hit F5 on it and actually watch it run anyways.

Shawn: Yeah, and the tests probably call into a bunch of other libraries that are internal that we would not ship externally anyway. The tests themselves might not make any sense. Most of the tests are stored in this big system called Mad Dog, which is kind of a giant database of test cases and test code, and I do not even know how you go about extracting code from that thing.

[laughter]

Sean: So Shawn, what do you think aboutthis is just something I was noodling on, trying to think of the right way to box it in given the time in terms of a questionso, Microsoft’s a closed source company, right. By definition if not necessarily for the sake of our interview. What do you think about closed source companies perhaps at times being better able to support the exposure of the source than maybe a company or a project that simply says ‘Well, just go check it out from subversion, from xyz open source project?’

Shawn: Right.

Sean: Because I look at the Visual Studio integration, and I look at robably how this will eventually weave into PSS, interacting with customers, and saying well, here’s an odd dynamic: is a closed source company perhaps better able to actually make the exposure of the source code, when they do it, a more positive experience that actually lets people leverage it more easily? I mean, absent the ability to fork, and some of the more foundational principles of Open Source, how an outside developer would use it day to day development and how a closed source company in this case Microsoft could support that.

Shawn: That is a great question. I think that the answer isone of the things that I’ve been trying to do, and that we’ve been trying to do, is look at the changes in the industry that open source has driven and really think hard about what the real value adds are there. Is the value add really that you can recompile your kernel, or is the value add that you can debug it and understand why it’s doing something?

And there are obviously cases where the answer is ‘both’ or ‘one or the other’. However, for I think the vast majority of folks, us being able to deliver the source in a way that’s easy for folks to consume and step through as a way to make them more successful on our platform is a really big win. I think that it fits well into our overall strategy of kind of keeping developers integrated in our overall processes, and it fits well into this another milestone of transparency.

We have kind of been trying to figure out ways to open up the envelope of transparency with Developer Division particularly and I think this fits into that portfolio really wellas well as the portfolio that the Shared Source Initiative folks are building. You have CodePlex on one end which is full open source, then you have kind of our proprietary source system on the other end, and there’s a nice continuum in between. I think that we do bring some assets to bear which are difficult for companies that are purely open source or service model driven.

Sean: OK, cool. Well, that is all I’ve got, and also to be sensitive to timeScott, do you have a closing question or two?

Scott: I think that’s a good note to leave it on. The thing that we always end with is: are there things that we didn’t cover that you think are really important for people to know? So, we’ll just kind of hand you the microphone at the end if there’s something you want to get out that we didn’t specifically ask about.

Shawn:Nope, I think we covered it all.Thanks guys!

Bookmark this:

If you are looking for subjective opinion, I recommend looking through the interviews we’ve done here. At the risk of sounding like a Microsoft fan-boy, the Microsoft interviews (in my opinion) demonstrate a company where secure coding is “in the water”. Code goes through threat modeling, risky function calls have simply been banned, code goes through automated and human inspection, and vulnerabilities that do slip through feedback into the process to determine how to prevent them in the future.

I simply don’t get the same feeling from the open-source people we’ve talked to. When we’ve brought the subject up, the response is almost universally “many eyeballs,” and faith (without data) that “many eyeballs” is effective.

Am I completely off base? Do things like the Linux kernel and Apache go through rigorous security reviews? Is there proof that “many eyeballs” in open source is at least as good as something like the Security Development Lifecycle in Microsoft? If you’re in a position to know, let’s chat!

Bookmark this:

Update: It seems that this isn’t seen as happy news by all. There’s an article on eWeek that’s just too irrational and frothing to pass up, claiming that this is all a ploy by Microsoft to kill Mono. As Microsoft is officially supporting Novell’s efforts in porting Silverlight to Linux (on top of Mono), the evidence would indicate that Microsoft is doing this to support .NET developers, and not as some clever conspiracy to kill off Mono.

Bookmark this:

ComputerWorld just did an interview with Microsoft’s Scott Charney, which provides more insight into their efforts to produce secure products.

Bookmark this:

Interviewee: Ryan Waite

Ryan Waite

In this interview, we talk with Ryan Waite, Group Program Manager for High-Performance Computing at Microsoft.  We talk about:

• Microsoft’s entry into the High Performance Computing area.
• Microsoft’s support of the open source Message Passing Interface (MPI).
• MPI has become bloated, and there’s no usability data (like there is with Microsoft Office) to indicate what should be deprecated.
• At Microsoft, select customers help shape the features for the next product release.
• Microsoft has always focused on ease of use, simplified administration, and uncomplicated setup. HPC is no different.
• Instrumentation helps find reliability issues. If partners are affecting reliability (with unstable drivers, for example) certification helps raise the bar.
• Parallel programming is now the only way to do more computation in the same amount of time. Processors just are getting faster like they used to.
• It’s easier for open source to be incremental. Proprietary software has to make radical changes.
• Hard drives + overnight shipping = really high bandwidth.

Scott Swigart: We are conducting a series of interviews to explore the premise that open source software and closed source proprietary software are built differently — that the software development life-cycle is different, and that difference is likely to manifest itself in concrete ways in the final product.

That’s not to say that one’s better than the other, but simply that there are certain expectations you can have from either open source or closed source software because of the way it’s built, as well as certain inherent limitations.

In the past, ISVs pretty much just wrote closed source proprietary software, but today they have a little more of a choice in how they build a product, and we want to look at some of the decision points around that. And if you’re looking at bringing software into your organization, and you have a choice between something open source and something closed source proprietary, again, what are some of the common decision points or expectations around that?

To get us started, tell us a little about yourself and what you do, to give us a little bit of context.

Ryan Waite: I’m the Group Program Manager for High-Performance Computing at Microsoft. What that means is that my team is responsible for writing the technical specifications for our products; we design which features are going to be necessary for the product, then we write up how those features should work.

We work in close partnership with the software development and software test teams. The development team, as you can imagine, writes the code itself, and the test team tests that code. I should also mention that my product actually combines both closed and open source software — both Microsoft-designed as well as open source software.

My staff is comprised mostly of computer scientists or computer engineers of one kind or another. I’ve been at Microsoft for over 15 years, and I’ve worked mostly on server products, including things like Exchange Server, BackOffice Server, Windows Server, and Small Business Server — all sorts of different pieces.

And the product I work on right now is called Compute Cluster Server. As an HPC product, Compute Cluster Server allows you to bind 50 or 100 or 1,000 machines together to work on a single computational problem or a single set of closely related computational problems. So it’s basically a category of supercomputing.

The idea is to take very large computational problems, like proteomics problems, genomic problems, problems in mechanical engineering, weather simulations, and computational finance; and you can then run these against this huge set of machines. And the great thing about these kinds of clusters is that they cost a fraction of the cost of traditional supercomputers. But clusters are very powerful, and they’re very well suited to certain types of programming problems. These kinds of clusters are based on the concept of what are called Beowulf clusters, developed back in the early ’90s at NASA. The idea is to bind together a bunch of commodity servers with a high-speed network, and you can run some very interesting computational problems on them.

Most problems can be designed to work on a cluster, although there is still a category of supercomputing problems that you really do need a traditional supercomputer for.

Scott: Thanks. You mentioned that you have closed source and open source solutions; can you talk a little bit about what you do on each side?

Ryan: Loosely speaking, you can think of our product as having three major components. One is what’s called a job scheduler, which schedules work across this big cluster of resources. If you have 1,000 machines, it’s not like every job takes 1,000 machines; some take 16 machines, some need 64 machines, some need 132 machines. And then they have time limits for how long they want the machines to go — for example, they might need eight hours of computational time. So we have a job scheduler.

One of the gentlemen on my team was an architect on the LSF job scheduler, at a company called Platform Computing — he’s a rocket scientist. The second thing we’ve built is systems management functionality, and this allows us to manage this big cluster of machines. There’s a woman named Cathy Palmer on my team who was at Cray and Terra Computing for 11 years and has a PhD in job scheduling; she’s a rocket scientist, too. These are some of the best people that ever I worked with, so I’m just going to glow a little bit about how great they are.

The third area is about high-speed networking, and the parallel programming model itself. That’s headed up by another gentleman on my team named Eric Lantz. Part of that is high-speed networking. On the high-speed networking end of things, you need to enable these machines to talk to each other very quickly, because sometimes the speed of your computation is dependent on how fast the nodes can communicate with each other as they’re running their computations. It’s not just about the CPU inside that machine, but sometimes the I/O, or the communication speed between those machines.

For high speed, tightly coupled communication there’s something called MPI. MPI stands for the Message Passing Interface. This is basically the communication interface used for people to write these kinds of massively parallel applications that run on clusters. There are some standards for how MPI is implemented, but the reference design comes from Argonne National Labs. And Argonne National Labs distributes that code under a BSD-style open source license.

So Microsoft is a licensee for that code. We’ve taken that open source code from Argonne, and we’ve made modifications to it and we ship it as part of our product. The folks at Argonne are genius UNIX and Linux developers — really great UNIX developers. They’ve been doing this kind of stuff for years and years. And message passing is a tremendously complicated area of computer science, and they’ve done a great job at building this code. They call it MPICH and MPICH2.

The problem is that they’re not Windows developers. They don’t have a lot of experience with that. They have one person over there that is their Windows developer, but we have a lot of experience programming Windows and can certainly help. And Windows and Linux, are different operating systems, so you write programs for them in slightly different ways. Not radically different ways; in both cases you use C or Java or C# or whatever to write your applications.

But the thing to keep in mind with them is that you have different efficiencies in different kinds of ways. And so one of the things we’ve done is to implement those efficiencies in the Windows version of MPICH. All of this allows us to have a very high-performance solution.

We’re about to contribute those changes back. We’ve just been working through some of the final issues with how we transfer the code back. It’s just logistical stuff. In the next month or two, you’ll see a press announcement go out that talks about our contributions to this code, these modifications that we’ve made back to Argonne. As far as I can tell, this will be the largest contribution Microsoft has ever made to the open source community.

With this particular code, we want anybody running Windows to have access to a high-performance MPI library. If they buy it through my product, that’s great; if they are just running Windows and they want it, then they can get it from Argonne.

Scott: You’re in kind of an interesting position, because you’re one of the few Microsoft people developing both closed source proprietary and open source projects. What are some of the things that stand out to you as key differences between how those two processes work?

Ryan: What’s interesting about MPI is that there are a large number of features that have been driven because somebody in a standards body or open source group thought it was an interesting feature for what they were doing. They’ve implemented particular APIs.

Now, that API may never be used by anybody else except them. And yet it becomes part of this code base. The really great thing about that is that if you need to do a very specialized API for the application that you’re writing, you can go ahead and contribute it to Argonne — and if anybody else in the world has that same kind of problem, they might be able to use that same piece of code that you wrote.

The disadvantage of that is that this is now a very large code set with over 200 APIs. It’s a bit of a joke, and I don’t have any proof that this is true, but the joke is most developers use six or eight of the MPI commands out of that 200 for the bulk of their programming, so there’s a whole bunch of stuff that just isn’t used. And part of the problem is that you have to maintain all of that code as it marches forward, making sure all those APIs work.

They’re all documented, they’ve all been shipped, and ideally, they all continue to be well-supported. I think that’s the struggle with that kind of model — the code has exploded pretty quickly. It exploded because everybody gets to contribute the little things that they need into that code.

Scott: That’s interesting, because in other open source projects, the maintainers tend to be pretty strict and conservative about what they’ll let in.

Ryan: There is no single person for MPI saying, “this is OK, this is not OK.” You don’t really have the same kind of maintainer making things cross a very high bar before they get included. And those bars can take many forms, other than just the quality of the code.

Sometimes people establish a quality bar for when code gets included and sometimes it’s a customer requirement bar — you might want to make sure that if people are going to use something, it works properly. We all have to have different reasons for what we decide to include in software and what we decide not to.

I think it’s just the nature of these projects and the way software projects mature. They get bigger. Take the Linux kernel — it’s huge compared to what it was five or ten years ago.

As these products become more and more general purpose, they have to be able to service the needs of a greater and greater community. And we have an enormous amount of experience with this at Microsoft with products like Office, for example.

We have more usability data about how people use Office than you can imagine. Because we shipped instrumented versions of Office during the beta period, we know exactly which features people use and we know exactly how they use them.

So, even thought there may be a thousand features inside of Office, we know that every one of them is used. Everybody may use a different 20 percent, but we know that all those features are well-used. And features that aren’t used, we deprecate or remove from the product. We have very concrete usability data to tell us that.

Scott: Contrast that with the closed source work that you do. It seems like Microsoft goes through a process where there’s a big wish list of features, and people start coding on things as the project marches toward release. Things get cut, and developer resources get put on the more critical things.

How do you see the closed source work that you do comparing with the way you’ve seen the open source projects developed, in terms of how features are spec’d, how they’re built, how features are cut, and how things make it into the release product?

Ryan: That’s a hard question to answer, because it’s so different depending on which open source group you’re working with. I’d say for us, the first step early on in a project is to figure out whether a product is going to be more driven by a release date, or more driven by a feature set.

We do both at Microsoft. If you’re more driven by a release date, you really make sure that you scope that product so you’re going to do well at hitting that release date. And Office is a good example of a product that really drives to release dates.

The video game industry is an even better example. If you look at Xbox or you look at any of the game title providers that are building stuff for Xbox, they have to hit Christmas. No ifs, ands or buts. Otherwise, they’re not going to recoup any of their development costs, and those companies could fail. It’s really important for those organizations to scope their features to a particular date.

For other projects, you might look at customer requirements and pick a date, but you’re really going to drive to implementing that set of features and making sure that they hit the appropriate level of quality. I think that as those projects move on, that’s where you may begin to see some features get cut in order to hit a release date, or just because the team realizes that they’re less important.

We spend a lot of time sitting with customers. We have a customer advisory board called the Technology Adoption Program, and they come out here for a couple of days. These are big technology decision makers at government national labs, biotech companies, and financial institutions like banks and stock trading houses. We spend two days presenting to them on what we’re planning to build, and they give us feedback about what they’d like to see.

We release public betas. In the Technology Adoption Program, people actually deploy beta code and run it in production prior to the release of our software. That lets us know which features are being used and which ones aren’t.

I also spend a lot of time with people asking them which things they think are dumb or badly implemented, with an eye toward cutting those and focusing on other stuff instead.

The overall process takes the entire universe of things that you could possibly build, and you continually cut, as you move through the software development process, until you have what is really important at the end.

Scott: Again, this varies from open source project to open source project, but a lot of them are built by developers for developers, so the way a feature gets into an open source product is that somebody just sits down and writes it. Their first interaction with the process is by submitting code for a feature.

If they can’t code it, then that feature isn’t going to get included, so the downside is that you only have coders to some degree, spec’ing features. The upside is that nobody writes something that they aren’t personally going to use.

At a company like Microsoft, when you’re building the next version of a product, you’re doing a certain amount of work that’s fairly speculative. You’re building what you think people are going to need — you’re really building what you think people are going to use, and sometimes that’s more successful than others.

I mean, sometimes Microsoft delivers stuff that’s wildly popular and everybody uses it, and some things are dead on arrival, even though there was a lot of effort put into building them.

Ryan: Right — do you remember the “Bob” operating system? It’s not a successful product.

[laughter]

Commercial software companies have to determine what people want, and what will be successful. That’s why we spend a lot time doing things like customer research, usability research — really trying to figure out what we should build. I talk a lot about market opportunity documents, which is a fancy title for “what do people want?”

It tells us first of all, what set of users we are going to help. We segment the market, and we figure out where we’re going to prioritize. That’s the first round of cuts — we’re not building something for everybody; we’re building something for a particular group.

Then we ask, “What do those people really want?” and we visit customers to try to answer that. One of the first questions I asked is an open-ended one, which is, “What drives you most crazy about your HPC System?” Windows or Linux — it doesn’t matter — what’s the thing that drives you most nuts?

If you have some really open-ended questions, you hear some really interesting things come back from customers. Remember that HPC was entirely a UNIX market in the past. A lot has moved to Linux, but there’s still a fair amount of Unix out there, and some of them are beginning to adopt Windows.

And because so much of it is driven by the open source people, you have a whole set of computer scientists really driving at issues that they think are important in high performance computing.

But when you go talk to the system administrators, they tell you things like, “I’m struggling with power and cooling — the systems just run too hot.” The second problem that they talk about is, “It’s so hard to get the cluster up and running in the first place.”

Sean Campbell: Do you think that either closed source or open source software reaches out a little more effectively to some of those audiences and meets their needs?

Ryan: Take systems administration software for example. How do I get servers up and running, and deployed, and get the whole system going? We do really well with that, and it’s a good thing to do well at since a lot of people complain about it.

This is where things get a little bit touchy, because there are a lot of UNIX administrators who are very passionate about UNIX or Linux. And Windows administrators like to talk about how great Windows is. People throw mud all the time about this. I’m on a bunch of user group mailing lists, and people there say, “Well, the problem with Windows administrators is that they’re inexpensive, and they’re all the same.”

On the other hand, I have heard people say, “Every time we get a call from a Unix customer, or a Linux customer, their whole network is set up in a unique way. Every time I get a call from one of our Windows customers they’re easier to support because all Windows networks are traditionally set up the same.” It’s true that there are better prescriptions of how to do things in Windows than there are in the Linux world, or the Unix world, and so that has some benefit.

I think we do a good job of servicing the needs of the system administrators. Certainly in the database space there are a lot of debates about it. I think MySQL has actually come a long way, and is doing pretty well. I’m not a big database expert, but SQL Server really changed the database market.

I don’t know if you guys have ever set up a Oracle Database before, but it’s hard [laughs]. The first one I set up was about 10 years ago on a Solaris box, and oh my God. I consider myself to be a pretty technically competent guy. I’m not a database expert, but I spent a weekend just trying to get through the ten volumes of manuals about how to get the thing up and running and performing correctly.

With SQL Server, you walk through a wizard for setup, and you’re pretty much up and running. So, I think we do really well at servicing particular user needs, and servicing a broad general market as well, even though it may not be as technically savvy as people that are making contributions into the open source community.

Scott: Another area that I’d like to dig into involves some of the differences around identification of bugs, and support, and so forth. I’d guess that when you’re dealing with high performance computing and different architectures, you see all kinds of strange things that might be hard to reproduce.

On the closed source side, how do you handle that? What is the process for a customer to come to you with a problem and get a resolution, whether it’s just sending knowledge back to the customer, or whether it’s an actual change to the product to resolve it?

Ryan: One of the things that’s important for my group to keep in mind is that we’re brand new in this space. Windows is eight to ten percent of the HPC market right now, which is really pretty good. But if somebody has an HPC cluster, it’s probably a Linux or Unix cluster, so what has been important for us is that we understand the way that people are using their existing clusters, and that we fit into the model.

We provide support through our traditional Microsoft product support services, and we also have newsgroups that we support. But we also behave a lot more like more open source organizations might behave.

We have a site called WindowsHPC.net, and there are a set of bulletin boards up there that people can search to look for problems that they run into. Our development team here in Redmond reads those groups all the time.

That helps us plug in to that type of traditional model for how people support these systems. And that’s how people can report bugs as well. We also get bug reports from a lot of our partners. In the HPC market, it’s not like building a packaged product like Office, where you kind of ship it out there and you’re done.

I have to get OEMs like Dell and HP and IBM and SGI all lined up to ship our software. We have network hardware vendors, software vendors, critical applications like Mathematica and MatLab, that are all integrated in with our products. We have to get all of that put together.

So, we get bugs from those customers or partners as well. And those come, usually, directly in to us. They talk to somebody that’s kind of an account representative for them at Microsoft, and we get bugs filed directly. We get on the phone with them, and partners get a really close level of support in that way. When end users run into problems, they post to the newsgroups, and we help them out there.

So, the interesting thing here is that we do both. We do stuff that’s a lot more like the open source community as far as finding out about bugs as well as stuff that is more traditional.

The bug reporting service built into Windows gives us really great, qualitative data about where things crash, like the fact that most crashes in Windows occurred because of video device drivers. I forget the exact numbers, but something like 50 percent of crashes in Windows used to be because of video device drivers.

And then we have something called WHQL, Windows Hardware Qualification Lab, and they run tests on drivers. In order for a driver to get WHQL certification, it has to pass a battery of tests. So, we would look at the ways these different video drivers were crashing, and create tests that we would then put into WHQL, and that would improve the quality of those drivers continuously. Because even if it’s a particular driver that crashes, you still blame Microsoft, because Windows crashed. And so, it behooves us to really go through making sure that all of that code is performing correctly.

Scott: In other words, you’re able to regulate the ecosystem to some degree by saying, “If you want to be certified for Windows, submit your driver to our lab.” And then if your lab is running tests and they’re getting the driver to crash, you go back to the vendor and tell them to fix it?

Ryan: Actually, the qualification labs are run by a separate agency. They can contact us to get a waiver for a particular company, and under some circumstances that’s worth doing, to grant a waiver if they’re not able to pass a particular test.

It could also be, as hardware continually evolves, that the tests may not always keep up. If a hardware vendor implements a totally new piece of hardware, the older tests may not work correctly, so you have to be flexible, of course. But everyone wants to go through that qualification process, because the OEMs — like Dell and HP and IBM, etcetera — all want WHQL-qualified drivers on their systems. Otherwise, they’re going to be getting support calls as well.

Scott: One of the things that Microsoft’s been really focused on for about the last five years is security — just increasing the security of the stuff that you guys ship. Is any of that included in your certification process?

Ryan: There are parts. We’ve been releasing a lot of our security tools to the community, for people to use the same security tools that we use in their own products as they build their own software. We have a lot of very sophisticated software analysis tools that help identify security bugs before they even pop up. We published a book called “Writing Secure Code.”

Scott: We interviewed Michael Howard, so we’ve got a pretty good background on that.

Ryan: Actually, Michael and I have worked really closely. As he was doing the first round of the security development life-cycle, I was responsible for security on mobile devices like Pocket PC and Smartphone.

I would say that we do really well on the security end of things. And if you talked to Michael, you know that the issue is that if you miss one thing, there’s some hacker out there that has years to try and find that one bug that somebody missed.

Scott: He also pointed out that as Microsoft hardens their products, hackers are focusing more on the ecosystem because they’re softer targets.

Ryan: I’ve read that same data and I think that’s fascinating, the way that these kinds of ecosystems change over time.

Scott: So, the third party that does the certification runs tests, but I’m guessing it hasn’t evolved to the point where they say to a driver vendor, “Well, submit your threat models.” Do you see things evolving in that direction, or is that more intrusive into how an organization does what it does than Microsoft would want to get?

Ryan: I would say it leans toward your second point, that when you hand over your threat models, you’re basically handing over the architecture of your product. And I’m not sure that a lot of people want to do that.

Scott: One of the things that Michael also pointed out is that certain APIs have just been banned, things as simple as strcpy(), because they’re vulnerable to buffer overruns.

It seems like you guys would be in a difficult position because high-performance computing, on one hand, is all about speed, but I’m guessing that there’s a security aspect to it, too. So, how do you balance security and performance, or do you see that there’s any tension between those at all?

Ryan: I don’t think it’s black and white. I think there’s a little bit of tension, but not a lot. Here’s what people traditionally do with HPC systems: you have a cluster — let’s say you have 500 machines — and you have what’s called a head node. And a head node is where you submit computational jobs.

That head node will be sitting on the public network, and when you submit your job and your job is ready to run against all the compute nodes in the cluster, those compute nodes are sitting behind a firewall on a private network.

In a sense, all of those compute nodes can have a very open relationship with each other. They may not be hardened in the same way that every other system on the network is. And that’s OK, because they’re all behind a firewall. And all the jobs, when they’re submitted, are validated. You know what user submitted them, and they have permission to submit jobs. And if they don’t have permission, the job is rejected, and so forth. Those jobs are submitted into that cluster and they run on the cluster at that point.

There’s a pretty well-established security model for how clusters run computational jobs already. In the military installations that we’ve talked to, they actually hook them up in separate rooms. They’re not plugged into any kind of public network at all.

Scott: I would assume that just for performance reasons if nothing else, you don’t want your cluster sharing a network, and it’s also probably on a much higher speed network than the rest of your organization. I would assume that it’s pretty isolated.

Ryan: Here’s where it gets really interesting. The thing that my group is really pushing for is moving high-performance computing into more mainstream areas. We don’t focus on share-shift of moving people from Linux to Windows in the existing market; our job is really to grow the HPC market. And we expect that there’s going to be a little bit of share-shift, and that’s great for us, but we’re really focused on growing that market in new spaces.

One of the areas that’s very interesting is departmental and workgroup clusters. Those people may set up the whole cluster on a public network. In those systems we need to make sure that that system is secure by default when it comes up.

Scott: Do you ever see a day where every computer in the organization is potentially also part of a high performance cluster? Or do you think there will always be dedicated machines that are separate from machines running the user-workloads?

Ryan: Are you talking about a situation where my computer in front of me would actually be plugged into a cluster on the back-end?

Scott: Right. Kind of like the SETI@home thing, where my machine is idle, so it starts working on this job.

Ryan: In some cases, I do see that happening. People call it “cycle stealing,” or “workstation clusters.”

Those systems only work well for a particular set of computing problems, though. For certain kinds of computational problems that have a high number of cycles that provide data, those kinds of clusters work great. So SETI@home is a great example; there’s also a protein folding project which works this way. When your screensaver kicks on, you start doing your computation as a part of the screensaver. There’s a whole set of other problems that don’t actually work that well. Seismic processing is a great example of a problem that doesn’t work very well, because they have very low number of cycles of computation per byte of data. I talked to one of the companies that does this kind of work on a 10,000 node cluster, and when they get customer data in from an oil company, it’s a whole pallet of tapes.

The whole thing gets loaded into a gigantic tape array, and then they farm it out across a 10,000-node cluster for computational analysis. They’re dealing with just ridiculous amounts of data, and that stuff wouldn’t work well with cycle stealing. When you really get these systems working in the enterprise phase, you’ve got to figure out how to make sure that data that’s residing in London doesn’t get shuffled off to China for computation, because it’s so inefficient to move it all the way around the world. You may actually spend more time moving data than you do computing it.

Scott: So, in other words, ‘high volume of data, low computation per byte’ is bad for that cycle stealing scenario. ‘Low volume of data, high computation per byte’ is good for cycle stealing, because you ship a little bit of data out to a machine, and it just crunches on it for quite some time and submits, again, a fairly small result set back.

Ryan: Yes. Here’s what I think is a more interesting direction, in terms of your question about what’s happening with my PC and my cluster in the background. We’ve hit a point where CPUs aren’t going faster any longer. 4GHz is pretty fast, and the amount of heat generated by a processor running at 4GHz is enormous. The reason we can’t go faster is they just get too hot.

So the silicon vendors have started moving in a multi-core direction, and so we see four cores right now, we’re going to see eight cores, and pretty soon a bazillion cores are going to be on these machines.

And this is an issue for the software industry as well as for Microsoft. What’s been so great about the software industry is, as processors get faster, we can add more and more features, right? The Linux kernel gets bigger, the Windows operating system gets bigger, we have richer features, we can do more sophisticated stuff than ever before. But now that processors have hit kind of their Gigahertz speed limit, we can’t just willy-nilly add features into our products any longer.

The bad thing that could happen is that the software industry would become very similar to the white goods industry. You don’t buy a new washing machine because ooh wow, this one has this great new spin cycle! No, you wait until your washing machine breaks, and then you buy another one.

What has to happen in the software industry is that all software development needs to start moving in the direction of being aware of concurrency.

That concurrency could be across multiple cores on your local machine, or it could be across multiple machines running in a cluster. And so, the thing that I predict we’re going to see in the future, is that programming models and the kind of programs that are being written will not only be better at taking advantage of multiple cores, but those programs will be better able to run on a cluster as well.

Scott: I’m sure that this is an area of just absolutely intense research, but right now, it’s just too hard for the average developer to write a good parallel program.

It’s too hard to write a multi-threaded program; it’s too hard to write a program that’s going to run on a cluster. Event driven programming became easy when it was just something the languages natively knew about. And right now, the languages don’t really natively know about threads, and concurrency, and all that stuff. It’s done through libraries; it’s not really done through core concepts in the language. You have to write the thread synchronization logic, and you can’t really just declare a synchronized integer, for example.

What do you see as the future of the tools, so that it’s not so insanely difficult to do it right? Do you see the way it’s done, with high performance computing, where you’re sending off batches of work and getting results back, do you see that as being a viable model for a multi-core system? Or do you think there’s some new paradigm on the horizon that might unify both?

Ryan: For general purpose programming, it’s the latter. We start moving to new programming models — not radically new programming models, because we can’t retrain everybody in totally new programming constructs, but you’ll see subtle shifts. Languages like C# will continue to evolve and understand better concurrency, and it will become simple for developers.

I think we have to get there, because I agree with you, trying to teach average software developers about how to do concurrent programming just isn’t going to work, as it stands right now. And if you think that’s hard, you should look at the MPI programming that people write. This is akin to Assembly level programming — it’s just outrageously complicated.

You get this 500-node cluster, and the communications are not handled automatically. One node switches to receive mode, the other node switches to send mode, and then it sends data to the machine that’s in receive mode. If they both switch to send simultaneously, or they both switch to receive simultaneously, it deadlocks! And your whole 500-node job halts.

Scott: Wow. Getting back on some of the differences between closed source and open source, to follow upon your statement that “you can’t just change everything because it’s hard to train everybody to do things differently,” open source seems to move very incrementally, because of the way it’s built, because things are submitted as 100 lines of code to a mailing list where they get scrutinized, and maybe it makes it in and maybe it doesn’t. And a lot of times, things are worked on in fairly small pieces.

It seems like it’s fairly impossible for an open source project to make a radical leap. There seem to only be two ways that happens: one is that somebody forks the source and does a fairly major change to it, or a whole new thing comes out of nowhere; you know, Ruby on Rails shows up out of nowhere, and it’s all the rage for building web stuff. People look at it, and they have to decide whether to stick with PHP or to learn a whole new thing.

In closed source, it seems to be a little different. It seems that there’s a lot higher barrier to just releasing something completely new that has no connection to an existing product. There’s a lower barrier to taking an existing product and maybe doing something a little bit more radical with it. Like in Office, they came out with the Ribbon, a whole new UI paradigm. I don’t know that something like OpenOffice would have ever evolved something like that.

Vista is obviously built on the same code base as previous operating systems, but there were some pretty radical departures there that had some impacts on application compatibility and issues like that. And again, it would be kind of a stretch for open source to do that. But on the other hand, you might come out with a whole new shell that competes with Gnome or KDE or their peers.

Talk about that a little bit. Do you think I’m onto something, or do you see things that run counter to my observation?

Ryan: I think it’s a very good point, and it’s something that I’ve also thought about and wondered about. Part of what’s powerful about the open source community is that it is incremental. And as they make their incremental changes, if they’re good, people use them; if they don’t, nobody uses them, and they eventually are discarded. A good example is user interface models. We have one, Apple has one, and those are very popular user interface models.

The X-Windows model was never really that popular. It’s popular among a particular group of people, but not like the Macintosh and Windows paradigms were. And they’ve continually competed and evolved with each other. And I think you see Gnome and KDE coming in and filing the best of what they like out of those models and evolving that, and maybe keeping a little bit of some of the X-Windows stuff that they like as well.

Those are highly evolutionary systems. Things like OpenOffice can really pick at what has been successful–like Office, for example–and take that and put it into their products. At the same time — and I think this will be true for a lot of commercial software companies — because we spend so much time trying to figure out what people would pay for, if we think something is going to be paid for, we’ll invest the money there.

Scott: It seems like there’s almost a pressure not to do things that are incremental, because unless it’s a big enough delta between this version and the previous version, people won’t pay for it.

Ryan: That’s right. We’re constantly in this business of having to generate value on a huge scale, because it’s not like we can build something that’s interesting for a hundred or a thousand people. It has to be interesting for millions of people. And so that’s why we can spend so much time figuring out the next big shift that’s going to come along.

Sometimes that means that we look at stuff that’s been popular in niche markets and figure out how to make that go really big. And sometimes it’s radical new things. In some ways, for example, you can say that the original Windows NT operating system was an evolution of traditional systems, based on work that was done at DEC, and work that was done at some other software companies, to create this new kind of kernel.

But NT was a fascinating concept. If we go back and think about it, the idea was that you could write a kernel that would span from the smallest laptop computer up to gigantic SMP systems.

Scott: The whole Hardware Abstraction Layer was an unproven theory and a pretty radical idea.

Ryan: Right, and it’s now the foundation for Vista.

You know, HPC is a really interesting market to study from an open source point of view, because it’s a full ecosystem, and it’s still pretty heavily driven by the academic community, which is, by nature, fairly open source-oriented.

We fund work at something like 10 universities and research labs around the world, like Jiaotong University in China, and Pacific Northwest National Labs, and the University of Tennessee where Jack Dongarra does all of his HPC research. And we do that because it’s still critical to really be working well with the academic community, in order to be successful in this market.

Scott: They’re the ones running the climate simulations, things like that. They’re the ones who, largely, are running the kind of models that need high-performance computing. One of the things that’s kind of interesting — it’s not really high-performance computing — but Amazon has in beta what they call their Enterprise Compute Cluster.

Ryan: Yeah. My friend Marvin Theimer is an architect at Amazon.

Scott: Do you see a future for hosted high-performance computing? Do you see a future for maybe ‘high-performance computing lite,’ where it is more tailored to the kind of jobs where you’re not moving enormous quantities of data, the computation per byte is higher, and you can have simplified communication between the nodes?

I wouldn’t have to have 500 nodes, but I could push a job up to some hosted environment that could put 10 nodes or 10,000 nodes on it. Does that kind of thing exist today?

Ryan: It does exist today. Sun has been doing this for a few years now; it’s at least two years. They have this model where they charge on a CPU-hour basis. We talk to those guys, and they’re very open about what they’ve been up to, or at least they used to be. And the way that those jobs work is you actually ship all the disks over there.

Jim Gray did some really interesting research here at Microsoft. He computed how much it costs to move a certain amount of data and how long it would take to move that much data over a particular kind of pipe, because the question is: how long does it take you to get a terabyte of data from one site to another?

Scott: DVDs and FedEx have an incredibly high bandwidth.

Ryan: Yeah. That was the result of the research, years ago: “Just put it on DVDs and ship it that way.” And so what Sun does is that you actually ship them a cabinet of hard drives, and they plug that into the cluster and use it to compute.

Your point is very good though, that a lot of customers will typically do small computational jobs and occasionally they’ll need a really big cluster. I think that is true going forward, that we’ll see more and more of that, and people will occasionally need to run a big job.

If part of what we believe is true, that this revolution will happen in workgroup and departmental clusters, it’s because workgroups and departments will need those smaller clusters for some of their work, but every now and then they’re going to want to hand jobs to a huge cluster for computation.

And that may be handled using grid standards that are generated by the open grid forum. That may be handled by just handing off to a place like Amazon or Sun.

I also think that, in closing on the discussion of open source and the HPC market and the commercial software that we’re building for HPC, I’m really hoping that what’s going to happen is that our group at Microsoft will help pave the way for how commercial groups at Microsoft are better able to work with open source projects.

Scott: Thanks, Ryan, for all your insight. This was a great conversation.

Ryan: Thanks.

Bookmark this: