Interviewee: David Campbell

In this interview with David Campbell we talked to him about:

• How the development methodology of SQL Server has changed.
• How the changes affect the ability of customers to give feedback.
• How virtualization makes it easier for users to test-drive SQL Server.
• As products and open-source projects grow, the developers have to realize that they are not the target user.
• David’s belief that the best product doesn’t necessarily have the most knobs.
• Building a community around a closed-source product like SQL Server.

Scott Swigart: David, thanks for taking the time to chat with us again. One of the things you mentioned in the first interview was the common engineering criteria. I’ve seen that mentioned in a number of different places, but I have never seen it explained. And I’m guessing that the common engineering criteria impact a product like SQL Server. It might be good to start off just by explaining what they are.

David Campbell: It started in the Windows Server system. We’ve been telling customers that one of the advantages of our stack is that it works better together. It does, but we started really challenging ourselves by asking, “OK, what are the real things we’re doing to actually make it be better together?” The way I think about it is, “What are the things that would add value for customers who require coordination or consistency across groups that otherwise wouldn’t happen organically?”

We started this a number of years back, when Paul Flessner was running the Windows Server group. We have a set of requirements, which are updated every year, and all the enterprise products within the Server and Tools Business (STB) have to adhere to those requirements. These are things like having a best practice analyzer, having a Management Pack for System Center before you RTM, having consistent user education, continuous publishing, a community engagement program, that kind of stuff.

Scott: Previously when we talked, you also mentioned that SQL Server went through a pretty big change in its development methodology between 2005 and 2008. Talk about that a little bit.

David: This is something that a small group started, and it ultimately took great effort across the division to pull off. It will never be “done,” but it has been an amazing, fascinating transformation. I’ve learned more in the last few years about culture and change management in a large organization than probably anything else.

I’ve been working on the product for a long time. Back in the good old days, we had something like 20 people who knew the whole server, end to end. Back then, as you worked on things, if you had an issue you just walked across the hall, sketched something out on somebody’s whiteboard, and went back and coded it up. Because things were so intimate, you could keep things moving at a rapid pace.

To somebody who understands the standard Microsoft development methodology, one of the things that’s very important is the daily build. We continuously build and test the product, and one major vital sign is the health of the daily build. Back when the product was small and there were 50 to 100 people working on it, you could do the daily build live, with everyone just checking stuff in and keeping it together.

But as the team grew and the size of the code base grew by several orders of magnitude, you just couldn’t have as many hands in the soup at the same time and get anything that tasted good.

In SQL 2005, we had sort of a standard large-team, waterfall model. If we were doing a feature that spanned several component groups, we would have one group get together and they’d do everything by the book. We had a process and everyone followed it. They’d write the spec. They’d develop a test plan. They’d review the test plan. They’d write the code. They’d test it. They’d check it in.

Then the next component team in the sequence, maybe it was the client data access team, would pick it up and they’d start working with it. And they’d go, “Hey, wait a minute. This interface isn’t quite right. There is something wrong here.” They’d go back to the first team and ask them to change it. Of course, the first team had moved on to something else, so if they had to go back and do major surgery, it was a hassle.

We had one feature in particular that, to be honest, we built two or three times during the course of SQL 2005 and we still didn’t have it right. So I said to the program manager at one point, “Look, either we take it out of the product or you get everyone, end to end, involved, get them in a room and just sit till you work it out and get the thing going.”

This involved the storage engine, the language parser, the query processor, the client data access libraries, the management tools. They had done it serially several times. They just never had it work correctly, end to end, when integrated.

They all got in a room. Of course, you know how that works. They were able to figure it all out and get it all done. We said, “OK, what can we learn from this?” We completely threw the development process up in the air and changed it around. How do we go “back to the future” and try to capture the intimacy of what we had in the “good ol’ days” now that the team and product are many times larger and more complex.

When we had a small team rebuilding the database engine for SQL Server 7.0, a number of people had the whole system in their heads. Furthermore, when we did SQL Server 7.0, we really didn’t need to think about the market that much. We were building a new relational database engine and introducing an OLAP system. The playbooks were already written for these technologies and the markets were already established. Now, we did it in a way that was different; we tried to make the product much easier to use so we could actually grow the market and bring the technology to a much larger base. For SQL Server 2000 and, to a lesser extent, SQL Server 2005, we were building out on the architectural base we had established with SQL Server 7.0.

The original model worked great for several releases, but when we got to SQL Server 2005, the product was much bigger, the team was much bigger, and the market expectations were much higher. Things just weren’t fitting together in terms of consistency across the product. So the other piece that we had to add for the SQL Server 2008 process was, instead of defining the release from the bottom up in terms of what we were going to do, we had to create a definition process that looked both at the market, top-down, and at what needed to happen to the technology base of the product and what people wanted to do, bottom-up, and bring them together.

The way we framed it to the team was, “Look. We want to have the PowerPoint presentation that we’re going to use in the keynote for the launch before we write the first line of code.” So we developed a set of themes for the release. With each theme we had this notion of scenarios that were end-to-end value propositions for customers. Underneath those scenarios, we had a number of what we called improvements, instead of features.

An improvement is what we would consider an engineering transaction. The idea is that we bring virtual teams together to work on an improvement. An improvement team is cross-discipline—development, testing, and program management—and it’s cross-component. For example, the database engine, data access, and management tools teams can come together to work on a single improvement. So if a particular improvement needs focus from four different components, they’ll all come together as a virtual team.

We assign an improvement team lead, who’s got a set of requirements they need to meet to integrate the result into the source base, but we’re not very prescriptive about how they run the team. If they want to use Scrum and have stand-up meetings and do a series of sprints, that’s fine. If they want to use a test-driven development methodology, etc., that’s up to them. We’ve got tools and a playbook that we encourage them to use, but we’re pretty liberal in terms of how they go about it. There’s a very strict contract in terms of what needs to be done before it is integrated into the main branch of the product source code, however.

These improvement teams work in their own separate branches in the code management system. Once they believe that the improvement is ready to ship to customers, then and only then will we integrate it into the product. What’s interesting, if you think about it from outside SQL Server, from the customer’s perspective the release looks much, much different than it used to look.

It used to be the case that—let’s say we were going to do 300 features. Everyone gets started on them. We put a bunch of stuff together and put it out for Beta 1. Everything would work about 30 to 50 percent. Then we’d keep going for Beta 2. We realized, “We’re not going to be able to finish 20 percent of these things, let’s cut them.” With Beta 2, things are working about 70 to 80 percent correctly. Then you have Beta 3, a bit better. And in the end, you’re still trimming stuff out, cutting it, and refining it to get it ship ready for customers.

We were constantly in the mode of having stuff that kind of worked in Beta and trying to get feedback from customers. We flipped it around with the new process. What customers see now in a particular drop is very high quality. They used to see the 120 percent of the ultimate features in Beta 1 in some measure. Now they only see 20 or 30 percent of the features early, but they all work. We also changed the name from Beta to Community Tech Preview, or CTP.

As we go through each successive CTP, more and more of these improvements show up. They’re baked, end to end, and they accumulate in number. What a customer is seeing is that the quality is actually very good the whole way through, so it’s an interesting change.

Scott: It seems as if the SQL Server team went through the same process the Visual Studio team did: Instead of putting code out there that is 30 percent functional and people can’t really get it to work end to end, they decided it’s better to wait until a given feature is 100 percent even if that means customers see it much later.

But there has to be a trade-off, because your window for getting customer feedback is smaller, and you have potentially invested a good deal more time in the original code before you get any feedback. Talk about that balance.

David: That’s a great question. One aspect of the previous releases is that we were frankly playing catch-up in an established market. We knew what we wanted to build and we knew how we wanted to differentiate in the market so we really didn’t need a lot of customer feedback on the feature set for SQL 7.0 because in many ways we were rearchitecting and reimplementing the existing product. This approach worked for SQL Server 7.0 and 2000, but one consequence is that we hadn’t built up muscle around how to engage customers earlier in the design process for completely new features. In some sense, we built stuff and threw it against the wall to see if it would stick. This worked well for awhile, but it made much less sense starting in SQL Server 2005.

Basically, you could say that the old process was quite inefficient, in the sense that we built a bunch of stuff, actually committed code, before we even knew what would stick in the market. Now we’re trying to do that through more concept testing, more up-front work, spending more time with customers to look at their real pain points. Are you going to build a set of features that’s going to be compelling? That’s one piece of the puzzle. Are you building something all up that will be valuable to your customers to you customers end to end considering they way they really work? That’s another pieces of the puzzle.

But then the question becomes, have you built each feature in a way that makes sense to the customer and is useful? So it’s less about the value proposition of having a particular capability and more about have you designed and built each thing correctly, on an improvement-by-improvement basis.

One fallacy around the Beta process, in my opinion, is that we would throw a Beta out there, and tens or hundreds of thousands of people would download it. A small fraction would really play with it, a smaller fraction would actually think about how they’d use a new feature, and a fraction of those people would actually give us any real feedback. It wasn’t very structured. Our TAP program tried to target specific features and scenarios but it mostly kicked in after we had written the code.

I believe it’s a bit different in the Developer division, because you’re close to developers. One of the challenges we have with SQL Server is that, for our real validation, you need to get closer to a production environment. And it’s very difficult to get the feedback from the production team.

Scott: Right. With SQL Server, a single dev banging away isn’t a real test. You could get somebody banging away at the feature, but that doesn’t really tell you whether it will scale, whether it will be reliable, whether it will be secure.

David: Right, exactly. So what happened with this release was, we had some teams that said, “Oh, crap. What are we going to do to get feedback for this really complex improvement?” For the database engine there’s an improvement called a Resource Governor. It allows you to constrain queries that consume specific resources so you can limit a particular connection to 10 percent of the CPU, and that kind of stuff.

The question is the mechanics of getting the architecture and user interface right. We think very hard architecturally about separating mechanism and policy. In this case, the mechanism would be things like how do we keep track of the various resources and who do we attribute the resource usage to. The policy would be how the user describes her intent in a way that allows the mechanism to enforce it. In the case of the Resource Governor, there’s a complex interaction between the policy and the mechanism, – not the least of which is that we needed to build new mechanisms to hold and enforce various policies. This is a great example, because what we would have done before was stare at the whiteboard, do a design, and throw something together. We’d ship it out in a Beta and hear, “No, that’s not what we need.” So it turns into a rock fetch. “No, this thing stinks. … This thing stinks. … You’re getting warmer …,” etc. It’s a pretty inefficient process all up.

In the case of the Resource Governor, we got together with some of our MVPs and key customers who had a need for the improvement and did more of an outside-in design process starting from use cases. What’s interesting is that this outside-in approach is completely natural for many people building applications, but for a development team that’s been creating low-level system software for 10-plus years it’s quite different.

The bottom line is that we definitely needed to figure out how to engage customers differently with our new development process, and there are opportunities for design councils, concept testing, early prototypes, and early builds. In fact, things like virtual machine technology allow an improvement team to create a build of their early thinking and test it with a small group in a focused way without having to integrate everything else that’s in various states of completion, like we used to do with the standard Beta process. They just create a VHD with their bits and have some customers try it. With this model they can iterate faster than was possible with the old Beta process. Ultimately, I think we’ll get better improvement feedback by finding the right 10 to 20 customers and working closely with them rather than putting it together in Beta and throwing it out to 100,000 people hoping they’ll give us valid feedback. I’m overstating things a little bit to make the point, but the general concept still holds, although not everyone here agrees with me on this one.

Sean Campbell: I’m curious about the point you just brought up, because Scott and I have been knee-deep in virtualization for a long time. The point you just made about letting people evaluate it on a VM, how has that impacted the overall development process for the SQL Server team? Has it given you the ability to basically give, let’s say, broken bits out earlier and get more people working with it earlier because they don’t have to install it, they don’t have to configure it? I’m sure in the past people would try to install it on top of whatever else they already had. You could tell them a hundred times, right? And they’re going to try to install it as a multiple instance on top of what they’ve already got.

David: It has, in exactly the ways you said. These applications are so complex, and the state that they put on the machine is so complex, and we work so hard to make sure uninstall really uninstalls all the stuff that it put there in the first place.

The other piece is that it’s hard for us to build and test the setup and the installation. With VHDs, we can basically handcraft them and then clone them, and they don’t pollute any other state on an existing machine. So it’s much easier to get together. The bulk of the savings is in the setup, for us to construct both the setup and the uninstall, to make sure we don’t pollute the machine state.

So I think from that perspective, it’s very, very helpful. I think we saw that, but we haven’t really perfected it as a process yet. I think we’ll probably go a little bit deeper on it in the next release.

Scott: In talking to a lot of people in the open source world, especially community-driven open source, like the Linux kernel and the Apache Web Server, I hear there is a pretty strong correlation between your ability to request a feature and your ability to actually code it. You will get on a mailing list and say, “Hey, it’d be great if the Web server did this.” And the response is basically, “Yeah, that would be cool. When will you have the code ready?”

What you have talked about is very different from that, in terms of interviewing customers, interviewing influencers, and putting together a team. Talk a little bit about how a specific feature happened from end to end, how the feature was initially conceptualized, how you validated it, the portions of the product it touches, etc.

David: Well, I’ll talk about it in the meta form, and then I’ll talk about a particular feature. It goes back to the thing I was saying about technology evolution. And it’s actually one of the challenges.

One of the challenges the open source community faces, and this will be a sweeping statement that I get flamed for, is that they don’t yet generally appreciate the difference between the consumer and the producer. By that I mean, as a technology matures, the gap between who’s using and who’s producing it becomes greater. It’s a challenge we face at Microsoft; we are no longer simply building products for people like us, the engineers. In one of the talks I give to the product development community at Microsoft, I stress, “Look. We’re no longer just building products by engineers, for engineers. The PC is moving toward a consumer electronics device for many users and we must build a product that interacts with them on their terms, not ours.” I then show screen shots of Task Tray bubbles talking about “Virtual Memory” and “Pagefiles,” etc. My friends and family shouldn’t have to know what virtual memory is.

We’ve seen the same thing in the database space. Twenty years ago, every DBA had to be a wizard and needed to know how to fiddle with hundreds of knobs, figure out how to manage raw disks, and learn all sorts of low-level details just to create and maintain a database. When we were building SQL Server 7.0, I used to troll the database newsgroups an awful lot. You’d see people asking perfectly reasonable questions and you’d see responses like “RTFM” or “You don’t belong here if you don’t know that.” So we started this campaign to fix what I called the “you dummy” questions. Basically, if an expert responded to a question with a “you dummy” tone, we saw it as an opportunity to address the issue by engineering—teach the product to do it rather than teaching hundreds of thousands of DBAs. We turned the “you dummy” on ourselves and did something about it.

I don’t have to be an expert on my automobile to keep the thing running, and that’s fine. If Toyota wanted to come to me, I wouldn’t have to talk to them about ignition advance, dwell time, and those things anymore. I’d talk more about my experience and what I wanted the car to do for me. I think that’s just an evolution, and you can certainly see it in the database space.

To get back to the specific feature, I think the Resource Governor is a good one, where there was a complex set of mechanisms inside our system that we needed to go off and engineer, but how we constructed those and how we presented them to users was the challenge, right? What is the policy? How does someone describe a policy around constraining resources in a particular query or session? And then how does someone bind different policies to different classes of users or applications or queries or times? What degrees or what dimensions do they need?

Another way to think about this, from the perspective of design, is that the software development community has not made the leap yet from expressing the product control surface as knobs on the underlying mechanism to capturing the user’s intent or objective and then acting to achieve that. A simple example is: Rather than having an administrator configure how many dirty database pages will trigger a database checkpoint, we can capture the user’s intent in terms of the trade-off between fast recovery in the event of a failure, which can be achieved by increased checkpoint frequency, or better run time throughput, which can be achieved by fewer checkpoints. Having a control that captures how long the administrator is willing to wait for recovery to complete in the event of a restart looks directly at the business intent rather than fiddling at the level of the mechanism. I mean, how many dirty log blocks can I have on a particular database and still have it recover in 60 seconds, anyway? Instead of publishing some equation in the documentation, capture the user’s intent and treat that as a constraint during run time.

So to go back to the Resource Governor—we go to the customer and ask, “OK. What sorts of things do you want to constrain? What dimensions are most important? Are they applications? Are they time of day? Are they this or that?” And then gather all that, design something that’s consistent from the user intent perspective, and then figure out how we build a mechanism underneath to marry up with it.

Certainly, there’s a lot of success in the open source community, using open source technology in consumer electronics. But I don’t think that they’ve gone end to end in terms of doing a great job of user-centered design.

Sean: Well, that’s one of the things I’ve talked to people about, too, and that I personally feel pretty passionate about, innovation in the usability area.

I am curious, from a closed source company perspective, about what processes you feel should be put in place in order for a development team to excel in terms of usability. You go out and talk to people about potential features to help make sure that a feature doesn’t turn into engineers building it just for engineers, and that you actually end up with something—like the Ribbon in Office—that honestly has been a success, if only because, after release, nobody complained about it.

It’s the proverbial tree that fell in the forest and nobody heard it, but everybody was freaked out about it. And I keep telling people, “Well, think about this. Microsoft changed the UI on the one application people use the most, on a daily basis, and yet nobody really complained once Office launched.” There must have been something that really went into the thinking, in terms of usability.

For you guys, especially with a product like SQL Server, where I’m sure you can lose yourself in B-tree discussions until the end of time, how do you ensure usability in the product?

David: It’s a great question and a great challenge. It’s funny, but one of the things I’ve been doing for awhile that’s had a good effect is to hold the mirror up to the product development community at Microsoft, to get them to see that we’re no longer building products for people like us.

I have a set of slides that I’ve been using for six years or so, where I have collected some crazy error messages and I have real-life scenarios. I’ve got a screen shot where, in Windows XP, if it extends to your page file, you get this balloon coming out of the system tray that says, “Your virtual memory is low. We’re extending the page file. Blah blah blah.” I came home one night and my wife hit this and she said, “Hey, I think my computer’s broken. It says something here.” So I started describing to her virtual memory and page files, and she said, “Shut up! Just fix it.”

It dawned on me that perhaps 99 percent of the people using PCs these days are not engineers, yet we’re still throwing these exceptions and doing these sorts of things as if they were. And we ridicule the users instead of ourselves. I go help my neighbors fix their machines, and they all start with, “Oh, I feel like such an idiot around computers.” And I just want to say, “No, you’re not the idiot. We are, because we’re not building computers for you yet.”

And so I have a set of rules, or things that I’ve captured, that I talk about, such as the “principle of least astonishment,” where a reasonable user action should do something reasonable with least astonishment. The example I use is another story from my wife. I don’t know how many years ago, when I first got her a PC, someone emailed her some pictures. She said, “How do I look at these pictures?” I said, “Just double-click on the filename right here.”

So, of course, up comes the picture in the picture viewer. And there are little buttons down on the bottom of the viewer: “Next picture,” “Previous picture.” What do you think happens if you click on “Next picture”? You don’t get the next picture in the set of attachments. You get the next picture of whatever the next file was in your IE cache or whatever it happens to open.

And she’s going, “What the hell is this?” She gets some random icon from the temp directory or something like that. So I started talking to her about temporary directories, the IE cache, and blah blah blah, and again she’s like, “Shut up! Just tell me why it’s broken.”

Sean: She’s saying, “I’m looking at photos. I want the next photo.”

David: Exactly. And you see that even in the database space. We’ve taken the market from tens of thousands of servers out to millions or tens of millions of servers. You can’t have every one of them require a highly skilled DBA. We just couldn’t do it economically. And so you have to get out of the mind-frame of an engineer and go adopt a perspective of what the customer needs and how the customer wants to express their intent.

That’s the other sort of piece I talk about: In terms of the degrees of control and the dimensions of control, get rid of all of those dimensions of control that don’t capture any user intent and figure out which dimensions of control are orthogonal with respect to the customer’s intent, and then figure out how you build your system, and express things that way. That’s been the most effective thing I’ve found to motivate the change.

The bottom line is that the software development community doesn’t teach “design,” certainly not with a capital D. Most developers believe that if they think for 10 minutes before coding up some function, then that’s design. It’s not. This isn’t a development methodology thing, but rather a cultural change that we need to go through across the entire industry.

Sean: That’s great. The chief creator of a project called Quicksilver for the Mac was giving a talk at Google, and I just saw the video of it the other day. His theme for the project is “act without doing,” which has an interesting similarity to what you are talking about. If you saw the app, you would probably see some interesting things in it.

To follow along that line, I have a question about Oracle. Oracle, through the years, treated the product as though the more complex the cockpit, the better it was. If you walked in and it looked like an old 727 cockpit with 50,000 switches, that was nirvana. And you were paid to know every position of every switch, right?

David: Yep.

Sean: The open source community has, to some degree, a similar bent, right? Not in everything, but in a fair amount of places. But even in a product like Ubuntu, which is seen as the ultimate of end-user experience—and it has made some great strides—there’s a massive list of steps just to join that machine to a Windows domain, for example, if that is something you wanted to do.

David: Yep, yep.

Sean: So, do you see any similarities with this line of thought in the way Microsoft helps you to administer a database and the way open source helps you with administration in projects like MySQL?

David: That’s another good question. Yes, Oracle—I think I mentioned it earlier, they marketed our ease of use against us, frankly. They said, “How can this thing be a real enterprise database product? It doesn’t have nearly as many knobs as ours does.” And when we left Digital—this is an interesting little story—we had a product, RDB, that had roughly as many knobs as Oracle did at that point.

And like engineers, we were saying, “Oh, shoot! Hardly anyone knows how to turn the knobs on this thing.” So we wrote more software, something we called RDB Expert, which was a physical database design and some other things, to recommend and actually turn some of the knobs for you.

I think the first step that you go through in this evolution is to put a facade on it, whether it’s a system that recommends turning the knobs or a GUI to cover up knobs in config files. But really, at that point, you haven’t got a good end-to-end design, because you haven’t, in my opinion, eliminated those points of interaction or control. You haven’t captured user intent.

It’s like having a TV that still has horizontal and vertical control, but you’ve put something on the TV that can watch it and turn the knobs for you. So I think the next step in the evolution is to have the system software itself become more adaptive.

Scott: That’s true. TVs used to have horizontal and vertical knobs on the back. They don’t have those knobs anymore, even though TVs are much more complex, and no one misses them.

David: And what you wind up with—here’s the key point, and this is something I didn’t understand a priori, but after the fact it really made a lot of sense. In SQL Server 7, when we did some of the memory work to have the system automatically adapt to the environment and adjust the amount of memory it consumed, we changed the system so that you could adjust the amount of memory that it was using dynamically. And the mechanism was dynamic as well, so we had an automated policy that was a closed loop, but for those people who wanted to get under the covers, they could change it themselves.

But the neat thing about the way we reimplemented it top to bottom, and not just put a facade on it, was that the underlying mechanism itself was dynamic. By that I mean you could change the amount of memory that SQL Server used while it was running and not have to shut it down and restart it to get it to adopt a new value.

It’s back to the separation of mechanism and policy. We made the mechanism dynamic, we made the policy automated, and, where it made sense to try to capture user intent, we allowed user intent to be captured and actually influence the policy. And that takes architecture, top to bottom. I think the evolution is: You recognize it’s an issue, you put a facade on it, whether it’s a GUI or whether it’s some other code, and then you have to step back and architect the thing, top to bottom, to do it correctly.

Scott: One of the attitudes I ran into back in my C programming days was, “Look, programming was hard for me. It’s hard for a reason. It should be hard. If it isn’t hard, anybody could do it, and that’s the last thing in the world you’d want.” I run into that attitude whenever I look at systems administration and other technical issues, where people say, “Look, this is complicated. It should be hard. You should have to be a little bit of a rocket scientist, otherwise you have no business doing it.”

So one of the complaints is, “Well, yeah, Microsoft’s made it so that any idiot can set up a Web server, therefore every idiot does, and they don’t keep it patched, and it’s vulnerable.” Same thing with the database: “Sure, people could install the database and set it up, but they won’t create their indexes correctly, and they won’t keep it patched.”

So there are certain people who would say you should stop more people at the front door and not let them in. Because if you do let them in, they get farther down the path, thinking that they know what they’re doing, and then they run into a disaster. What would be your response to that kind of attitude?

David: I think it all comes down to, or goes back to, design in the sense that if you have the means within the technology to bring it out to a broader market and let people do the job, then I think it’s incumbent on you to try to figure out a way to do that. It’s a great example and a great point around physical design of the database—how do I design the index set? It’s very difficult.

One of the things we did here is, we designed the Database Tuning Advisor. This was work with Microsoft research where we’ll look at a workload and actually propose indexes based upon the workload, and do physical design that way.

And the way we test that is kind of interesting. We bring databases in, we measure their response against a real user workload. Then we strip all the indexes and we use the Tuning Advisor to rebuild the index set. More often than not, the Tuning Advisor does a better job than the guy who gets paid a lot of money to do it by hand. Of course we can’t see the reporting jobs that are running at the end of the month if they haven’t been part of the workload, but it does a pretty darn good job. So I think it’s a matter of how far can you take the technology.

Another sort of thought experiment here is if you go back to the TV front. Go back to 20 or 30 years ago. Every small town had two or three TV repairmen. The technology is more complex now than it was, but it’s perhaps 50 or 100 times more reliable. So you don’t see the TV repairman in every small town anymore. The TVs just go and go and go until you decide you want a slick new plasma one. I think that’s the way software is going to move as well.

Sean: Obviously, one of the strengths of open source is the community. I think that is one contest they tend to win hands-down, if for no other reason than that open-source projects were driving community long before closed source companies were. But at the same time, I know Microsoft has been making some significant efforts in this area too.

You mentioned in the previous conversation about CodePlex. I knew about CodePlex, but I have not really looked at it with an SQL Server focus before. A brief glance at it showed me some of the activity that is SQL Server related. Talk to me a little bit about what is happening there, how that activity is affecting the product team, etc.

David: I agree with you. I think I mentioned earlier that tapping into the energy of the community is super valuable. I think of a lot of these things in terms of closed loops and how quickly can you run the loop, how fast can you get the feedback, and how fast can you incorporate it. Tapping into the community is a super way to do that. There are lots of examples of closing a loop, like Watson, to have the machine send back error information and actually close the loop and automate things to enable continuous improvement.

As I think about community, absolutely I give the open source guys credit for tapping into that phenomenon first, but I don’t think it requires open source in terms of the development or distribution model to engage the community. I think about community engagement in terms of layers. With the product itself there’s a way of tapping into the community by harnessing their feedback to improve the product over time. I think with the documentation content and general knowledge around how to use the product, there’s a deeper way of tapping into the community. As we publish content for the product, can we allow people to modify it, link to it?

Some of our content could provide a spine for community activity. For example, we could build a collaborative site around the product error messages and allow the community to link their responses into each error message to help one another out when they hit a particular issue. Today much of this happens in forums and newsgroups, but it’s not done in a way that closes the loop effectively. If we did it in the way I just mentioned, it would be much easier for the product development team to review and mine the activity to improve the product going forward.

I think the next level beyond documentation would be extensions and tools. Things that you could build around the product where the core of the product is still in a closed source model but you could open up and distribute add-ons, add-ins, and other sorts of tools and actually build the community around that.

Sean: This has been a great conversation, David. Thanks for taking the time to chat.

David: Thank you.

Bookmark this:

Interviewee: David Campbell

In this interview with David Campbell we talked to him about:

• His background with the SQL Server team and Microsoft
• What about planning, implementation, testing, delivery, and servicing do you think are unique to a product as high profile and critical as SQL Server.
• The Security Development Lifecycle and SQL Server.
• Usability and SQL Server’s Development.
• His experience with OpenSource databases.
• How much the experiences of mySQL and other open source databases have affected SQL Server’s development.

Sean: Tell us a bit about your role at Microsoft with the SQL Server Team and as a Microsoft Fellow.

David: I have been working on SQL Server for roughly 13 years in a variety of roles but my passions are in systems and product development. Currently, I am heading up a team we call “Strategy, Infrastructure and Architecture” – SIA for short but I’ve worked on SQL Server as an SDE (Microsoft lingo: SDE = Software Development Engineer), Development Manager, Product Unit Manager and General Manager overseeing a number of component groups within the product.

Sean: SQL is a product upon which enterprises bet their business. So much of what’s mission critical depends on SQL Server. What about planning, implementation, testing, delivery, and servicing do you think are unique to a product as high profile and critical as SQL Server?

David: In the business world, enterprise database products define “mission critical”. A crash in many line of business applications may result in a disruption of service until the server or application is restarted but a severe crash in a database server resulting in data loss can cripple a business. I learned this lesson the hard way before coming to Microsoft when I worked on database systems at Digital Equipment Corporation (DEC). One day I received a page from our support staff and found that a product defect had resulted in a production line shutdown for a major semiconductor manufacturer. They had to send folks home; some of the physical equipment suffered damage as a result of stopping the production with partially completed batches of material and they reminded me that they were losing roughly a million dollars an hour in business while things were stopped.

Getting to your question might require a little background on SQL Server. Some folks might know that Microsoft hired a bunch of people from the database industry in the early to mid 1990’s to work on SQL Server as Microsoft tried to become a player in enterprise database systems. We acquired a source license for Sybase 4.2 and shipped two versions, SQL Server 6.0 and SQL Server 6.5, on that architecture. We hired a number of query processing experts and they started building a completely new query processor from a clean sheet of paper. I worked on the storage engine team and SQL Server was getting beaten up in the market around this time since we didn’t support row level locking as the Sybase architecture we inherited only supported page locking. I was responsible for the design of the row level locking feature for SQL Server 7.0 and the more I dug into the Sybase architecture the more challenging the design became since the entire Sybase transaction and recovery system was predicated on page level locking and it was very difficult to do a clean row locking design without a number of major compromises. After a number of sleepless nights and arduous design meetings we ultimately came to the conclusion that we would have to rewrite much of the Sybase storage engine to do this feature correctly. As part of this major architecture shift we made a decision to change the on disk format for SQL Server 7.0. This meant customers would have to unload and reload all their data as part of the upgrade to the new release.

So, now we have the context to start answering your original question. We started what would become SQL Server 7.0 with 2 strikes against us: It was really going to be a V1 product with a V7.0 name and we would require every customer to completely unload an reload their data to migrate to the new product. We knew that poor quality would mean strike three. Since we had a number of people that had experience building enterprise database systems we weren’t lacking in design knowledge so the real success factors for the release came down to doing a great job of architecture, implementation and validation. On the validation front we did a number of interesting things that could probably fill a book but I’ll highlight a couple of them here.

Given that we were going to make everyone migrate their data we knew that we need to make the data migration process both highly performance and rock solid. We engaged our sales force to ask our customers to help us by sharing their databases so we could convert them to the new format in the lab. We called this the 1,000 DB challenge and we wanted to get 1,000 real customer databases of all different sizes and complexity so we could run them through a database conversion lab that we created. The other interesting thing we did in this space was to write a playback system. This consisted of a capture utility that would log the customer activity on a production database and then a playback utility that would allow us to play back the actual customer workload against the customer database in our lab. We could play the workload back in “real time” which included the dwell and think time between queries, or “compressed” where we just jammed the queries into the server as fast as we could. In this mode we could stuff a day’s worth of work into the system in an hour or so and really stress the server. We’d ask customers to take a backup of the database that corresponded to the workload and to capture the actual work using the capture utility and then send us the database and capture log. This data allowed us to test the conversion of the database from the old version of SQL Server to the new version and also let us test the new version of SQL Server by replaying the actual customer workload on the new system. Later on we started to capture query performance of replay on the old vs. new version of the product to find performance bugs.

The next interesting thing we did on the validation front came from Don Slutz, a long time database veteran that was working in Microsoft Research at the time. He wrote a program he called RAGS that was really a model based testing system that used the SQL language grammar and the schema of an existing database to generate bizarre, but syntactically legal, SQL statements and feed them into the query processor. Basically, he married the state domain from an existing database schema with that specified by the SQL grammar and was able to probe all the dark corners of the search space programmatically. He wrote an MSR technical report that is available on the Microsoft Research we site. The way this played out was pretty interesting. At first it was pretty easy for Don to crash the query processor. So he filed a bunch of bugs, the developers fixed a bunch of bugs then Don ran it again, etc. After a couple of iterations of this cycle RAGS needed to generate some pretty ugly queries to crash things. You’d wind up with these 5 page SQL queries that looked like random gibberish but was legal SQL syntax and the query processing team would spend a bunch of time figuring out what the query was supposed to do and then figure out why it crashed the system. Later on Don used RAGS against different versions of SQL Server and other database products generating queries over equivalent schemas and comparing both the results and the performance of the different products across a wide range complex queries.

I could list more if you guys want to write a book…

Sean: It’s hard to imagine a product that has higher security requirements than a database server. It has to talk on the network, and it has to be able to store sensitive information. Since the introduction of the SDL, SQL has seen a dramatic reduction in vulnerabilities. How does the SDL play out on a day to day basis? How does it affect your architecture?

David: Great question! To fully appreciate the change you have to understand that 10-20 years ago there was very little widespread security knowledge in the software development world. In some sense the environment didn’t require it; most systems weren’t interconnected and remotely accessible. This meant that security breaches required typically physical breaches and people had a model in their head for physical security. It was easier to understand locks on doors than buffer overruns. What is interesting about the SDL and SQL’s evolution is that we’ve gone from a period in which security review and validation was done mostly after the coding was done to today’s world where we formally design security threat models as part of the design process before writing code. We also have a great training program in place that everyone touching the code needs to take. We also have refreshers that developers must take for emerging threats. Additionally, we’ve developed a great set of static code analysis and run-time tools to avoid and detect potential security issues. In 2002 we had a “security push” where the entire development team stood down and reviewed every line of code in the product. We did a smaller push in 2004 for SQL Server 2005 and with our current release we have truly integrated the security best practices into the development process and don’t need a separate security push as security is simply part of our day to day process. One challenge in the security space is that the threats are constantly evolving so we can’t rest and, as long as the bad guys are learning new tricks, we need to up our game and having a process in place for rapid mitigation in the event of a new threat or vulnerability.

In terms of how security has affected our architecture we are much more mindful of the threat environment each line of code is executing in. For example, there is a small amount of code that performs the initial client authorization before allowing a connection into the server. Since a remote client executes this code pre-authorization, any remote code that can access the port over the network can execute this portion of code. Architecturally, we strive to keep this code to a minimum and it is very thoroughly reviewed. Similarly, the security base is designed in a layered fashion so you have smallish amounts of thoroughly reviewed code providing services that other aspects of the security system are built upon.

Sean: How would you respond to a statement like, “If you really wanted to make SQL secure, you’d forget about the SDL and just open-source it so that many eyeballs could look at the code.”

David: I think there is some benefit to having more eyes on a piece of code. There is benefit from each person’s fresh perspective, benefit from varied knowledge, etc. However, eyeballs alone are not nearly enough. The SDL process evolution has been really interesting in that we have changed the culture, habits, and processes of every Microsoft developer in a way that is much more effective than having many otherwise competent programmer’s looking at the code. With SDL we have many eyes looking at threat models, many looking a the design of a security feature and ultimately, many looking at the code following a pretty rigorous and proven process. I think there’s sufficient independent objective evidence to say that SDL is working for us. If you have a few minutes go to the National Vulnerability Database at nist.gov and check out the vulnerability reports for SQL Server vs. other databases. I’ll warn you, if you try to search for Oracle flaws they are hard to find since every major and minor release of the Oracle database server is listed separately so you need to do a little aggregation to get the real picture.

Sean: It’s one thing to design powerful functionality, it’s another to make it easy to use. Talk about how Microsoft insures not just functionality, but usability.

David: OK. Now you’ve hit on one of my personal passions. I believe that many technologies go through stages of evolution as they mature. I define three major phases – nascent, developing, and refined. There are many examples that can serve as a lesson for software developers.

Consider televisions; thirty to forty years ago when TVs were a nascent technology you almost needed to be a technician to own one. Certainly, you needed to know how to take the back off and pull the tubes out so you could go to Radio Shack and test them when one of them went bad. Furthermore, there were knobs on televisions that were there solely because the technology hadn’t matured to the point where they weren’t necessary. I often ask audiences how many people miss the horizontal and vertical control knobs on their TV and we’re getting to the point where many in the audience don’t know that these knobs even existed to keep the picture from rolling and waving in early TVs.

Twenty years ago TVs entered the developing age as they became fully solid state. They were much more reliable and the technology developed to the point where many of the knobs disappeared. In this phase they were good enough for the masses. You didn’t need to be a technician to own one but it was nice to have one in the neighborhood. Frankly, I sort of think this is where PCs are today.

Today’s TVs are refined in that the user’s control surface captures the user’s intent rather than exposing the control surface of the underlying technology. For example, instead of fiddling directly with color temperature, saturation and hue to adjust the picture, my TV has a control that asks me if I want to watch sports, movies, or regular programming and adjusts the color parameters accordingly. Furthermore, some TVs are aware of the operating environment such as ambient light and compensate for that. You can do this same thought experiment on automobiles, microwave ovens, etc. When viewed through this perspective, most system software still has a long way to go to become a refined technology. Of course, there are cases where these maturity phases ripple and repeat through a single technology where advances happen in waves. I think my new Smartphone is an example of that. It’s way more capable than my previous cell phone but I never had to reboot my earlier one.

So, how are we doing on SQL Server? One simple example is the work that we did in the database engine in SQL Server 7.0 when we got rid of many of the knobs and made many of them self-tuning. SQL Server 6.5 had roughly 100 knobs whereas SQL Server 7.0, which was much more complex in many ways, had roughly 20. Many people thought that more knobs meant more control but, in reality, we found many systems were performing poorly in the field due to mis-configuration. We classified the knobs into those that should simply take care of themselves – things like the number of locks the server could allocate when it booted, the number of hash buckets in the cache manager, etc. These were our “horizontal and vertical control knobs”. For other knobs we set them up where the database server managed them by default but if an administrator wanted to impose constraints he could. The amount of memory allocated to the server was in this category; by default, SQL Server manages memory in cooperation with the demands in the operating system but you can set low and high watermarks if you want to. In other areas we used control theory and feedback loops to have the system adapt dynamically to the environment and control things based upon instantaneous system response. The adaptive systems work we did was really interesting in that existing control knobs were often static; a good example is “sort memory”. In many database systems before SQL Server 7.0 you set aside a portion of memory for sorting to perform queries and build indexes, etc. During times where you weren’t building indexes or didn’t have queries that required a sort that reserved memory was just laying fallow and wasted. Further, if you had a sort that needed a little more space than what you had reserved it would spill to disk because the sort wouldn’t fit in the reserved space – even if there plenty of memory unused elsewhere in the system. In SQL Server 7.0 we created an internal memory broker that could use server memory for whatever purpose made the most sense over time so things like the procedure cache, workspace, sort, and the buffer pool all cooperated to use memory in the most efficient manner over time. The result was fewer knobs and a more efficient system.

This was great work but we ran into a situation where we were ahead of the market in many respects. DBAs were worried we were going to put them out of a job. They thought they were getting paid, in some part, to respond to their pager at 2:00 AM because a big batch job failed because they hadn’t allocated enough lock blocks for the server. Our competitors also used this maturation against us – things like, “How can SQL Server be a real enterprise database system – it only has 20 knobs and our system has 500!”. What’s interesting is that if you look at the major database systems they have all made major investments in self-tuning and ease of use.

Market dynamics also demand that we make these systems easier to use and self managing. As an industry we’ve expanded database deployment from an installed base that was likely measured in the small 100,000s of units 20 years ago to one that is likely measured in 100,000,000s of units today. They had better be easier to use than 20 years ago!

Scott: How do customer needs and requirements make it into the planning process? How do you handle situations where the customer is asking for the wrong feature? (The customer asks for a setting so they can tune X, and you realize that if a subsystem was redesigned, they wouldn’t need to tune X)

David: We have had many situations where customers have asked for features that they have seen in other systems that didn’t make sense for SQL Server. Often customers ask for performance features that other products have that may provide a large advantage in those products but, given the way that SQL Server is architected, these same features may provide little to no benefit on our architecture. One example is raw device and partitioning support. 20 years ago many UNIX systems didn’t have advanced I/O features such as the ability to avoid the file cache, scatter/gather I/O or great asynchronous I/O. In fact, many early UNIX file systems had 32 bit file offsets so the maximum size of an individual database file could only be 2 or 4 GB in size. NTFS, in contrast, supported 64 bit file addressing, great asynchronous I/O, and the ability to do unbuffered I/O from the start. So, whereas other products needed separate partitions over multiple files with an I/O thread per file to simulate asynchronous I/O – SQL Server didn’t need any of this. This didn’t prevent customers from asking though. Of course, once you get up to very large databases, partitioning makes sense for a number of reasons such as the ability to physically manage large tables in index in smaller pieces but the point is that SQL Server didn’t need partitioning to get I/O parallelism in the same way some other systems did. We had to educate many customers on these points so they understood how our architecture achieved the performance that other systems did but through a different architecture.

I’m also mindful that control doesn’t always represent progress and often simplicity is the best approach – especially when it affords an opportunity for the software to do a better job. I think one good example involves a feature known as “tempdb in RAM”. Prior to SQL Server 7.0 you could allocate a region of RAM to hold the temporary database which is used for scratch tables and intermediate query results. In certain environments, placing “tempdb in RAM” could provide a significant benefit given the way that SQL Server 6.x was architected. Unfortunately, it was easy to under or over allocate the amount of memory and wind up spilling to disk or paging excessively. Another point is that the amount of RAM allocated to tempdb was statically determined; once set you needed to reboot the server to change it. Since the amount of space needed for tempdb varies depending on the workload, the optimal caching strategy for tempdb is dynamic. We removed “tempdb in RAM” in SQL Server 7.0 and did a number of other optimizations under the cover to better manage tempdb pages in memory so the actual customer result was much, much better across a wide range of scenarios for SQL Server 7.0 but customers who had improved their performance 2-3x by using the tempdb in RAM feature in SQL Server 6.5 screamed loudly. I finally wrote a long mail and included some experimental results that proved that the new approach was better but I still received hate mail for several years after that decision.

Scott: Areas like the developer division have strived for greater transparency. In open source, all development and decisions are transparent. Talk about how SQL Server views transparency during development, and how you have to balance expectations from customers who want full transparency, vs. not shooting yourself in the foot by disclosing early and giving a closed source competitor like Oracle an advantage?

David: You touch upon a very real challenge. SQL Server has matured to the point where we do a very good job on the fundamentals and, as a result, it’s more important that we listen to and work with our customers to continue to produce a product that helps them better run their business. We’ve talked mostly about the core relational database engine thus far but today’s SQL Server includes data analysis, data mining, reporting, enterprise class ETL, etc. The solutions we deliver in this space touch a broader range of customers and the pace of innovation is much faster than that seen in the core relational database engine. As a result, we need to be much more in tune with our customers to produce the right product. We’re making some real progress on including key customers and experts in our design process. For some of our more complex SQL Server 2008 improvements we’ve done joint design with key customers and MVPs and they’ve helped us make many of the tough scenario and design tradeoffs required to deliver the right feature. Done well, this reduces the need for iterative field testing to get solid feedback on new features.

Disclosing early is a real risk and, yes, our competitors do listen and respond. It’s funny, they pay much more attention to what we say now than they did 10 years ago.

Sean: What experience do you have with Open Source Databases?

David: Yes, I have experience in open source and certainly follow the key open source databases from the perspective of technology evolution and adoption. I don’t look at any source code from the open source databases to avoid any potential IP issues.

Sean: What do you think are some things that are easy to accomplish in a closed source model that would be challenging in open source?

David: I think one thing is something I would call “consistency in the large”. This isn’t necessarily easy in the closed source world but in a coordinated engineering environment you can align things in a way that lead to a degree of consistency which creates customer value. For example, the products within Microsoft’s Server and Tools business have a set of “Common Engineering Criteria” (CEC) that we all follow. The criteria include things such as common processes and UE content that lead to a more uniform customer experience and aligned features, such as a Best Practices Analyzer or having a management pack for our management tools when we RTM. This sort of broad consistency doesn’t happen organically and is one of the challenges that large scale open source efforts have to wrestle with.

: How much is SQL development and features informed by things like MySQL and DB2? Are there features in SQL that were inspired by these products?

David: Certainly; and it goes both ways. Things like persisted views have been done by all major database players. I can’t recall whether Oracle or IBM did it first but now SQL Server, Oracle, and DB2 all have a form of persisted views with varying degrees of updatability and query matching sophistication. Oracle was the last major database player to have a credible fully cost based optimizer. I would say that SQL Server led in the advancement of real self-tuning and ease of use. Unfortunately we did it in 1998 and tried to sell it to a market that didn’t understand its value and IBM did a great thing (for IBM) later on by coining the phrase “autonomous computing” and selling the value. I don’t think MySQL has contributed too much yet to the big 3 but I do like MySQL’s notion of installable storage engines with implementations optimized for various scenarios. What’s interesting is that SQL Server 7.0 was architected to support this concept with the OLE-DB interface between the relational and storage engine but MySQL has really gotten value from the concept.

Sean: If you had the opportunity to borrow more from the development model of the open source community when it comes to SQL Server what are the top couple of things you would like to bring over to SQL Server development that you see out there today in the Open Source community?

Scott: In general I like the notion of tapping into a large development community’s collective energy. Different people have different passions and, if you can harness the energy effectively, it can lead to some interesting results. For example, if someone gets excited about adding a specific feature or fixing a particular bug they can often just make it happen. Of course, one challenge in the open source world is maintaining architectural and feature consistency. Smart people don’t always agree on what’s important for the customer or know how a particular feature should be implemented in a way that maintains the system’s design tenets. I think a key aspect of great design is in providing the most value with the simplest and most intuitive implementation and user model. The open source project maintainers have a very tough job keeping this in check.

Another aspect of the open source world which is powerful is the ability to generate an ecosystem of tools and add-ons around a core technology. This is an ecosystem effect that may or may not require open source access to the core technology. For example, imagine if MySQL were a closed source project but that it had a vibrant open source community building installable storage engines for various scenarios. We are starting to do some of this with SQL Server through Codeplex.

Sean: How much do you think SQL Server’s development has been impacted by Open Source development efforts in terms of how they build community around various database products?

David: Enabling the community to help the community and giving the user base a direct voice to the development team is one of the most exciting and powerful concepts to come out of the open source development model. We have learned a lot from this and in our product review meetings we regularly review input from our community including how they vote on design change requests. Rather than us guessing or asking a small sample we can now reach out to our community quickly and efficiently. Frankly the Internet and access to our community have enabled us to produce a much better product. Things such as Watson, SQM, and our community product provide us direct feedback that allows us to respond in ways we couldn’t even imagine 10 years ago.

Scott: Give us some understanding of the structure of the SQL Server development team in terms of some high level estimates in terms of the number of testers, developers, number of machines in the test lab, length of time it takes test suites to run, etc. Just so folks can put the SQL Server development effort in perspective to other efforts they are familiar with.

I do not have exact numbers in my head but let’s just say we have 100’s of developers, 100’s of testers and probably 10,000 machines in various test labs. We literally have millions of individual test cases and the test system is highly automated. Many of our test machines are in offsite data centers and we’re able to connect to and control them via IP KVM. We have evolved our test methodology to include much more model based testing rather than individual test case generation… One interesting thing is that we completely revamped our development methodology between SQL Server 2005 and the current release. This was a huge cultural and process change and we have learned a lot from this experience. Perhaps we can chat a bit out this next time.

Bookmark this: