Interviewee: Britten Martin

In this interview with Britten Martin we talked to him about:

• The role of support inside Microsoft.
• A walkthrough of how a support request is handled.
• The ways in which the outside community can interact with the internal support processes.
• How a product is supported differently in beta than when it’s finally released.
• How a bug eventually become a fix in the product.

Scott:  Britten if you could just introduce yourself for the record. That is usually a good place to start, and then I’ve got an initial question after that.

Britten:  Cool. So, my name is Britten Martin, I am a group manager within CSS which is Customer Service and Support. I’ve been with Microsoft since 1999 where I came in as a support engineer supporting Windows 2000. So I have been a support engineer, I have done some work with our outsource partners in a partner technical lead role, then I moved into frontline management and did that for about three years. In my current role I am a manager of managers, I have an organization of about 75 people and I am the global sponsor for System Center Support, which is your SMS and MOM products.

Scott:  So talk a little bit about what falls under the scope of support inside of Microsoft. I mean, engineering rather has its domain, marketing has its domain: what is support defined as and what are kind of the responsibilities roll up under that inside Microsoft?

Britten:  That is a good question. It really depends on which segment of support. It is a great clarifying question, Scott, because I am within Commercial Support which is the enterprise-based support, your back‑end, your servers, many of your big customers that buy dedicated support contracts, the government, many of the major companies around the world. We also have certainly have a consumer end which is your home user running Vista, Office, also Xbox, Zune. Any product that you can think of that is released by Microsoft has support. So really what it is, the thing that separates, I think, Microsoft from other companies in that, is that no matter what’s out there, you can pick up the phone, call 1‑800‑MICROSOFT, tell them what your product is, and you’ll get through to someone to support your issue. Although it certainly depends on the customer segment that you are in, who you are routed to, where you go and how much it costs.

Scott:  Sure.

Britten:  In Commerical Support, we support customers who have technical how-to questions about the products and certainly those who are getting error messages.† Also, if you need help setting it up, if you need more consultative type services, we do that as well.

Scott:  OK. Then one other thing I was somewhat curious about too is ‑‑ one thing I have always been curious about I guess – is how a problem works its way through the cycle.

Britten:  OK.

:  So, you start out when the phone’s ringing and you have one customer, two customers, and three customers: they seem to be having a similar issue. How does it work its way through the process from the technician helping them on the phone, to maybe later on deciding it needs a KB on it, to maybe a hot fix, to maybe being rolled up in a Service Pack?

Britten:  That is a good question too. OK. I will speak in today’s terms because major things have shifted over the last couple of years. There used to be a lot of process behind just what you were saying, which is, you know: "When do we do a KB? How many issues before we make a hotfix?" Nowadays with the advent of blogs, which is kind of, what you are doing, you know, what this interview is about, in support we have become a lot more proactive with getting immediate information out to our customers. I will give you an example from my group, within System Center: last week we had an issue with SCCM, the new version of SMS. Just like you described: one call, two calls, three calls. All of a sudden we are like, "Wait a minute, there’s something going on here!"

So we got some information together, dumped it out into our blog so it was out there. On the back‑end our support engineers, who are your front‑line resources, are working with the escalation team, our escalation engineers who do the debugging, who look at source code, who work with the product group. They are debugging the issue, and they get it down to, "OK, this is clearly a bug." At that point, that escalation engineer has a hotline per se to the product group, they pass along all of their research, they are reproducing the steps, reproduce the issue, send it on to them, all the documentation.

Then it is kind of in the product group’s hands to decide, "Is this something that we need to fix now. Do we need a hotfix, is it just something that we can put documentation out." You know kind of that form of decision making process. In this case it was a major issue, it was a deployment blocker, huge customer pain, so they released a hotfix. Then once again, depending on the urgency, is it a publically available hotfix or is it something that you have to call support to get a hold of? In this case, we knew this was a major issue so we made it available out on Microsoft.com, put a KB article out there, and linked the hotfix to it. Therefore, customers could immediately just go and download that hotfix.

Down the road certainly that’s going to be in the Service Pack 1 or similar, so you can pretty much figure for all of our major products, any bugs that we fix, or any hotfixes we release up until, there’s a cut‑off date, will make it into the Service Pack.

Scott:  And then talk to me a little bit about the timeframe, right, between the time where maybe the first few calls came in, until the time that hotfix and that KB article are out there.

Britten:  Well, once again it is very, very dependent upon the criticality and the widespread nature of it.

In this case from first call to KB/hotfix was about five days. In addition, some of those delays ‑‑ they had identified the code, they had written a hotfix within about two days, but there were some delays, you know, going through legal on official KB review, and getting everything up there and replicated. So from beginning to end, about five days.

Scott:  Yeah, that is fast. I do not think most people would ever think of Microsoft turning something around that quickly.

Britten:  Some times, I think Microsoft is seen as very slow to respond, but in reality, itís often not the case.† †If it is truly a deployment blocking issue or something that causes widespread customer pain you will see a quick turnaround on it and you will definitely see the documentation out there in a timely manner. I have been here eight years, I have seen issues where it was tough to figure out what the fix was, what the resolution was. Where it was something that would hide itself into multiple components so it was a complicated fix to write, it takes a little longer. However, if we can identify it, if we do a good job on the support side with identifying it, providing reproduction steps to the product group, you know, they can turn it around quickly.

Scott:  So one question then too, would be, what does the triage process look like if after reproducing the bug you quickly realize that it has security implications versus something that’s a deployment blocker, which might be the next layer and then further down which are just kind of usability or general problems in using the software?

Britten:  Yeah, I think, once again when you are running into security type issues what you get with that is even more resources thrown at the problem. We have security aliases internally, they have product group members, and they have certainly a high-level support presence as well, that if something comes out we can fire it up and quickly. There †is a whole process that is enabled. Let us say there is a new Sasser or Blaster or any of those. We have processes in place that will quickly call together members of support, product group. We have open lines 24 hours a day. It is kind of the war room type mentality.

It has product group members on it. It has support members. So people on the field can call in and say, "Look, I have got an issue with this bug or with this security hole and just want to make sure I have the up to date information". So they can call into this grid, they can get the information.

On the back‑end, it is hard to say how long it is going to take, depending on the fix but you are going to have even more resources available. You will see a fix as quickly as we can get it out there. After the Sasser/Blaster/Nimda pain, we really turned the corner not only on product development but also on the support side to ensure we could have a quick turnaround.

:  What do you think the level of community involvement is in the support process because obviously that is a huge piece of the open source model. Right down from many eyes looking forsecurity bugs to a very well populated mailing list where there is a high discussion level, traffic, and things like that. However, at the same time with the closed source support process while the team is staffed potentially adequately with full time employees, there are general concerns out there with a closed source model in terms of how much the community can get involved.

So what are the avenues the community can actually affect the support
process at Microsoft and/or see what is happening under the covers of the support process?

Britten:  Yeah, I will tell you that it has really been something we have been working on for the last few years. The product groups and the support teams have been trying to find ways to tap into that community model because there are many folks out there that do our support day in and day out with customers that have a lot of knowledge that could be helping the broader community.

So, one of the things that we have is the MVP program, the Microsoft Most Valuable Professional. One of the things that these folks do is to actively take part in the public newsgroups for our products. I can point to a couple of products ‑ Small Business Server is the poster child for that. It has a very strong community. Customers can go up to the Small Business Server forums and post problems and the MVPs will quickly get involved to help resolve the issue.

There is a lot of energy around building a stronger MVP community and empowering the MVPs to help solve customer issues. Many of the product groups have CPE, Customer Partner Experience, teams, that are really focused on tapping into that partner community.

Another example from my world System Center Essentials, or SCE. We actually launched support for that on a complete forum based model, where we had engineers from CSS monitoring the forum, and we worked to build strong partner participation. If the partner could not answer the question, or did not answer the question within 24 hours, someone from support would jump in. We are trying to grow more community presence around support.

Globally, there is more partner presence out there as well. A good example is SoftGrid or Softriciy, a recent acquisition. In Europe, before Microsoft took over, there was not really a ton of dedicated support. It was all done through the community. Therefore, we have been looking and trying to learn from them on how they involve the partners, and how we can keep that type of interaction going. I know there is a log of energy around this and it is a
huge initiative.

Sean:  Well, one question which is really back to the second part of the original question which, is that a lot of the product groups in Microsoft are being affected by the open source model.† For example recently we talked to Shawn Burke who has been instrumental in getting the source code† for the framework out the door. How much of push to learn from the open source community is affecting the support team in general? For example could you see a day where the status on bug fixes is more open and less a black box until the actual fix arrives out on the web?

Britten:  So thinking more about ‑‑ I am partner, I am out there. I find a bug or a hole, I put it out there, and really being more transparent if that is going to be fixed. Is that kind of…?

Sean:  Yeah, exactly. I realize it’s not going to be a plate glass window ‑ it never is. But at the same time I think a lot of people will look at support right now and say, once I send in my support request I don’t really have a lot visibility of how they are stacking it, ranking it, prioritizing it, and where it goes, until maybe I get an email that says you have been deferred. Right?

So I am just curious, I know a lot of the product groups specifically are trying to expose a fair amount of their development processes, so I was curious if maybe that was making its way over to the support side.

Britten:  You know it is a fair point. Let’s say you are running a beta version of the product and you are involved in the one of the official beta programs, such as TAP ñ our Technology Adoption Program.† †The beta version is fully supported and you can actually go and directly file a bug. In addition, I know that they try to make a lot of that process transparent, so you know if the bug is a duplicate. Is it being fixed? Is it being deferred to the Service Pack 1…whatever. So actually, I know on some levels we do that.

On the support side, I think we make good use of our blog. There is a huge blogging effort, because there is a lot of knowledge within our support engineers and within our internal databases. For one reason or another that has never been made public. So I think we try to do some of that as well. But I can see the point. I mean maybe someday and maybe that some day is not too far away where you can log in and say, here is a bug that I submitted. Alternatively, even better for Windows, here is the bin of things that are going to be fixed in the next two weeks, months, whatever.

Sean:  Exactly

Scott:  Well, and so you know not every support request anymore comes in through a phone call, right? There are things like the Connect site. There are things like Windows Error Reporting services. I guess if you would not mind talk for a minute about some of the different channels that, a problem can kind of come in through and… Go ahead.

Britten:  No, no, it is fine. So today, and if I use Americas Commerical Support as a kind of a measuring stick, somewhere around 85% of our support issues come in via the phone ‑ very traditional means of operation. We are really trying to push more of our online type offerings. One is online submission where you can go up and create your issue; you can attach data, submit it to a support engineer. That support engineer reviews that data and calls you back, hopefully with an answer.

The Connect site is a lot more beta focused at this point. I am not an expert there, but that’s where you get into many of the cool things where you can actually submit bugs and all that. We also have the managed forums, which we talked a little bit about. The managed forums are a completely different avenue. Most of those are staffed by Microsoft support engineers as well as product group representation and the partner community.

So you have an issue with Windows, you can go up to a specific Windows forum, post a question. You might get an answer fromsomeone in the product group, a partner, or a support engineer. Therefore, you have those as well.

On the consumer side, they do really cool things with automated responses. Therefore, you submit an issue and it checks your data, checks your error log to see if it is a known issue. Before you would hear back from the support engineer, you might get an automated response with, you know, try these two things. If this does not work respond back and the engineer will contact you, then as you said, you had the whole Watson or error reporting. The error reporting stuff is very cool, because regardless of bugs filed or anything else that goes on, all the product groups take all of their Watson or their error reporting data and they make sure that as they look to roll out service packs, with new versions of products especially, they want to handle those top 15, top 20, top 50 Watson issues. They want to make sure that they are knocking those off.

Therefore, it is an interesting mix of many different avenues for customers to submit data, and in some cases, a Watson log, the error reporting, customers might not always think they are really contributing to the next version of the product, when in essence, they truly are.

:   I realize, too, that you are right, that is largely a mechanism for when a product is in Beta. I guess, I have two questions, but one is a follow up on this thread initially. Talk a little bit, I guess; about how a product is supported differently while it is in beta, versus once, it has been released?

Britten:  That is a good question. We have a few different types of beta opportunities for customers.† There are more formal relationships, such as TAP (Technology Adoption Program) and RDP (Rapid Deployment Program).† With these formal programs, customers agree to deploy this product into some type of production environment. In return, they get access to submit bugs, they get access to product group members, and they get
access to dedicated support engineers.

From within support, let us say we have a new version of Windows, we might have three or four support engineers in the US dedicated to this beta support. These engineers get a hold of the product early, they get access to product specs, they visit the product group, and they may even get to go out on site to customers who are deploying this product so they get some real world hands on experience.

Now, that is for the structured beta program. If you’re not a customer in one of these programs, you likely can still download beta copies of the software. Even in these situations, there is support out there, through the forums that we talked about before. Once again, you probably get answers from those same beta engineers and some of the product group folks. You just do not have that one-to-one interaction in a lot more, you know.

Does that make sense?

Scott:  Yeah. Yes, very much so. One of the other things that ‑ I will kind of circle back to something you said earlier – If it needs code to solve the problem, you know, if it is not just configuration file that was messed up or any one of the myriad of things that could cause a problem. You know, some registry key got corrupted or whatever. However, it actually needs the coding hot fix, you could hand it over to the engineering, send it over to the product team and they triage it, compared to other stuff.

I mean, have you ever run into situations it seems like this would inevitably happen where they’ve got their engineers dedicated to building the next version of the product, and meanwhile, there’s an endless list of bugs in any complex software that they could be tasking engineering resources with. In addition, it seems that they kind of have to make that call about pulling people off engineering new features to work on a hotfix or something like that.

Britten:  Yeah.

Scott:  And how does that process of triage work? Because support is kind of on the front line with the customers, it seems like sometimes you guys might have to go back to them and say, "Um, you know the time frame that you’re looking at fixing this in probably isn’t really going to work, so think about it again."

Britten:  Right. In addition, I will tell you, it is really dependent upon the product group. You look at something like Windows. Windows has a whole team, and you hear it from time to time called Win SE. ,Windows Sustained Engineering. They have a whole team dedicated to current version, and whatever supported versions are out there, on the hotfix front.

With some of the smaller products, you may run into situations where they have to pull developer resources off the next version to fix current version or legacy version problems. However, I tell you, in my time here, I have never seen that as being a blocker in getting something fixed. If there is truly an issue that needs to be resolved in the current version of the product, especially if its security related, you can be assured that they will find someone to handle it.

However, as a general rule of thumb, all big product groups have sustained engineering, folks that are dedicated to current version problems, or legacy version, so it is not necessarily that huge resource problem that you would think.

Scott:  OK. Therefore, many times, they are not pulling people off features. They have people who are dedicated just to the maintenance and hot fixes.

Britten:  Right. That is your sustained engineering folks, for the most part. As I said, I think you would run into some smaller product that might not be in that same boat, but the major ones have sustained engineering.

Scott:  It seems, too, like support, it is a little bit of a difficult situation, because, I guarantee, if I call with any bug, it is a critical show‑stopper. The world will end if it does not get fixed. So how do you kind of manage customers, who are generally going to feel the importance of their bug is high, but Microsoft does not necessarily determine it so in the grand scheme of things?

Britten:  Yeah. I tell you, our people are experts at conflict management and really setting customer expectations, and certainly the empathy piece as well. I mean, you have to be empathetic with the situation, but you also need to be very transparent and upfront.. It is saying, "Look, here’s where the bug is at this point. It is something that may or may not get fixed. Here’s the timeline."

With many of our customers in the Commercial space, these customers also have account managers that are Microsoft employees that really serve as the liaison between support and the customer. Therefore, we do a lot of work with the account managers to help us on the customer maintenance side of things, when maybe the messaging isn’t as good as we’d like it to be.

However, frankly, all major problems end up getting fixed. They might not get fixed in the period the customer would like to see it, but, between a hotfix, the next service pack, and the next version of the product, all issues do end up getting resolved.

Once again, it is really just dependent upon how many customers are having the same problem. Is it something that other customers could run into? In addition, really prioritizing, like you said before. I mean, the product group, while they have a sustained engineering team, has a limited number of resources, and they need to prioritize what are the big-ticket items that need to be worked on.

However, that skill of conflict management is one of the key pieces for successful support engineers, because, every day, when we answer the phone, when we respond to an email, that is generally that networking administratorís worst day of the year. The servers are down. They are catching heat from their management.† Something is going on, they are being yelled at. Our engineers deal with this on a day in and day out basis and as a result, they get very good at handling the heat.

:  So one question I am curious about is, how do you deal with it when the support engineer knows that it is really a usability issue and not a bug, sure there is a sequence of phone calls and bug reports, but it is pretty clear that this isn’t a bug, it is more of an issue of usability. And it might be in the land of the undiscovered feature right? We were told, if I remember, some person in the office team told me that something like 90 percent of the feature requests for office, are already in the product, but nobody knows the feature is in there. So how do you people educate the product teams and what does that flow look like? How do you finally classify that, "OK this isn’t really a bug, it is an issue of usability?"

Britten:  That is a good question.

Sean:  Because obviously the product team, like everybody else is blind to what they build, right? Of course, it is useable. Because I built it, I mean every developer has fallen to that problem and we all have seen some pretty horrible UIs over the years. So how does the support team educate the product teams on this point?

Britten:  Yeah. In the past, if you back up five years, 10 years, support was as you said an island. We were out here, taking all the heat and there was limited interaction with the product group. About five years ago, things drastically changed.† we started doing what is called supportability in our world. We review what are the top issues, how long does it take us to solve those issues and how much do they cost the company for us to support these issues?

Therefore, we came up with a list of the top 10 issues. We then present that back to the product group. Now the interesting thing about that is the product group actually funds us to do support for their product. So, when they see that a problem in their coding is †costing them $800,000. they to take some notice , and say wow OK, what does right look like, what are we missing here?

Sean:  Well, just a quick follow up to that. What are some examples of things where you feel like you touched the product, and you made a real delta that the customers ended up bettering from, but at the same time you folks probably would never personally got credit for it

Britten:  That is right. I donít have a list in front of me but DNS is one that consistently scored high for Windows 2000 and even some into Windows Server 2003.In addition, there was so much push and so much interaction that there were some major changes that happened later in 2003 and into 2008 that have drastically changed the way DNS works.

Another one I can give you that clearly had all the supportability people driving this was terminal server licensing. Terminal server licensing, when you go back to 2000 and once again, kind of into 2003 with a complete and utter nightmare for something that should be very simple. Consistently that silly process ended up top two, top three on our supportability list for Windows Server. And some very small tweaks based on advice provided by support completely changed this, dropped out of the top 10, out of the top 20 in support issues and saved the company a lot of money.

Those are two examples of technologies I know we beat on. However, on all the products out there, SQL, Exchange, Windows, SMS, MOM, they had the supportability groups that review the top 10 issues. In fact, I was just in Redmond last week, looking for the SMS side and they do cool things with it. I mean they look at top 10 issues for previous version of the product, look at the top 10 issues for current version, and review whether or not they fixed the top 10 call generateors? If not, what do we need to do to make sure it is in the next version that we do? Support is a huge data mine that we are just starting to realize the power of all the data that we have.† We are a great voice of the customer.

Sean:  One quick follow up to that and then Iíll turn it back to Scott. I am curious because you mentioned that five years ago there was some type of a change, right? In addition, all of a sudden, you folks were a lot more integrated. What drove that? In addition, I realize with all changes ongoing, there is probably stuff you can say and stuff you cannot, but I am sure somebody reading the interview is going to be like, "Wait, wait, and ask him that." So what drove it at a high level?

Britten:  I think that is fair. I wish I had some great and gory backroom drama to share, but if it happened, I just donít know the details.† In my opinion, you know, part of it was just the fact that, wow, especially on the Enterprise side, Windows Server 2000 started to really turn the corner and become a huge enterprise related product. People were out there buying it and implementing it and someone on the CSS side as well as, I can guarantee you, the Windows product group was like "We need to tap into this.î

Around this time, I was a support engineer and I suddenly realized that the product group was really listening.† In all businesses you started hearing about some monthly conference calls where we were there to provide feedback on top issues.

It was a time when we were moving from being just a consumer, home user company to being a legitimate player in the enterprise. In addition, there was a lot of pressure at that point to make sure these products were just better, and I think someone realized "Wow, CSS has a lot of knowledge out there; we need to tap into it."

Scott:  Well, how much now do you guys get involved in product development from the standpoint of, you guys are looking at the product, while it is in development and saying, "You know what, if that error happens, there is no way we will able to trace it back and figure that out. We need instrumentation here, we are going to need these things, spit it out to a log file, we are going to need these kind of support features built into the product to be able to kind of quickly
and officially troubleshoot things."

Britten:  That is a huge piece of our beta involvement. I know we talked about it before. For each beta, we dedicate engineers from our support teams to the pre-released product. Well, one of the way they make an impact is by providing that ongoing feedback to the product group. They are not only working with beta customers, they are not only answering newsgroups, but they are also messing with the product itself.

Therefore, they filed their own bugs. They have supportability war room meetings where they say, look there needs to be some logging here, or this error message does not make sense, or if this component is blowing up, I need something to trace it back. That is generally our involvement.

We do spec review. So many of our engineers have been on betas before, they are hooked in with the product group contacts. As the next version of the product is being planned, our engineers get involved to review specs, and just generally throughout the beta process from the beginning of our involvement through RTM, we are providing that ongoing supportability feedback.

So not only are we always supporting our customers, but we also look to help the product group on the supportability side. It is like you said, I mean, they are focused on here is a new component, here is a new thing that customers can use, maybe not always focused on how would I support that if it breaks?† Letís be honest Ö if we donít provide our feedback to the product group on what we need, it just causes more pain for us once the product is released.

Scott:  One other one question is, so a new product is coming out and how do you get support folks up to speed on it? In the OpenSource model the development team in some cases is the support team and there is a tight integration between that team and the open source community so there isn’t really a need to educate the "support team" in this case. So how do the support engineers come up to speed on a given product?

Britten:  This is another benefit of our involvement on the beta side.† †We get access to the source code. We get engineers playing with the product. We get that constant interaction with new builds of the product as it comes out. As a result, months, in some cases years, before the product actually ships, we have experts on this product.

Now, our experts certainly might be great technical writers, but we have a group called Global Technical Readiness. These folks are really program project managers who will say, "All right, we need Vista training for our support engineers." Therefore, they get their list of top issues. They get a list of new components, features, all of that. In addition, they start mining for data using these beta resources to come up with, "Here’s what engineers need to know. Here are the troubleshooting pieces; here are the top issues we have seen with our beta customers. Make sure engineers know this and that."

Therefore, over the course of this beta cycle, we are also in the background writing our own training. So these engineers write the training and as RTM approaches, depending on the training strategy, they might train all their engineers or a subset of the engineers. However, the knowledge is based off spec; it is based off beta engineer experiences, and beta engineer customer interaction.

In addition, the product group, in recent years, has really stepped up to help with training. They will review our training. They will provide insights. They will attend beta deliveries of the training and help the trainer get comfortable with the material and the product.

In some cases, the product group will also do a session of live meetings on, "Hey, here’s how we intended you to support the product. Here are the supportability features that we thought would be helpful." Therefore, you get a lot of interaction with the product group as well. From the beginning of the product to the end, not only are we helping customers on the beta side, but also our selfish win is to get the training that we need.

Scott:  So, let me ask the flip side of that, right? So, you mentioned Configuration Manager. You’ve got System Center Configuration Manager 2007 coming out. Everybody wants to learn it…

Britten:  Sure.

Scott:  …because everybody is motivated. Have do you rather incent people to keep wanting to support the previous version when there is something new, shiny, and interesting.

Britten:  At least on the Commercial stage, I do not know that I have ever really run into that problem.

Scott:  OK.

Britten:  Just simply because, in the enterprise customer segment, what you generally see is not the consumer model where a product releases, and everybody goes and gets the new version. However, it is a gradual uptake, usually. So, say Configuration Manager. Right now, and I can give you the specifics on this one, it is 10% of our total SMS volume. Therefore, engineers get a couple of SCCM cases, but they get the majority of SMS 2003.

In our minds, that is beautiful. Because what happens is, slowly over time they can build their SCCM knowledge while still providing that long held knowledge on SMS. They can help those customers resolve their issues. Therefore, it is a slow mix that generally happens. I do not know, it is ‑‑ they are tech junkies. They love the new stuff, but I guess maybe we all know that we have to pay our dues with the old stuff as well. [laughing] I do not know.

Scott:  OK. That makes sense.

Britten:  That’s a good question but it never really happens quite that way on the Enterprise side of support.

Sean:  What it sounds like for you folks, is that from an enterprise standpoint, you can point to pretty clear statistics on customers who are significant accounts. Alternatively, you could point to customers who, for lack of a better phrase, probably spend a fair amount of revenue, right. In addition, it is probably easy to also look at the segment of the market and say, "We’re not going to give up 15% of SMS just because SMS, the latest version is out" ‑‑or in this case, obviously the rebranded version.

Britten:  Right.

Sean:  Because you folks can point to real metrics in terms of your percentage of the number of people who are putting in support requests and things like that.

Britten:  Right.

Sean: Thanks for taking the time to talk with us Britten.

Britten: No problem.

Bookmark this:

Interviewers: Scott Swigart , Richard Bowler

Interviewee: Matt Gibbs

In this interview with Matt, we asked him about:

• Approaching open and closed source SDL
• Development challenges for UI Framework and Services team
• Handling security within feature development
• Incorporating SDL into ASP.NET
• Launching multiple distributions with open source
• Testing against multiple operating systems
• Testing cycle for features
• Managing bug fixes and feature stability
• Following a development cycle
• Microsoft philosophy on product development

Scott Swigart: Hi, Matt. Before we start, go ahead and introduce yourself.

Matt:I’m Matt Gibbs, and I’m the Development Manager in the UI Framework and Services at Microsoft. My team is responsible for ASP.NET and AJAX as well as parts of Silverlight.

Scott Swigart: Thanks! We’ve been commissioned to look into the premise that open source and closed source SDLs [Software Development Lifecycles] are fundamentally different, and that the differences manifest themselves in the final products in tangible ways.

That’s not to say that one is better than the other – certainly both models have their strengths and weaknesses – but it’s really designed to get information out there for two audiences: one is software vendors who are looking at building products. One of their first choices is whether they are going to go with an open source or a closed source model, and we want to look at some of the ramifications of that choice.

The second audience is people bringing products into their organizations, and we want to uncover some of the basic expectations that they could have around open and closed source, or some of the characteristics of each model that might impact their choices.

Richard Bowler: You’ve been the Development Manager for ASP.NET for about a year, right?

Matt: Yes, about a year.

Richard: What would you say are the major development challenges with that product?

Matt: I think for us, the biggest challenge is that we have a large customer base, so as we innovate, we have to be very sensitive to backwards compatibility. We have a fairly complex platform, so innovation is challenging when you’re trying to make sure that existing applications people have built with it continue working.

Richard: It’s a pretty complex project, and it’s built on another complex project, which is the CLR; to co-develop with the CLR team, do you just take what they give you and work from there, or how does that work?

Matt: We work really closely with them. In fact, we meet regularly to go through code changes we’re making with members of the CLR team, so that we’re all in sync.

Richard: It sounds like you’ve been able to direct feedback to them, saying, “We really need this kind of capability, let’s fold it into the next release,” or something like that.

Matt: Yes. And they’re pretty sensitive to how their changes impact us. They run compatibility tests and stress tests against our scenarios, as well.

Richard: It seems like a huge challenge, when you consider the number of software products that Microsoft has on the market that interoperate. It seems like a huge challenge to keep them all working together.

Matt: For the framework we’ve actually held a pretty strict line. I am part of a committee where we review things that were classified as “potentially breaking a developer’s code,” and the default was that those weren’t allowed – that was kind of the highest priority. Things like “security fixes” would be allowed. Then there always needs to be a way to get back to an earlier version’s behavior, through a configuration parameter, for example.

Richard: Speaking about security, we talked with Michael Howard, and also with James Whittaker about the SDL. It looks like it’s really paying off for you guys. Were you involved in the development internally when the SDL was being instituted at Microsoft?

Matt: Yes, actually I worked with Michael Howard on IIS where we ran into a few security issues that helped highlight the need for a more structured approach.

Richard: I know what the benefits are, because we discussed it with Michael and James. But what were the main difficulties in shoehorning those new processes over the top of existing SDL processes?

Matt: I don’t think it really came down to process difficulties, as much as it came down to just helping developers recognize that the mindset they had been using needed to be front-loaded with security all along.
In our first round, everybody recognized that we needed to pay even more attention to security, and make it a higher priority. But then the realization was that it can’t be one phase in the development process. It really needs to be something that we pay attention to all along.

Richard: To put it another way, then, you had to get a wrench on the culture. You had to make security important in everyone’s mind.

Matt: Right, and it really had to be the case that everybody came to the same conclusion. You couldn’t just tell people it was important – they had to see what kind of impact it had to customers, and recognize it themselves.

Richard: Did you get much internal resistance from individual developers, or was there pretty much across-the-board buy-in?

Matt: It was pretty wholesale – everyone recognized it. I think particularly since I’ve been involved in web platforms, and we saw a couple of places where we’ve made mistakes, everybody recognized, “OK, we have to do something different, and be more practiced in this,” because otherwise we’re not going to get the right outcome.

Richard: Where was ASP.NET in the process of instituting the SDL? Was Version 1 out before the SDL really came to the fore?

Matt: No, when the process was launched we were already in development. It was during that cycle where we decided to take a break, and we went back and said, “We need to invest time in this process.” Everybody did some extra security training, and then we essentially stopped development, and did a pass for just security. We looked at every feature, all the code, looking for specific issues, just to make sure we had done all the right things.

Scott: One of the things that I found looking at the open source side of things, is just the sheer number of distributions that they have to ship for a product. MySQL for example, I think has 16 distributions, because it has to run on all of these different flavors of Linux, none of which the people working on MySQL really have a lot of control over. Those versions of Linux are all rev’ing independently of each other, so the underlying operating system that it’s dependent on is changing all the time. You’ve got a whole ton of applications that are built on top of MySQL, and are therefore dependent on it.

I can’t even really imagine exactly how to test that, and I know that in the open source world there are issues about, “This version of MySQL isn’t compatible with this version of WordPress, which isn’t compatible with this version of Linux.” So, they are always fighting these compatibility issues as things just rev independently of each other all the time.
Microsoft has some rather similar challenges, in the sense that IIS revs with the server. ASP.NET revs on its own timeline, which isn’t tied to the base operating system, and Atlas is also on its own timeline to some degree.

Talk a little bit about the kind of test matrices that Microsoft builds, and the amount of resources in terms of hardware and testers that come to bear when you guys are getting ready to release something, or while you’re developing something to ensure compatibility with all the versions of all the things you ship.

Matt: It’s probably more expensive than you might initially think, because in the same way that there are a bunch of different Linux distributions, we also still test against a lot of different supported versions of the operating system.

We’ll test against XP and Windows 2000 and Win2k3, with different levels of the service packs and with components installed; with Visual Studio installed, without Visual Studio installed, and then different flavors of IIS. We’re already testing with IIS 7 on Vista where we’ve got a lot of integration work going on. When we go to do compatibility checking and functional testing, we have a fairly large lab. And the run time, unfortunately, is on the order of weeks to really ensure compatibility.

Scott: What I’m envisioning is that somebody pushes a big button, and a hundred machines get configured with tests. It all comes up and runs, and results get cranked out — test, pass, or fail. Is it something kind of like that?

Matt: Yes, and they categorize tests into different levels of priorities so they can say, “We’re going to do a quick sanity check on something,” and that can take a couple of days. Or, they can say, “We want to run the gauntlet, and run everything we’ve got,” like before a service pack or something, and that will take much longer.
It covers everything from different language packs, to different service packs installed. One time-consuming thing is that errors that are false positives come up, and they need to be investigated. These might be just test issues, and that takes time as well, because you don’t want to ship with something that might be a problem, so everything has to be investigated.

Scott: I know that Microsoft’s got certain terminology around this. I’ve heard of something called the BVT, which I think stands for Build Verification Test. There are things that have to happen before something gets released as a CTP, Community Tech Preview. Can you talk a little bit about that, in terms of what the different levels of testing are, and what kind of things have to get verified on those different levels?

Matt: There’s a scaled approach. On a developer’s machine, we’ll have what we typically call Developer Tests.
The developer cranks out a lot of small standalone tests against the feature he’s building, against his APIs. He’ll run those every time he goes to check in code. We’ll aggregate those for our dev team, so that you’re running a fairly lengthy set of them. But, they still run within just a few minutes, so that you can continue making fast development progress. Then we’ll run a superset of those before we check in.

Scott: So, that’s to make sure that one developer’s change didn’t break some other developer’s code.

Matt: Yeah, that the basic functionality is intact. Then, the test team runs what they call a set of nightlies. They’ll run those on every daily build that comes out, they’ll kick off an automated set of nigthlies. They’re small enough in scope that they can run overnight, and any failures can be investigated fairly quickly. They are what they call “Pri0” test cases; and that would be mainstream use-cases.
Then, the superset of that is something that’s maybe ten times the number of test cases, and that’ll run for days to verify the full functionality. That’s literally thousands upon thousands of test cases.

Richard: So, your nightlies maintain a level of sanity about the ongoing nature of the code base – yesterday it was still good, today it’s still good, tomorrow somebody’s checking in something bad. Let’s back up and fix this so you never get too far out of whack.

Matt: Then, you may find something when you go through a full test pass. When the tester originally made the test plan, they may have said it’s a fringe scenario, so it was classified as a priority two test case. You might only run it before a release, and then something breaks.

Scott: Microsoft has a lot of people who blog, so there are obviously a lot of people on the product teams who are fairly transparent about where they are and what they’re doing. I think Scott Guthrie put up a big blog post where he said, “You know, there’s a point in time where we don’t necessarily fix every bug because we have to weigh the risk of impacting the stability of the product.”

Every time we touch a line of code, there’s a risk that we’re going to hurt the stability of the product, so there’s a point where we switch from telling a certain team what we’re fixing to actually having to get permission to fix certain things because of where we are at in the stabilization process. Am I characterizing that right?

Matt: Yeah, we’ve spent a lot of time mining our bug tracking database, and you look at data points like, “For X number of bugs, how many fixes actually cause some other problem?” The closer you get to shipping, the more likely you are to look at something and say, “This is really not a very important bug, and people are just not really going to see this. It’s good that we did the due diligence and found it, but this late in the game, you run the risk of breaking something more important.” You might just leave a bug in there and fix it when you have more time to shake out the problems.

Richard: I guess if you didn’t do that, you’d never ship.

Matt: Exactly. With new engineers, you’ll regularly see them hit this kind of frustrating point when they have a bug, they want to fix it, but it’s time to ship and they get told no. Their point of view is that it’s a problem that should be fixed, and they have to be told that they can fix it, but not until the next version

Scott: I have to admit that it’s odd to think that leaving bugs alone raises quality.

Richard: In my own process over time it seems like when we get down to the mode when we’re primarily testing an implemented product and engineers are working on bug fixes and not new features and behaviors. It seems like as you get closer and closer to when you need to ship it gets harder and harder to jump through the hoop of where a bug ought to be fixed. At some point it’s got to be an absolute showstopper to stop the shipment; it’s got to be a crash or something like that, otherwise you’re just not going to fix it until the next release.

Matt: Yeah, we get to the very end where we call it recall mode, where the question is whether you would actually take the product off the shelf in order to fix this issue. At this final point in the process, unless we’d recall the product, we don’t fix it.

Scott: Another thing that’s pretty interesting to me is the whole way that Microsoft does milestones. Again, my impression of it from the outside is that somebody will say, “OK, these are the features that we’re targeting for milestone one, for M1,” and so developers will go off and they’ll be working on those features. There will be a graph of the bug count that’s just climbing, climbing, climbing; the number of test cases written are just climbing, climbing, climbing.

At some point, it seems to switch over to where new features aren’t being written, it’s focusing on stabilization for the M1 milestone. So, there’s a trend down in the bug count. It seems like there are a lot of different variables that are being looked at in those stages; what’s happening to the bug count, what’s happening to the perf numbers…
Can you talk about how all that stuff comes together? Am I characterizing these milestones correctly? Is that sort of how it works?

Matt: In the past, that is how I would have characterized it – a sort of waterfall approach where we crank out a bunch of features and accumulate bugs, and the test team cranks out test cases, and then we play catch-up and try to satisfy all those test cases and bring the bug count down isn’t the most efficient. But we’ve actually shifted in our group to a little different approach that we think is working better.

Instead, we’ve gone to what we’re calling a feature crew model where the dev, test, and PM work together from the start on defining the design, the set of test cases, and the unit tests, and they drive for more quality up front before the feature even gets checked into the product. So, they’ll work in an isolated source code branch getting a feature to a higher level of quality and then get it into the product.

Richard: It sounds like you’re moving, at least to some extent, toward a more Agile model.

Matt: Yeah, it’s ended up being a hybrid of Agile development with features of SCRUM, but you couldn’t really call it one or the other. It still falls into a sort of waterfall effect where we have a milestone of a bunch of features we’re going to try to get done by a certain time. There is still this mentality of feature crews driving for a deadline, so we picked the pieces of a bunch of different methodologies that seem to work well for us in delivering commercial software, and we are trying to make them work together.

Scott: Now, I’ve heard a little bit about these feature crews. Correct me if I’m wrong, but that seems to have been developed after the ship of Visual Studio 2005. It came out of … I think they called it like, MQ or Milestone Quality, or something like that?

Matt: Yeah, it was kind of built around the milestone for quality.

Scott: So, some of the ramifications of that are that if there’s a feature you’re dependent on to build your feature, it might take you longer to get that. But, when Microsoft releases things like Community Tech Previews, they’re going to be more stable by default, because things were only checked into the source base that passed a higher quality bar. Is that right?

Matt: That’s the goal, and we were able to benefit from the approach with the ASP.NET AJAX release. We embraced the feature crew model when it was still ramping up for the rest of the division, and it worked out pretty well for us. We learned a few things we could do better.
We also learned that some of the dependencies became a little bit difficult to manage. If you truly had a feature that needed something else to be completed, then you ended up with a staggered set of feature work waiting for future builds. You needed an equal set of things that could be done early, in order for everybody to keep being productive.

Richard: It sounds like what you’re doing is trying to make sure that your interim quality is higher by testing feature-by-feature, and by enforcing a quality standard on individual pieces instead of waiting for the entire conglomeration. Is that a correct characterization?

Matt: Yes, hold the line on quality all along. Don’t accumulate debt in order to get more features done earlier.

Richard: That seems really smart to me. There’s an old saying you can’t test quality into a product, and that’s really true. It seems really smart to me. Was that inspired by the SDL’s focus on early process involvement of quality assurance?

Matt: Yes. Before, we almost had two instances of lifecycles. You had development working on one lifecycle model, and you had the quality assurance teams running along six to eight weeks behind the dev team.
But, that wasn’t really working efficiently because all we were doing was having one team try to play catch-up with the other team. The dev team was cranking out features, and the test team was finding problems with it. Then, the dev team was trying to catch-up on the problems that the test team found.

Scott: With Visual Studio 2005 when it came out, there were a lot of issues around stability and quality, and things like that.
It was easily the most complicated development environment that Microsoft had ever built, maybe that anybody had ever built. It had amazing levels of cross-product dependencies; there were big products of their own that were put in there, like Team System.

Some of the impression, at least from the outside, was that it didn’t all come together and stabilize at the end the way that everybody had hoped. So, is that correct? Do you feel like maybe the initial version wasn’t as stable as everybody hoped, and so some of these things have been done as a response?

Matt: That may be the conclusion from my team’s perspective, focusing on the platform. I think we probably feel like we’re in pretty good shape because we have criteria and goals that were met.
But, I know in the MQ timeframes a lot of people focused on a lot of little things that weren’t quite right that they wanted to fix up for the next release. That was primarily targeted at the development tools environment, not the platform.

Scott: You work in the open source world in a sense with Atlas, right? Because there are a set of controls for Atlas that are effectively open source in that Microsoft puts the source out there. They’re maybe not open source in the sense that anybody in the world can just modify the source and submit changes to it, but please talk a little bit about this foray into open source with some of the Atlas controls.

Matt: The AJAX Toolkit Project is a separate team around the corner from us, and it is shared source. They take code submissions from the public. They have released the really snappy UI controls that everybody uses on top of the Atlas part of the platform.
It’s been well received, and people are able to grab the source, and build it, and contribute. It’s a little different for us in comparison to what we’ve done before. We’ll see how much the community itself drives lots of new controls, or if it’s something where they like the fact that the controls are out there, but they don’t really actively contribute a lot.

Scott: Certainly a key benefit of open source is that the source is out there for you to learn from. I think that even taking the Windows Forms controls and the ASP.NET controls, certainly a lot of best practices might have been gleaned from being able to look at how Microsoft did what it did.

Matt: Well, we’ve done that with the Atlas release now, too. That’s a first for us, and I’m really proud of it. When we released ASP.NET AJAX, we released the source symbols for it so you can attach a debugger, and step right through our source code.
It’s not the same as taking source code in from the community, but we look at it as being very open and transparent. It’s one of those things where in the open source community, everybody says, “I have the source, so I can do what I want with it.” From my perspective, though, people tend to say that but not do it.

Scott: I think they call it the Berkeley Conundrum. It’s a variant on “if a tree falls in the woods, but nobody hears it, it doesn’t really make a sound.” If there’s so much open source code out there, and not really enough eyeballs to actually look at it, does it really matter that it’s open source?

One of the things that seems to me like a huge challenge is that inside of Microsoft you can control exactly what people get to do to. You can control the training that they have to have around security, and performance, and best practices. You can control the unit tests they write, and you can control the process that the code goes through.
When you take a look at open source, you just get code delivered to you and you don’t have any control, or any view into the process that was used to create that code.

From Microsoft’s perspective, you’re taking submissions from the community; how do you ensure things like security, performance, and reliability, when anybody could be submitting the code, and you don’t have control over how they got there?

Matt: I’m not intimately familiar with it, but my understanding is that the team has essentially taken on the burden, so far, of ensuring some of the security themselves by essentially laying our security process on top of the code before it goes out to the public. I also think there’s an element of caveat emptor.

But, because it’s coming from Microsoft, there has been a certain expectation of quality. At this point, because we have a dedicated team working on it, they’re in the code and it’s still going through quite a bit of the Microsoft process.
Also, at this point, you can’t just submit and have it go right out to CodePlex. You submit as a team, and your submission is then under review for inclusion on CodePlex.

Richard: What do you think are the advantages of closed source over open source?

Matt: In my mind, it comes down to probably two things. One, the customer knows they have somebody to go back to, that there is a company that they’re buying software from that is backing it up. Two, there is a full-time developer that is working on advancing the software.
Open source has done fantastic things with having a community effort drive things to do good software. But, the fundamental difference is that with proprietary software you have an entity that’s responsible and people employed to push things forward.

Richard: What about the development of that process itself — the mechanics of designing, implementing, and shipping a piece of software. Do you see advantages to the closed source model, and if so, what are they?

Matt: I think you may have a little bit more of a culture of discipline but I’m not sure that it’s really that much different. I think there’s so much passion from people in the open source community in what they’re building that it may be the same kind of thing. The same kind of respect somebody has for their full-time job.

Scott: There are two myths that I frequently encounter, one related to open source, and one related to proprietary software. The open source myth that I hear again and again is that many eyeballs looking at the code ensures security.

Michael Howard really threw down the gauntlet on that and said, “Show me the data, because there’s just no data to support the idea that you can make sure code is secure just by open sourcing it and letting people look at it.”
On the closed source side, one of the things that I often hear as being an advantage is this notion that there’s a company to go back to, there’s a company that stands behind it. My impression is, I don’t have any guarantee that a bug that I find will get fixed in the product. I don’t have any guarantee that a problem that I’m running into will get solved by the company that produced it. I can file a bug, I can call product support; maybe I’ll get a resolution, maybe I won’t.
Do you think that Microsoft or any other company really offers that level of guarantee?

Matt: In my experience having worked as a quick-fix engineer where we investigated customer reported problems from PSS, it was very serious. If you couldn’t find some way to workaround it, it needed to be a product fix. Now with that being said, if there was a reasonable workaround, then you’re back to the issue of whether it’s worth the risk to go patch product code.

There’s not a case I can think of where a customer had a serious problem and we didn’t either find them a workaround or patch the product in order to get them back on track. The open source community can’t guarantee every requested change will be made either.

Scott: I guess the only caveat to that might be that obviously, it’s not going to be instantaneous, in either the closed source or the open source world. I remember a problem I found, and the answer was it’ll be fixed in Service Pack 2, but Service Pack 2 didn’t come out for another six months.

Matt: Very few fixes in proprietary or open source development are instantaneous. More complex issues may take longer, and again there is the concern about compatibility. When something like Service Pack 2 ships, it’s had many hours of testing devoted to it to ensure that even small fixes are getting good coverage.

Scott: Clearly, from your perspective, that guarantee of supportability is real, but there are just certain kinds of physical constraints on how fast the resolution might be available. But in your experience if there is a critical problem, either the customer gets a workaround for it or Microsoft starts working on a fix?

Matt: That’s been my experience in ten years here.

Richard: We talked to one of the guys that works on the Linux kernel. He basically said what happens is that people turn code in to the mailing list, and then everybody takes a look at it. If they’re interested and they snipe at it, then the developer makes some changes. Then, ultimately it either gets adopted by the maintainer – the person who owns that section of the kernel – or it doesn’t.

So, at least in some sense there’s no process to enforce quality during implementation beyond peer review. It seems to me intuitively that that’s a disadvantage. But, I just wanted to get your take on it.

Matt: Of course we’re not working on the kernel, but I know that before a piece of code I write is going to get anywhere that it’s going to get exercised quite a bit. It seems like inherently that’s an advantage.

Scott: Matt, talk a little bit about motivation inside of Microsoft. How much is it sort of computer-generated, and how much is it, “Developer Number 13 had this much code checked in, this few defects, found this many bugs…?” How does it work in the “big house?”

Matt: For us it’s all about our developer customers. It’s about the level of enthusiasm and people’s feedback, and it all feeds off of that. We get our customer’s excitement, what they want to see, what they’re going to be able to build. They say to us, “If only you added this feature you could save me time, and here’s what I want to see next.”
The people here in Microsoft are completely driven by what our customers want to see next. We may be in a bit of a unique position because our customer is a developer, and developers get really excited about the platform.

Scott: I’ve heard from people who’ve interviewed at Microsoft come back and say, “There really is a sense that by working at Microsoft you have an opportunity to change the world.” I would guess that many of Microsoft’s products … if you take a look at Office, Visual Studio, Silverlight, and how you hope those products progress over time, most of these are big bets. In most of these cases, you are part of something that you hope is going to be really big and industry-changing.

Matt: Yes, it’s the kind of thing where I’ve had people come up to me at ASP.NET conferences and say, “You changed how I do my work. It’s been an incredible experience and I can’t wait to see what you guys do next.” It makes an impact on their life. It makes it fun.

It’s different than what Scott was driving at; the “how many defects” and “how many lines of code” kind of thing. We really run off of developer-driven enthusiasm. There’s an enthusiastic, sharp, hardworking group of people that are really focused on the customer and how to make better developer experiences.

Richard: Well, thanks very much for talking with us. I appreciate it.

Matt: Sure.

Bookmark this: