Interviewee: Leon Shklar

In this interview we talk with Leon Shklar – EVP of Media Technology for Reuters. In specific, we talk about:

• Moving Reuters.com to an open source stack
• Choosing and supporting open source software
• Making code changes to open source packages
• Licenses and governance in open source projects
• The perception and reality of open source companies

Sean Campbell: Leon, could you introduce yourself?

Leon Shklar: Sure. I am EVP of Technology for the Media division of Reuters (now Thomson-Reuters Markets).

I hold a PhD in Computer Science from Rutgers, and I have been doing some part time teaching there as well. That came in handy in team building – the team includes some of the former Rutgers students and other people working together through a couple of startups and other jobs.

I have been at Reuters for a little over two years. Prior to that, I was responsible for the technology side of the Wall Street Journal Online. I am pretty proud of what our team accomplished both at WSJ.com and here at Reuters so far.

To wrap up on the history, I worked for Belcore from 1990 to 1996. Before that, I worked with a small software company, called Computer Corporation of America, which made DBMSs for IBM mainframes.

In 1996, I took the plunge into the world of startups. We started a small company in ’96, and then it got acquired in ’99. Then everything went up in 2000 and came crashing down in 2001. When things started going south, I joined Dow Jones, and was responsible for the Wall Street Journal Online, from summer 2002 to spring 2006, which is when I went to Reuters.

On a related subject–with a friend who worked with me together in a couple of jobs, we wrote a book on Web Applications Architecture that came out in its first edition in 2003, by Wiley. The second edition is supposed to come out later this year.

Sean: It looks like it has some really good reviews. Scott and I have actually been lead authors on three technical books too. The joke my CTO gave me once was that “All authors like the act of having written. The act of writing, though, they are not so sure about.” I’d have to say I agree with that.

Tell me a little bit about the technology stack that Reuters sits on, in terms of how you guys use open source, what are the major products, and how that underpins the organization.

Leon: The responsibility of our team includes both consumer web sites and business-to-business media products. The former include the family of Reuters.com web sites–eighteen of them to be precise, for different countries and regions. The latter include text, pictures, and video products distributed over the Internet and via satellite. Overall, quite a mix of current and legacy systems, and not all of them are compatible. Our team is on a mission to bring all of media systems to a common platform.

Over the last two years, we migrated Reuters.com sites from Microsoft .NET technologies to the open source stack. There is some similarity in what we have done back at Wall Street Journal Online and here. When I started at Wall Street Journal Online, there were a bunch of disjointed pieces. The site itself was based on Vignette, there were some scripts, and there was a pretty big buy-in on the company side into Solaris and Sun and commercial tools. The company already had licenses for WebSphere and Oracle, which of course are pretty expensive.

I don’t think the open source components were as robust in 2002 as they are today, and the migration there was away from Vignette, heterogeneous scripts, and all kinds of obsolete CGIs and stuff like that, to a cohesive platform based on the Oracle tier, WebSphere, and a bunch of applications built on top of that.

The OS platform was Solaris. It wasn’t a cheap environment to run, because Solaris systems weren’t cheap. The Oracle licenses and WebSphere licenses are not cheap either, in terms of maintenance and upgrades. But we ultimately got the system to be pretty solid, by the time I left in 2006.

When I interviewed for the job at Reuters, the main question that I was asked, well, can you fix the site? It is not standing up.

Sean: Tell us a little bit about that. Was the issue page load times, or that users felt the site wasn’t responsive; what was the real issue?

Leon: It was a combination of problems. Response time was not very good, and the site wasn’t able to sustain the load at peak times. The other problem was that the implementation of changes was very slow.

The site was implemented as one big blob. To make even small changes, you pretty much needed to deploy the whole code-base.

That was a very difficult thing indeed. I am not religious about open source versus Microsoft, but one thing that’s true is that the bar for doing good work with open source is much higher. You need much better people. To build simple things with .NET is pretty straightforward. So you may have lower average technical levels on the team, when the requirement is just to write to code according to a functional spec. My personal feeling is that .NET by its nature is too permissive to bad coding practices.

Sean: Where do you feel .NET is too permissive in terms of coding practices, by comparison to open source?

Leon: Ultimately, all the door-opening in .NET is done through tools. There is not reasoning from first principles. You write functional code in tools and it gets compiled and then it all theoretically comes together. .NET is designed to do things for you, and that is a perfectly good thing when you’re building simple web sites.

It becomes not necessarily such a good thing when you’re building complex and scalable applications. By the way, that’s part of the reason why .NET is so successful–80% of sites are rather simple, and they don’t have an incredible requirement in terms of performance flow and even the frequency of changes. So for those, it’s perfectly good.

It’s not a very high bar that you need to pass in order to be able to write code for .NET. And it’s much easier to hire people, which means the team is not as expensive.

All that’s fine–it’s just when you get to something like 10-20% of the general population of sites, that you start running into real challenges in using .NET. I’m convinced that you can write good software in anything, and I expect that the implementation of MSN.com, for example, is done well (though you never know). I would guess that the people who are doing it have good expertise and knowledge of that technology.

Unfortunately, that is more of an exception rather than the rule. On average, the expertise level of .NET web development teams I’ve seen is considerably lower than the expertise of open source teams. With open source, you have a big challenge in building a really strong team. The advantage is that you have full control over the platform that you’re building, and ultimately, when you have a problem, you can reason from first principles.

I have seen the best .NET sites with very annoying intermittent errors in them, and nobody knows how to find them. The reason for that is, if the tools do help, there is not much you can do. It’s very difficult to look at elements of the code; it’s very difficult to trace HTTP messages through and to figure out, “Well, it’s this missing header that’s really causing this strange behavior.”

Scott Swigart: Talk a little bit about the infrastructure that you guys are built on now. Are you on sort of a LAMP stack?

Leon: It’s LAMJ.

Scott: So it’s Java instead of PHP or Perl.

Leon: Right. We have Linux boxes with Apache fronting Tomcat. The backend is MySQL with master-slave replication.

We have a number of application components that we built on top Struts. Of course, we built our own platform with our own application components, but we tried our best not to reinvent the wheel.

The content management system is something we had a lot of pain and discussion over two years ago, but we were not able to find the one that everyone liked. I’ve seen a number of commercial products, and generally, the problem with them is that the amount of effort to customize and support them is close to that of building your own.

Drupal was not there yet, and we ended up building a content management system, which is specialized to support our custom applications. Because it is specialized, it doesn’t pretend to be generic, and users are quite happy with it.

Scott: Mark Frons expressed the same kind of thing–he sees that as an area that’s ripe for some open source project to coalesce around, because right now everybody seems to be building their own CMS to some degree.

Leon: That’s true, and there are not very good commercial products either, that I know of. Vignette was trying to build a content management system; I don’t think they succeeded. There are a bunch of other companies like Documentum and Interwoven that have content management systems. They’re very generic, and they don’t appear to be very robust in what we need.

So what ends up happening is that if you use one of those, the effort involved in installing, supporting, and customizing it is as high (or higher) as the effort for building a very custom system of your own.

Scott: In terms of the stack that you’re built on top of–MySQL and Linux and things like that–do you rely on your own expertise and the community and things like that for support and figuring out problems? Do you get support from a vendor?

Leon: I would say that 95% of the time we rely on our own expertise and we rely on the information that’s available in the community. When we were first deploying MySQL, we didn’t have enough expertise, so we had to get help from MySQL. All the tuning they did for us was very, very useful.

We have a support agreement through UNISYS that covers MySQL, Apache, and Tomcat. We almost never contact them with questions regarding Tomcat, Apache, or MySQL, but when we need it, we have this backup.

Scott: One aspect of the value from their acquisition by Sun that MySQL talked about was that they had been looking at doing an IPO in a fairly turbulent market. And with Sun, they felt that Fortune 500s would feel safer about using MySQL if it is backed by another Fortune 500. I would be interested to get your take on that, as a MySQL customer with long-term involvement with open source products.

Leon: I am not necessarily excited about it. This is conjecture, but when I heard about Sun acquiring MySQL, it didn’t make sense at first. The only context in which it does makes sense is that Sun starts distributing MySQL with Solaris, because Sun is trying to gain acceptance for Solaris, as an alternative to Linux.

My concern with that, and one of the reasons we went with Linux to begin with, is that the open source tools are built for Linux first, and only then ported to Solaris.

Sean: It’s the LAMP stack, not the SAMP stack.

Leon: Exactly. So if MySQL is going to become part of the SAMP stack, sooner or later they are going to start doing development on Solaris and porting it to Linux.

Sean: Some people say, “Hey, the great thing about Linux, and Apache, and a lot of these kinds of things are that they are open source, so if I really run into a problem, I can get into the code and fix it.”

But other people say, “Well, I have no interest in doing that. I don’t want to run my own private fork of Linux or Apache. I use those as a product. There is value in the fact that they were built using an open source methodology, but they are not something that I am going to really go in and muck with.”

What’s your take on all of that?

Leon: It is not a good thing to create your own branch of an open source product. That said, if we are in a critical situation, it’s not completely out of the question to go into the code and put a fix in, and then submit it, and make it part of the community so that it goes into the next release. Ultimately, when the next release becomes available, you replace this little emergency fix that you made.

But going into the code is the last resort, and you would only do that if there is no other way. We have people that know (or can analyze) the code for open source products pretty well. At the same time, we do have support agreements with Unisys and MySQL, and we would rather they did that, if deemed necessary.

If need be, we can work with them to figure out what the problem is, and try to come up with some temporary fix or workaround. It is clearly not a sustainable scenario to constantly make changes to open source components. To be specific, making changes to Apache, Tomcat or MySQL code is not a good idea, except as a short-term measure.

Sean: It sounds like, in a practical sense, those situations just don’t really come up very often. The products that you are building on are mature, and you are not doing such exotic stuff that you are taking them into a use case where they have never been used before.

Leon: We try not to. These are fairly sophisticated products. When we first deployed our system, we ran into the kinds of bugs that required people to go into the code. Not to make a change, but to understand the code and to find a workaround.

Most of the time, if you have an exotic error, you don’t have to fix the code. Once you understand the implications, you probably can come up with a workaround.

Sean: So you are talking about having a really high bar for your staff, if people have to be able to trace into the code of the web server, and things like that. But on the other hand–like you said–the scale of the stuff that you guys are doing is way outside of that 80% kind of case for web sites.

Leon: Yes, it is. And let me put it this way. I don’t remember a scenario where we actually had to change Tomcat code. We never changed the base Apache code either, but then there are plug-ins that are distributed with Apache. We did change those, even the ones that are considered part of the standard distribution.

Sean: I suppose that the plug-ins are a little more special purpose than the core web server, and they are not necessarily used by everyone the same way.

Leon: I would also say that they are not as mature, because not so many people are using them. The number of people using Apache is enormous; the number of people using a particular plug-in can be two orders of magnitude lower.

Sean: Many times, people are using a lot of open source stuff around the LAMP stack, like components, unit testing tools, automated build systems, and libraries.

Leon: We use that as well–we use Subversion as the versioning system, we used Maven, and we used to use Ant. That’s pretty much the primary environment.

Sean: How do you evaluate which open source projects to bet on? If you just go to SourceForge, there is a lot of stuff there, some of which is kind of abandon-ware, whereas some is very actively maintained and being worked on.

Leon: You try not to use the libraries that don’t have a lot of activity around them. You also need to look very carefully at their license; you don’t want to use a viral license.

There are a couple of different licenses, as I am sure you know, for open source. One type lets you do whatever you want. Another type lets you do whatever you want, but you can’t resell it. Then there are a number of really bad licenses, where you use the code as you wish, but all the code that you write becomes open source.

And those, we don’t want to use.

Sean: We had an interesting conversation with the president of the Apache Foundation, and he talked about some of their tenets. One of them is that any Apache project has to have maintainers from at least three different companies, so that no one company can force the direction of the project a certain way.

Which is a little different from how MySQL works, or a lot of the Sun stuff works, where it’s very much corporate governance versus community governance.

The other thing that he talked about was that they strive for a very commercial-friendly license, where you can embed stuff with the Apache license in commercial software and you’re not required to open source your derivative works.

Leon: That’s critical.

Scott: On one hand, you’ve got open source, and then a sub-community of that is the Free Software Foundation, which is very much ideologically oriented around the idea that “the world would be a better place if everybody released their source code for everything.”

Leon: I don’t buy into that. I’m very much in favor of the commercial model, and I guess my conviction is very close to that of MySQL, because they do derive commercial benefit from open sourcing their code. They made a conscious decision; and they’ve got many people in the community contributing to the evolution of their code. At the same time, they obviously invested a lot of money.

In the end, their revenue is not coming from licensing–it’s coming from services, and they’re the primary experts on the database. Consequently, they have the ability to charge premium money for those services. I’m totally in favor of this sort of approach to open source where software is a commodity, and you get interested parties to contribute to its evolution.

Scott: On the closed source side, if you’re using Microsoft stuff or VMWare stuff or stuff from Adobe or whoever, do you feel like you feel like you have to look at the EULAs to the same degree?

Leon: With open source, you do have to pay more attention to the licensing. With commercial software, generally I haven’t found it to be a problem.

As a side issue, open source and closed source are not born equal. I wouldn’t use an open source component in a Microsoft environment. If I were forced to use a Microsoft environment, I would probably get a commercial database. If I had to build a system on the Microsoft OS, I would use SQL Server. I would not use one of open source projects, because I don’t feel they’re robust enough. There are not enough acceptances. There is not enough power and attention from the community. It’s very, very limited.

I would rather not run MySQL on Windows in production. I would most definitely not run one of those open source databases built for Windows, which are not MySQL. I don’t want to name them, but there are a number of them that were built for Windows and are considered open source projects, and they’re not so good.

Scott: For open source that only runs on Windows, the community is probably a magnitude smaller than open source that primarily targets Linux.

Leon: That’s one consideration, and the second consideration is that Microsoft is not the primary environment that attracts the open source community. I don’t want to sound like a bigot, but I do believe that the average expertise level of developers that work on Linux is higher than the average expertise level of developers that are working in Windows. And it’s not because Windows is not good and Linux is good. Windows is an excellent operating system that went through a couple of evolutions.

It’s just a completely different scenario. It’s relatively closed, and the conscious decision that Microsoft made at different times was to build something that targeted 80% of the development community, and it’s done well for them.

Sean: One of the reasons that I like my Macbook is that I can run any OS on it that I want at this point.

Yet they’re a closed source company, by and large. And they’ve got this massive innovation that they’re driving. I just saw the iPhone sales numbers this morning. Yet you take another example of a big company like Google, who gets a lot of credit for being open source, but they’re not really in a sense.

They’ve built their business on open source and they have some projects, but it’s not like they turn around and say, “Well, here are all the algorithms we use. Just go deal with them.” And then you’ve got companies like MySQL, that are purely open source, and they’re out to make a profit.

So I’m curious, as you look out across these different vectors, how do you think innovation plays into that? Because from my perspective, when I look at open source products, the more technical the product, and the more the product is by developers for developers, the more innovation seems to stream out.

But when they get into more commercial products like Open Office, to be frank, does it really have the same innovation level? That’s not to say that I’m in love with Outlook. I’ve got my problems with that too.

From the standpoint of acquiring technology in your role, when you’re looking for stuff that’s innovative, do you see a similar sort of breakdown, where the more technical the open source project is, the more likely it is to be innovative?

Leon: First, there is something very interesting that you said that I was kind of itching to respond to immediately. Think of Google, think of Apple. They’re not open source. But there is a perception of open source.

I’ll give you an anecdotal example from real life. We started using the Google appliance for search, and we were planning to deploy it and they had a bug that was preventing us from going live. And they actually guard their code and algorithms quite closely.

So we said, “All right guys, when are you going to fix the bug.” And they said, “Oh, the next release is in three months.” But that doesn’t work for us. We went back and forth and in the end they said, “You know, guys, we understand. We can’t release it sooner than that, so here is the code.”

I put a lot of value in that. It’s a different philosophy. Yeah, it’s closed, but Microsoft would never in their life do that.

Sean: Microsoft actually does make the Windows source available to certain customers, governments, etc., but I don’t think that’s widely known.

About the second part of the question, is there an innovation continuum in open source, where certain projects have innovation that is arguably better than in a closed source model, whereas in others maybe it struggles to really hit the mark and bring things forward?

Leon: You definitely have a strong point, and you gave an example before. I don’t think Open Office is necessarily innovative. I don’t use it. It has kinks still, and it’s too much trouble.

Sean: That’s an easy one to throw a stone at in terms of its innovation, but I want to get your sense as someone who has to acquire and find the right technology, where you think the soft spot is.

For example, the Mac is amazing for UI innovation and fidelity of user experience. On the other hand, go try and find a really good database product for the Mac.

On the flip side, I don’t’ think people have seen Vista as the paragon of great UI innovation, you know what I mean?

Leon: Just to elaborate a little bit on this example of Open Office, I don’t believe it’s quite fully an open source project. It’s heavily sponsored by Sun, and if it was strictly an open source project, the community would not spend this much effort in building an office product.

So there is a difference in an open source project that is supported by the community and a company deciding to open source something. When Sun open sourced Solaris, it didn’t suddenly become an open source operating system in the full definition of the word.

The fact that something is not closed source doesn’t make it open source. There is also a community element.

Scott: Right–who’s allowed to contribute and who controls the direction of it.

Leon: I believe that open source projects with strong community interest in them and strong community participation in them are all pretty innovative. Not all projects that are based on public code are innovative open source projects.

The other part of that is, taking Apple as an example, it is proprietary, but at the same time I can take an Apple laptop and open a command window and I can type my Linux command.

As far as business decisions go, catering to 80% of your audience is good, but it has a price. In the case of Windows, you don’t get a very strong following on the side of cutting edge developers. Because developers don’t love Windows machines. I don’t know a single one.

Scott: Oh, I know some extremely passionate .NET developers.

Leon: Ok, maybe it’s my bias coming through…

[laughter]

Sean: As usual, we have a lot more questions we wish we could have asked, but I want to be sensitive to the time, as we’d scheduled an hour, but thanks for taking the time to chat. This was a great conversation.

Leon: Thank you.

Bookmark this:

Interviewees:Shawn Burke.

In this interview with Shawn Burke of Microsoft we asked him about:

• Shawn’s motivation for helping to make the framework source code available.
• Whether there are any limits on who gets access to the source code.
• Whether there was any internal resistance initially to providing the framework source.
• Why Microsoft doesn’t just provide a zip for the source.
• Whether Microsoft’s individual developers write code differently knowing that their actual source might be available for the world to see someday.
• Whether or not Microsoft is planning on making available other development artifacts such as test scripts, threat models, etc.
• Whether a closed source company or an open source company is better positioned to support external developers who need access to the source for a given project.

Scott: All right. So if you take a second and just briefly introduce yourself, that is probably a good place to start. Then we can drill into questions specifically around the source for the.NET framework 3.5.

Shawn: OK. I am Shawn Burke; I am a director here at the.NET development platform at Microsoft. I run a group that we internally call the agility team. We run small high value projects to try to help deliver value to customers in agile ways. Not necessarily using agile technology, but just being agile about how we deliver.

Scott: :So in other words, it sounds like you are looking for sort of low hanging fruit where you can do things that are a little bit outside of the norm but would not necessarily be an enormous effort, but would deliver a lot of value to the customers quickly. Am I characterizing it correctly?

Shawn: That is right. Our team works a little bit outside the bounds of the typical projects and we are not constrained by the same rules they are. We are able to do things that are more customer focused and a little less process focused way.

Scott: Just as a point of clarification, people have said Microsoft is “open sourcing the.NET framework.”That is not what your Microsofts intention is and not really what you are doing. You are making your source code available for reference. You are not really open sourcing it the way that, you know…

Shawn: Nope, some people in the community used characterization, that is definitely not what were doing.Were making the source available as a tool that helps people be more successful with the platform in its current state.

With this effort, my goal has been from day one, a simple scenario, which is you are debugging against an API, you are getting an exception from the API, and you cannot figure out why you are getting it. It is often very difficult to figure that out just from documentation. However, your ability to step into the API often makes it really clear what you are doing wrong.

Scott: Right. In the example that Scott Guthrie put up on his site, this is something I’ve run into,? At the point where you DataBind something, all kinds of stuff happens like down in the plumbing.

Shawn: Yes.

Scott: And the exception that you get, you know, you do not really have a lot of visibility and sort of what you did wrong in your code that caused the DataBinding exception. And so that’s like…

Shawn: Yes.

Scott: That’s like the key scenario that you went after.

Shawn: Yes. That is like the thing that you are unableto do currently you know, because I work here I have the benefit of being able to do that a lot. Quite frequently, the amount of time it saved me , “Oh, Ill just grab the symbols” and you do and start to debug with them and immediately you’re like “Oh! Because I forgot to do this or that.” It becomes so clear that I really wanted to enable that sort of service for customers because its such a powerful thing

Scott: Yes. And just to back up another thing that you said, too. I never read it. I was not indicating that Microsoft was somehow trying to create the impression that they open sourcing this. I know when I looked at it, it was clear to me that Microsoft was releasing the source for reference purposes, you were not releasing it so that people could…

Shawn: Yes. I get it. I did not mean to sound defensive there!

Scott: [laughs]
I’m sure it comes up a lot.

Shawn: Yes. That was the first thing we saw on a large website that is often not terribly pro Microsoft

Scott: [laughs]

Shawn: Fortunately many people came on there and said, “No, no, no.” They never said that, that is not what they are doing. I think most people get that now.

Scott: So what you are enabling is a sort of step into debugging experience where normally, as soon as you call into the API, you could not continue to drill down and step in. Now you can.

One of the things I was not clear on is who gets this? Is this available to all versions of Visual Studio and all customers or is there a sort a subset? Who is this available to I guess?

Shawn: If you are running on Windows, it is available to anyone who can use it or to look at it.

As far as the Visual Studio integration, you need to have VS Standard or above. You cannot do it with Express versions. You know the express versions of Visual Studio?

Scott: Sure.

Shawn: Express does not get it for some kind of fundamental reasons.The main reason there is it just simply because Express does not expose the UI that you need to set this up or have some of the other facilities internally to take advantage of it

Scott: Sure. Sure.

Scott: Yes. The only reason I ask is that some of the people in the Microsoft ecosystem have made their source available kind of under premium plans with their customers. However, that is not the approach that Microsoft is taking with this. This is not sort of a premium thing, this is just generally available.

Shawn: Yes, that is Code Center Premium (CCP), this works similar to that CCP , but there you need to have an account and a Smart Card actually. And you have to go through much more security than in this system. This is a public reference release of the code, so there is really no credentials required. There is none of that.

Scott: So when you sort of floated this idea internally, it sounds like it was generally well received. People kind of jumped on board and wanted to help make it happen. Just behind the scenes, what were some of the things that it took to enable this? It is more than just dropping a zip of the source out there to make this happen.

Shawn: Yeah. Well, yeah. It is good that you brought that up. This whole thing started quite a few years ago. When I was on the Windows Forms team, which I was for seven years, I always felt like we should be releasing the source for many reasons. Others had done it. I felt like it was good for customers. When I was the Development Manager on Windows Forms in early 2005, I did a blog post about this, saying I wanted to make this happen and talking about some of the challenges to doing so. It was kind of justI was just kind of brainstorming aloud about different ways and what some of the challenges were. The response was enormous.Hundreds of comments.

A funny story, I did that blog post, and then I went down to Cabo San Lucas with my girlfriend a couple of days later. When I turned my phone on halfway through the trip, just to see if it would work in Cabo, I had a flood of text messages that said, “Please call work.”

Sean Campbell: [laughter] All right. So you learned a lesson of do not blog anything but “I’m going to Cabo” on the blog before you go to Cabo, basically.

Shawn: Yeah. I think if you would really pull it down to basics, that is it. Well, actually, I had always done tons of blog posts that no one’s…

Scott: Meanwhile, you created this firestorm of all these people running around going, “What have you done?”

Shawn: Yeah, exactly. I like to joke that it created, basically, an international incident because it got picked up by eWEEK, Mary Jo Foley, CNET, and all the rest. Quite a few people picked it up. It generated a lot of interest. That was a good impetus for me that there was a lot of excitement in the community about this.

If you fast forward a couple of years, I did some more ground work on this thing in 2005 and 2006, got busy with new role I’m in now, and dropped it for six or eight months, and then picked it back upI think it was late last year, late 2006and started looking at how we would get this done.And Ive been shepherding it along since then.Getting this out there really is the realization of a dream that I have been chipping away at for quite some time.I really hope Developers find it as useful as I think they will!

You guys mentioned pushing it out as a zip. One of the real problems was that I did not want to have a situation where every team that did this put a zip file or MSI out every four months, so the customer had to go download, install, and set up his little studio to use. What if you have a hotfix or different version? You know what I mean. I did not want to have a situation that was going to be cumbersome for us to ship and for customers to consume.That would kind of defeat the purpose.

What really got me rolling on this was I started talking with the guy who developed this technology we use internally. Internally, you can get the source and demos for almost every product we build here at Microsoft off this thing called “Source Server.” Source Server has versioning built in, making sure things do not collide so you can have side by side versioning, and it all works very smoothly. The server side piece for source server is just a web server. There is no magic there so its very simply to deploy. There really was not any barrier to making to using that technology externally.

So what we able to develop was a system where teams could publish source very, very efficiently. After a first source publish, they can continue to do source publishing at near zero cost to them, which is great. On the customer side, you have a scenario where a customer has a few simple set up steps. Then they kind of auto magically get source and symbols for what we have chosen to make available at that time for free. No searching around on MSDN or having to deal with where to put the stuff on the machine.

Scott: It sounds like, too, one of things that you are saying this accomplishes is it sort of knows exactly what the client is running and building against. If they have installed the service pack or have not installed the service pack, they get the right source. They get source that matches up with their environment.

Shawn: Yes, if it is on the servers they can get it. One of the decisions I made in getting this done was that, the cheater thing I did was I said, “You know what, I ‘m not going to define policy on this. I’m going to get a system developed where individual product groups and teams can make decisions about how often they push source.” Meaning, does it make sense to push source for every new release we do? Maybe, I dont know. Does it make sense to push it for every service pack? Maybe, I do not know. I think those might be different for different teams. I think we are going to do learning in working with the community about what really makes sense there. However, the system is set up to handle it, in any case.

Scott: Like you said, generally people were receptive to this. Were there internal objections that had to be overcome? Where there people who had concerns that, “Hey, this is crazy. Why would we do this?”

Shawn: Well, if you go back into 2005, I think I had many senior people that I knew, vice president level people that were generally supportive of this, but very cautiously supportive. For me a little bit, one of the reasons it’s been a multi year effort is I had to push carefully and let some perspective change as the industry has changed and the environment changed over the last couple of years.

It has gotten significantly easier. Something happened, I think, in the greater consciousness of the industry over the last 12 months or so. A lot ofI do not want to say a lotresistance is too strong of a word. A lot of the real cautiousness and kind of by-default-no talking-into-yes stuff dissipated and switched to actually a lot of enthusiasm. We need to be careful about protecting the IP that we spent billions of dollars developing. However, I think people really see the value of this now. It has been great for us and for our customers for a subset of products. Now I have seen teams are signing up to push their code out really quite enthusiastically.

Scott: With the.NET framework, it is in a little bit of a unique position because, on one hand, Microsoft has invested a lot in it, and generated a lot of IP. However, on the other hand, it was never really a secret what was in the framework anyways. You could use reflector and other tools to disassemble what was in the framework and sort of figure out how it was doing what it was doing. However, it did not enable this “step into it while you are debugging” experience. It did not make it super easy to learn from the framework. This might have been a little unique in that the IP was not all that protected to begin with.

Shawn: Well, yeah. Yeah, it is funny. The case that I built for this, with managed code in particular, definitely had that aspect to it, that our level of secrecy around how this stuff works, as well. One of the problems with reflector, using reflector, the EULA does not really allow for it.

Scott: Ah, interesting.

Shawn: The EULA does say that you are not supposed to reverse engineer. That said, I felt like it was a good idea to give customers a way to do this within the license, which this is.

Scott: Hah! Yes, I guess I never really realized that that might be in the EULA. I think, especially among the more advanced developers and in the blogs, and in all that kind of stuff, it is such a common practice.

So, now that you have done this, you have taken this first step and you’re going to make source code available for certain subsystems of the framework, and that’s on a feature by feature basis, right?

So, Windows Forms will decide when they push their source code in, ASP.NET will decide when they push theirs in, am I understanding that right?

Shawn: Well, we have pretty much what you would consider, if you go back to.NET 2.0 and think about what made up the Framework at that point. We’ve got pretty much all of the big pieces there. We’ve got BCL, Windows Forms, ASP.NET, Data and XML.

So those guys are already taken care of. Therefore, we’re going out with essentially what is the original.NET framework. The other stuff will fall in. It is really a matter of me getting with a given team and through the process. The code preparation process is actually fairly labor intensive. It takes time to schedule that and manage it.

I had teams that were able to do that. A lot of teams have been obviously really busy getting VS 2008 and.NET framework 3.5 finished up. We will continue to push out more code as time goes on. We wanted to get the big ones first.

We also managed to get WPF in there as well, so its not just the older stuff.Lots more is in the pipe to come online as time goes on.

The choice was either we wait a couple of months to do this, and make everybody wait, or we get the big ones out there fast and then we push in the other stuff as we have time, and as it comes online.

Scott: So one of the things that is kind of interesting is, you’ve got these individual developers in Microsoft and they’re writing code. They have worked in Microsoft for a long time and they have a feeling that, you know, what happens in Microsoft stays in Microsoft, I guess, for lack of a better term.

The whole world is not going to see the line of code that they are typing right now. Now, with the source being released, how do you think that is affecting, if at all, how individual developers are doing their job? How product teams are looking at how they build features going forward, knowing that the source for it is going to be out there.

Shawn: Even product teams that are not the framework team, they are thinking, that’s great, they did that over there, so does this mean that someday my source is actually going to be out there? Yes. Yes. They will be thinking that. And they should be thinking that.

In fact, one of my to do list items is to do some broader communications across our Division about that exact point Software developers are pretty much the same around the world. Software developers, in general, always try to do the right thing. They are also always, in general, under a certain amount of time pressure. Anything that is going to encourage them to take a little extra time to do something right, in the kind of capital ‘R’ right, way, is a good thing.

Whether that’s testing, whether that’s static analysis like FxCop, or whether that’s security reviews or code reviews, or whether that’s the code is going to be visible to their peers or the public, I think those are all good incentives for software developers to think extra hard about how they do things.

So, you know, honestly, we went through the code, we’ve done these code reviews, and these code scrubs and we found a lot less stuff than we thought we would.Actually, we found almost nothing of any real concern, which was nice.

Sean: Are there other things that Microsoft is thinking about exposing around the development process?

There are elements like threat modeling, specs, test scripts, all kinds of things that Microsoft might be thinking about exposing. Are there any thoughts or plans along those ways about how far this maybe goes, eventually?

Shawn: That is a really good question. No is the answer. [laughter]

Sean: You’ve accomplished enough by ruining one Cabo vacation. Right? You can put that post up later, you know, on your next vacation.

Shawn: I think that stuff…that is a good point. If you start doing, a curve on the cost for us to do that versus what percentage of customers that level of access is going to be valuable for. For the threat models, and even for the specs, I think that is a harder case to make. I think those things are also harder to vet, or release, than the source is. They are also less useful to a broad set of customers.

Sean: Well, one of the things that crossed my mind was test scripts, and test routines that Microsoft creates because that would be helpful to others just from an educational standpoint. That does not really expose Microsoft to too much more overhead. I am not trying to pitch the idea here, But go ahead, Scott, what were you going to ask?

Scott: No, same thing. Just, Microsoft has given an inch so watch us try to take a mile.

Shawn: Oh, yeah, we know it. Absolutely.

Scott: A lot of times that’s what people want. They say, OK, there is this API. I want an example for how would use it. I would go look in the help. Well, that example is not what I really want. Microsoft I would assume writes really comprehensive test suites.

Shawn: I do not know if the test code is what you want, either. We call it auto PMEs. The PME stands for Property Method and Event. It is super granular. A lot of that code does not relate to any real scenario. It is just wide unit testing code. There are other types of tests as well. A lot of the stuff is that. The challenge of the testing stuff.

I actually think that the testing stuff is a good point, and an interesting thing and I have been thinking about, not really in terms of us releasing test stuff for previous products. I will get into one in a second. The question is going forward, is there a way for customers to leverage the kind of stuff we do in tests more broadly?

However, the reason, going backward, most of our test stuff is written using internal test harnesses and internal tools. There is this huge slippery slope there. You need this and you need that, and it would be really a lot of work.

Scott: Right, right, so they would have this test but they would not be able to hit F5 on it and actually watch it run anyways.

Shawn: Yeah, and the tests probably call into a bunch of other libraries that are internal that we would not ship externally anyway. The tests themselves might not make any sense. Most of the tests are stored in this big system called Mad Dog, which is kind of a giant database of test cases and test code, and I do not even know how you go about extracting code from that thing.

[laughter]

Sean: So Shawn, what do you think aboutthis is just something I was noodling on, trying to think of the right way to box it in given the time in terms of a questionso, Microsoft’s a closed source company, right. By definition if not necessarily for the sake of our interview. What do you think about closed source companies perhaps at times being better able to support the exposure of the source than maybe a company or a project that simply says ‘Well, just go check it out from subversion, from xyz open source project?’

Shawn: Right.

Sean: Because I look at the Visual Studio integration, and I look at robably how this will eventually weave into PSS, interacting with customers, and saying well, here’s an odd dynamic: is a closed source company perhaps better able to actually make the exposure of the source code, when they do it, a more positive experience that actually lets people leverage it more easily? I mean, absent the ability to fork, and some of the more foundational principles of Open Source, how an outside developer would use it day to day development and how a closed source company in this case Microsoft could support that.

Shawn: That is a great question. I think that the answer isone of the things that I’ve been trying to do, and that we’ve been trying to do, is look at the changes in the industry that open source has driven and really think hard about what the real value adds are there. Is the value add really that you can recompile your kernel, or is the value add that you can debug it and understand why it’s doing something?

And there are obviously cases where the answer is ‘both’ or ‘one or the other’. However, for I think the vast majority of folks, us being able to deliver the source in a way that’s easy for folks to consume and step through as a way to make them more successful on our platform is a really big win. I think that it fits well into our overall strategy of kind of keeping developers integrated in our overall processes, and it fits well into this another milestone of transparency.

We have kind of been trying to figure out ways to open up the envelope of transparency with Developer Division particularly and I think this fits into that portfolio really wellas well as the portfolio that the Shared Source Initiative folks are building. You have CodePlex on one end which is full open source, then you have kind of our proprietary source system on the other end, and there’s a nice continuum in between. I think that we do bring some assets to bear which are difficult for companies that are purely open source or service model driven.

Sean: OK, cool. Well, that is all I’ve got, and also to be sensitive to timeScott, do you have a closing question or two?

Scott: I think that’s a good note to leave it on. The thing that we always end with is: are there things that we didn’t cover that you think are really important for people to know? So, we’ll just kind of hand you the microphone at the end if there’s something you want to get out that we didn’t specifically ask about.

Shawn:Nope, I think we covered it all.Thanks guys!

Bookmark this:

Update: It seems that this isn’t seen as happy news by all. There’s an article on eWeek that’s just too irrational and frothing to pass up, claiming that this is all a ploy by Microsoft to kill Mono. As Microsoft is officially supporting Novell’s efforts in porting Silverlight to Linux (on top of Mono), the evidence would indicate that Microsoft is doing this to support .NET developers, and not as some clever conspiracy to kill off Mono.

Bookmark this: