Interviewee: Gwyn Fisher

In this interview we talk with Gwyn. In specific, we talk about:

• The value of automated code review
• The advantage of code review at the developer level
• Judging the security of open source codebases
• Coordinating bug resolution between distros and upstream projects
• Educating users about vulnerability in the face of complacency
• The changing face of computer attacks
• Why isn’t everyone using code-validation tools?

Sean Campbell: Gwyn, why don’t you start with your background and tell us a bit about Klocwork?

Gwyn Fisher: Sure. I’m the Chief Technology Officer at Klocwork. I’ve been with Klocwork for a couple of years now, and as a consultant, actually, working with them for several years before that.

Klocwork was founded in 2001 as a spinoff from Nortel Networks. The Nortel research group that created our basic technology was centered on the head of the CASE department at ISPRAS, part of the Russian Academy of Sciences and affiliated with the Moscow State University, who was very interested in the notion of very large scale software architecture, model discovery, efficiencies and inefficiencies in incredibly large source code bases.

As it happened, Nortel had a specific business problem that they were trying to deal with that required massive re-engineering on one of their switch operating systems, and so as part of the existing partnership between Nortel and ISPRAS, he was funded to help solve this problem. Following the spin-off and creation of our company, he and most of the team relocated to North America, where many of them continue today to be the kernel of the ever-expanding domestic product development group within Klocwork.

We stayed in that space for several years, helping customers in very similar situations, who had requirements for re-engineering very large, very old, very complicated code bases, in order to simplify, componentize, and improve their software for newer devices, reuse, maintainability, and all that good stuff.

And then some of those same customers–and these are all very recognizable brand names–came back to us and asked us to do defect analysis for them. So, in addition to finding things that are wrong with their architecture and the fundamental way they’re building their software, we also started locating things like where they were leaking memory, doing bad things with pointers, generating buffer overflows, and other basic software problems.

After a couple years of research, we brought the first iteration of a product suite to market for defect and security vulnerability location and analysis.

So, since we were founded in 2001, we have sort of morphed over the years, away from a pure architecture and model discovery kind of technology, into a much fuller footprint in terms of what we do around defect analysis. That includes operational defects and security vulnerabilities, as well as continuing to emphasize our strength in architectural and maintainability issues.

There’s also that whole backbone of model discovery, visualization, and re-engineering–all the great stuff our customers need to be able to do in these very large code bases to manage them over time. We help them keep building out across more languages, more platforms, more tool chains, and all the good stuff that goes with working in a production software development environment.

Scott Swigart: What are the things that are really hard to find with standard code reviews and QA that tools like yours can quickly make apparent?

Gwyn: The quick, sort of palm-to-forehead, kind of “Why wouldn’t we have found that? Why didn’t we think of that?” sort of issues tend to be in the area of what is called inter-procedural analysis.

If you have a very large system, it’s made up of a whole bunch of functions, methods, and modules all linked together into large system images. And the problem, whether you’re using code review, runtime testing, or whatever, is finding enough combinations so you can trace through, from source to sink point, exactly what’s going to go on with a pointer, buffer, array, or whatever it happens to be.

We can point out across multiple different function boundaries, that this pointer you’re grabbing from over here, you’re using inappropriately over here, or this memory you’ve allocated here, you’re never releasing on this particular thread. Those sorts of instances would require your average high level architect to really bend their mind around them for days or weeks at a time to try and find manually. They’re the sorts of things that we can point out to someone and they can understand it very easily within 30 seconds or a minute.

Obviously, we all bring our own biases to bear, in terms of how we think about our products and how we think about our tools. And it’s perhaps obvious to think about a tool like this as a QA tool, and certainly the heritage of tools like this has been very much driven by a downstream audit or an adjunct to code review. But, really, that’s not where this technology comes from–this is a developer’s tool. The first one of these was Lint, back in the ’70s.

And as a developer’s tool, well, Lint’s got a pretty bad name. Let’s face it.

[laughter]

As sort of a history, we’ve all sat there and watched the CRTs desperately trying to refresh as acres upon acres of warning messages come out of these things. But when tools like Klocwork bring modern analysis technology and accuracy and complexity into that equation and deliver it back to the developer, all of a sudden you’re not just inundating them with all kinds of stuff that they can’t deal with. You’re actually giving them a few very useful bits of information before they check code in.

That’s our approach, instead of trying to kick in downstream and then figure out who the right developer is to blame, instead our goal is to build analysis much more organically into the development process so it becomes an enabling environment rather than a blaming environment.

The uptake within an organization changes overnight, as soon as you can get through that cultural issue. Rather than giving QA a new tool for telling the developer that they’re an idiot, we are giving the developer a tool that will help them prevent being called an idiot. [laughs]

Scott: In some of our conversations with Microsoft, they’ve talked about modeling tools that they use similarly to what you’re describing; before code ever gets checked in, the developer does analysis of the code and fixes issues that are flagged.

Talk a little bit about the benefit of having that done at the developer level versus, for instance, in a QA process that happens once or twice a year.

Gwyn: If you do take a milestone approach or do it just before you roll out, the challenge tends to be that you get an overwhelming number of issues to deal with, often all at a critical time in the product life cycle. From the individual developer’s point of view, it can be an awful ordeal.

Scott: In other words, the developer would tend to build up a big pile of issues that all come back for resolution at once, when they are periodically unearthed. On the other hand, if the issues are dealt with proactively, you never accumulate that big, ugly pile of bugs.

Gwyn: Right. The other thing to consider is that this ongoing system means that the developer is dealing with mistakes that they made just recently, instead of six months ago, so it’s still fresh in their mind.

Sean: You mentioned that you guys do some work with different open source communities. Educate me about your input into their process–are there open source analysis tools that are commonly used?

Do you find that developers are running tools that they have available prior to submitting code to the mailing list, or is it more the case that they really just trust the mailing list and are also interested in the scans that you and other vendors do and what those uncover?

Gwyn: There is no doubt about the wisdom of the thousand eyes approach, particularly for the more mature open source projects, and people do use as many techniques and tools as they can get their hands on.

In the Java space, there’s a very nice open source analysis tool—FindBugs–that can really help developers get started with analysis. It doesn’t perform inter-procedural analysis, so in the words of the tool’s developer, they’re really focused on “low hanging fruit”, but it’s definitely a great place to start.

In the “C” world, well, not so much. There are a couple of tools that check coding style and things like that, but while they can tell you things like you should declare your copy constructor as private or you might generate memory leaks, their ability to find actual defects (as opposed to poor coding constructs) is very limited, unfortunately.

Sean: And sometimes, it’s really hard to spot with the naked eye looking at apps committed to a mailing list.

Gwyn: Exactly.

Sean: This is in kind of a different area, but given your perspective and what you’ve done in terms of looking at a variety of codebases over the years, what tools do you think organizations should utilize to ascertain whether a given open source project is secure?

That’s assuming they’re not going to go buy a tool, do a massive code review or utilize a third party; they’re just out there having the internal conversation of Apache versus IIS, or MySQL versus SQL Server, or MySQL versus Oracle.

Different people make different claims about which is more secure, but how do we prove it? You end up in this kind of absence of evidence argument, I think it’s termed. Scott did a look through one of the mailing lists for one of the major projects, and he didn’t see anything about threat modeling. That doesn’t mean it’s not occurring, but that also doesn’t mean it is.

What’s your take on that? What should an organization do to really ascertain whether a given project has a robust development lifecycle around security?

Gwyn: To be blunt, they should do everything they possibly can. You just have to assume that the guys on the other side of the fence haven’t done it yet. Do you believe Coverity, who says their favorite 11 open source projects are perfect? Or do you believe Fortify, who comes out two weeks later and says the sky is falling and they’re all insecure? Do you believe Mozilla, who says they have a thousand eyeballs looking at their code all the time? Or do you believe the users who say, “Yeah, but I filed almost as many bugs against you as I do against Microsoft”?

Nobody has the answer to this question. I’ve talked to several people over the years who said, “Wouldn’t it be nice if we could put together an indemnification agency that would use tools like yours to help some of our customers have more confidence in open source?” The idea would be to say, “If you want to use Apache (for example), here’s everything you need to know about it. Here’s the support protocol around it, and most importantly here’s a security rating for it.” In essence, they’d be attempting to provide a warranty for a particular revision of a particular open source project.

There are various organizations around the country and around the world who do little bits of this, and some of them are quite successful, but no one has ever gotten to the point of taking responsibility for saying “Tick or cross: you should use this.” It all comes down to providing a lot of data for the customer to evaluate, because no one is willing to take on the legal liability of saying what people should consider safe to use.

Because of all the sources out there and because all the tools and techniques are out there, you’d assume somebody would at some point do it. But there are enough litigious arguments against it that I don’t think it will ever happen, unfortunately.

I think it would be a great thing to do, and I think that most of the senior committers on those mature projects would want to see that happen as well. But they’re giving their time for free; it’s not something they’ll ever want to do themselves. And I think any commercial entity is just going to be scared witless of some bank somewhere going, “You know what, you told me to use X.Y.Z of Apache, and it turned out it had this vulnerability in it. We’d like you to compensate us for Grandma’s password finding its way to the Ukraine.”

Scott: We’ve talked a little bit about the value of static analysis to define things that aren’t necessarily easy to see with the human eye, and the value of doing it up front. One of the interesting things I find in open source is that with a proprietary company like Microsoft or Oracle, for the most part they own the code from soup to nuts. If there’s a bug, there’s a place to file the bug. They fix it, or they don’t fix it, but it’s their codebase.

When you’re dealing with something like a Linux distro, on one hand you’ve got this downstream distro that’s packaging a lot of upstream projects. And so, you’ve got the distro provider who has hired people to do the packaging and things like that. But, they’re also paying people to work on these upstream projects, whether it’s the shell, the web server, the kernel, or whatever. Then, when a vulnerability happens, there’s a relationship necessary for a fix posted to the distro to get posted to the upstream project.

Distros also talk about things to make these upstream projects enterprise ready. In your experience, do you find at the distro level that people are using tools like yours so that if an audio player for example didn’t go through the same kind of analysis upstream, they maybe are doing it downstream and working with the upstream project to improve the quality of the code?

Gwyn: Wouldn’t that be a good way of organizing life? Unfortunately, it doesn’t happen so much. Some of the distributions today are backed by serious individuals or serious companies. It’s not just two guys off in Europe trying to bang a CD together, and in some ways, they’re doing phenomenal things. Look what Ubuntu has done around the user experience. They’re doing amazing stuff for the operating system as a whole.

I would say the distro manufacturers as a whole certainly validate in an industrial strength way and do the needed extra development, but they don’t really take responsibility. In terms of indemnification, guarantees of quality, or anything like that, they tend to fall back on a very dinosaurish kind of mentality. Got a problem? Hey, it’s only a patch, suck it up.

In just the same way as indemnification players don’t really want to step up and say it’s secure, distro vendors really don’t want to take this step of guaranteeing not just observed quality but absolute ground-up code quality, because of the exposure that would entail with all kinds of people over whom they have no control. Who do they go to if they have to build a fix on some kind of SLA? Of those thousand eyes, half of them are asleep at any given time. And the other half work for somebody else.

I’m certainly not saying you should go and buy commercial software all the time because you can always go and get hold of somebody. At the same time, though, the community just doesn’t have the level of leadership yet that lets it take that particular kind of responsibility for the quality and security of the software they’re producing. Perhaps the fact is that there’s just not that level of requirement being surfaced by the enterprise customers.

At the end of the day, nothing’s ever free. I think that open source has absolutely proved that software can be as free as you like, but that’s just one small component of the total cost of ownership of these things.

Scott: I imagine that you run into the idea of a security IQ that starts out fairly low, where people say “Well, we haven’t had a problem, so we must be doing things right.”

Gwyn: [laughs] And my five users are all very happy with it.

Scott: Right. So, how do you approach educating somebody about the fact that just because nothing has blown up and it hasn’t been the end of the world, they might not want to assume that everything’s hunky dory.

Gwyn: It is actually fascinating how you can never feel anybody else’s pain. I think that’s a rule of life, whether it’s physical or intellectual. You can’t actually empathize with all the people who are going through it until you’ve done it yourself.

At one point early in my career, we got this very polite email from the DOD saying that they had taken 65 servers offline because they found a vulnerability in one of the products I was responsible for. And this was back ages before anyone was really worried about such things.

It was a tremendous wakeup call to realize that we had just taken away a significant capability from a very important agency just because of an error on our part.

They were very polite about it and didn’t intend to cause any significant commercial ramification, but as professionals, how were we to get to the root cause of it? How would we stop it from happening again in the future? Until that moment, I had never felt that pain, and I didn’t really understand it.

Increasingly, we get people coming to us from the device space to ask specifically about security. And they’re not coming to say “OK, we know we need to worry about this;” it’s “why would I have to worry about this? Everyone’s talking about this and I don’t get it.”

So, we were talking with this one medical device manufacturer, and they make truly scary stuff. If this stuff goes wrong, it’s not going to be happy days for anybody. Their senior architect was sitting in front of me and he said, “Marketing tells me I have to put an IP cable on the back of this box because he wants the paramedics to be able to send stuff off to the hospital before the guy gets there.”

And I asked him, “OK, and what are you doing about making sure that your device is doing what you think it’s doing and hasn’t been hijacked or something?” And he responded, “Why would anyone ever want to do that?”

Explaining to people that just because they would never think to go and hijack somebody’s super-fridge or their pacemaker or whatever it happens to be, doesn’t mean somebody else out there doesn’t have too much time on their hands and would just think it’s a huge joke.

In many ways, it’s that first moment of enlightenment that’s the hardest for people. You’ve got to get across that notion that just because they would never do it, and just because they can’t understand why anybody would, doesn’t mean it’s not going to happen to their system or device.

Once they’re through that, then it becomes a very pragmatic process, but that very first thing you’ve got to have is for that senior architect, VP, or whoever to feel the pain.

Scott: It seems to me that in a way, Microsoft got a little bit lucky (though I’m sure they didn’t feel that way at the time) back in early 2000, when Code Red, NIMDA, Blaster, and Slammer came out. These really high profile exploits were very embarrassing, and they were the kind of exploits that brought a whole company network down. And so, they decided to just pour dollars and resources into solving the problem.

Today, it seems like the nature of exploits is very different. They are far less likely to absolutely saturate your network and let you know that you’ve been hit. It’s more about setting up a botnet and what some people call crime as a service.

It’s a lot more valuable to have Sally not know her computer’s been compromised, or to have some Linux server in a data center doing command and control for a botnet, without its admins knowing about it.

Like I said, Microsoft got lucky in the timing and the nature of the exploits. In some of these projects, there’s a false sense of security because they say “Well, where’s our NIMDA? We’ve never had one.” Well, they’re never going to have one, because that’s not the kind of exploits people are writing.

Gwyn: Yeah, exactly. Over the last ten years, we have moved from script kiddies doing dumb things to these really very, very smart attack vectors. It is a whole new day.

We can’t go a whole interview without me poking at Microsoft. [laughs] Look at the animated cursor type of thing that hit the public eye a year ago. This was vulnerable code that had been in Windows since day one, but all of a sudden, out of nowhere, you don’t have to see the person, you never have to see their machine, you can do everything remotely, and all of the sudden that thing is yours.

And what’s more, if you’re smart enough, the person who’s getting compromised doesn’t know what’s going on. It sort of became that perfect storm of an attack vector. And at the end of the day, what is it? It’s a simple exploit of a buffer overflow vulnerability, albeit a phenomenally complicated one. Those guys should get a PhD just for creating those things. [laughter]

You’ve got to give them kudos, right? But, that happened at the height of Microsoft’s secure development life cycle, and that bug had been reported. It was in their queue, and they knew all about it, but it had been de-prioritized and was buried in a pile of sixty eight thousand other things they had to worry about.

So, they are at one end of the spectrum because they have so many products and so much going on, but if you take the whole open source domain as a thing, there’s probably more code there. And you know that same level or magnitude of vulnerability is out there.

Scott: That’s one of the benefits of tools like yours. On one hand, there’s putting it in the hands of every developer so that they can have a ‘get clean, stay clean’ sort of mentality. But, the other value I see is that the most dangerous code in the world is probably the old code that nobody is looking at anymore, because it was written before people knew about integer overflows and buffer overruns.

I remember, way back when, we bought these Unix systems from a vendor. We were writing some low level network code. We did something that crashed the server. We had a bug in our code, and it was sending a packet where the frame was the wrong size or something. The response from the Unix vendor was basically, “Hey, there’s an RFC, and if you’re operating outside that and bad stuff happens, that’s not something we’re going to worry about.”

[laughter]

Gwyn: There you go. Good rationale. On that same tack, IBM’s X Force does all these amazing things all the time, like showing how forcing an invalid pointer usage can compromise Flash applications, etc. — seriously smart people.

One of the things they do is to issue a “top ten” offenders list for attacks during a year and something that struck me was, I think, in last year’s report. No prizes for guessing who is the topmost vulnerable vendor, because Microsoft wins every year. But, of all the reports that they had coming to their NOC over the year, Microsoft products were responsible for three percent. Three percent of all the reports had something to do with Microsoft software, somewhere along the way.

The fascinating thing I saw was at that number five or number six–I forget exactly where they sat–was Mozilla. How many products do they make? They have one fantastic browser I use every day, and some other random stuff that is much less popular, but they are responsible for 1.4 percent, 1.3 percent … don’t quote me on the figures, but some relatively close number of available vulnerabilities, compared to all the things that Microsoft puts out.

Now, in reality, how many of the things that Microsoft puts out are actually getting attacked at the same level that Firefox is getting attacked? But still, it makes you think… Oh, and that top 10 was only responsible for about 15 percent of all reported attacks, so over 80 percent were against software and vendors you’ve never heard of.

Scott: Well, I want to be sensitive to the time. We’ve covered a lot of great stuff, but what have we left out?

Gwyn: If I were in your place doing the interview, I would ask what’s wrong with these tools? At face value, they are no brainers. Everybody should be using them, so why not? What’s the barrier to acceptance? Is it just because they are very new? What’s really going on here?

We try to answer this question for ourselves all the time. Why is nobody in this market a Microsoft yet? Why are we still a very nascent segment? Obviously, there are some elements of market maturity there. Most of the companies in this space have been around for a very short time.

It doesn’t matter which community you are working with, whether open or commercial. There is an overwhelming thought process involved around analysis. People tend to look at this process the same way that we started this conversation about it today–thinking of this as an audit environment.

People tend to regard it as a milestone activity and as something that is an onerous, annoying part of the software development process akin to code review. I can’t imagine anything more mind numbing than having to revisit the days in which I did code reviews all the time.

If this kind of technology is equated to that area of things, that’s a problem for this industry, and for the service technology type. It behooves companies like us–and open source projects who are trying to do the same sort of thing–to get over that, and to engage with the developer in a meaningful discussion around what you actually need out of one of these tools.

Is it that you need the most accurate tool with the lowest noise ratio? Or do you need something that finds every bug imaginable, regardless of the noise ratio? Do you need something that’s really fast, or that integrates with every environment, or something you never see and just pops up when you do something dumb?

For various vendors in this space, we have markedly different models as to how we engage with development organizations. As for ourselves, we are very upstream. Everybody we ever talk to is in development. Other people in our space are very downstream. They talk to auditors and CFOs–guys who in the morning are buying firewalls, and in the afternoon, they are buying software validation tools. There is a completely different feel to how you engage with a development organization, when you start getting that viral distribution down at the volume end of the buying pyramid.

I think it’s an interesting challenge in this space–how to get that massive distribution. When did IDEs suddenly hit the tipping point from being a bizarre luxury that a few companies used, to being something that every developer on the planet sees as a right and a prerogative?

It’s the same with runtime profilers. It’s the same with memory profilers. It’s the same with all these tools, which are just part of today’s normal, everyday SDLC.

When did they get there? How long had the market been around? Is it just a volume-based thing? Is it a market approach thing? I think that’s the fascinating question in full scale analysis now. Obviously, it has value and is having great effect in both commercial and open source, so when does it go big time?

Scott: That’s a great closing thought. Thanks for taking the time to talk with us.

Gwyn: You bet. Thank you.

Bookmark this:

Interviewees: Brian and Larry, and Michael.

In this interview we talk with Brian, Larry, and Michael. In specific, we talk about:

• OpenID’s offerings for decentralized online identity management
• Securing online identity: technology and beyond
• Aggregating identity information for multiple resources
• Spurring adoption by developers, service providers, and web sites
• Responding to objections about centralized identity management
• Closed source versus open source credibility
• The future of online identity management

Sean Campbell: To get us started, could each of you introduce yourself and tell us about your current role as it relates to OpenID and JanRain?

Larry Drebes: I am in engineering, and these days, I spend most of my day thinking about our software and how it relates to the market needs.

I was one of the co-founders of Four11, which produced a product called RocketMail, the underlying platform for what became Yahoo! Mail. Yahoo! bought RocketMail back in 1997, shortly after Microsoft bought Hotmail.

After that, I co-founded a company called Desktop.com, which was earlier than Google Docs, but similar in concept. Most of my background is with large-scale software, in software as a service deployments.

Mike Graves: I’m the CTO. I’ve been here a little over a year; I was previously at VeriSign, where I was CTO of one of their two divisions. I arrived at VeriSign in 2000 as the founder of a dot-com-era startup company called Signio. It became the merchant-payment processing system for VeriSign until they sold to eBay last year.

I took a little time off and came back to VeriSign to help lead the charge into new markets where I first became familiar with OpenID, and eventually ended up here at JanRain where we’re exclusively focusing on OpenID.

Brian Kissel: I’m Brian Kissel, the CEO. I joined the company in January of this year at the point when the team felt like OpenID was really starting to reach an inflection point of market adoption. The OpenID V2.0 specification had been recently finalized, large corporate sponsors like IBM, Google, Microsoft, Yahoo, and Verisign were announcing their support, website adoption growth was accelerating, etc.

We wanted to take advantage of the extensive development work that the team here had accomplished over the last couple of years in creating next generation platforms and technologies for user-centric identity and authentication management via OpenID.

Scott Swigart: Thanks. Give us a little bit of a description of OpenID, and what you as a company do around the technology.

Brian: OpenID is the equivalent of single sign-on that you would get inside an enterprise. If you log in at the beginning of the day via Exchange or something like that, there’s an abstraction layer that also logs you into backend systems like CRM, ERP, HR, Accounting, and so forth. You only have to log in once.

You may be familiar with Microsoft’s Passport and InfoCard technologies, which is the idea that you have one user name and password that you can use on any enabled web site.

That’s really the same concept behind OpenID–it’s for users on the open Internet to have just one or a few identities that they can use to access all web sites that are OpenID-enabled.

Scott: And in terms of your company, what are some of the things that you do around OpenID?

Brian: Our goal is to be the platform provider of choice for OpenID implementations, for the end user, the OpenID provider, and the web sites that are accepting OpenIDs for registration and login.

In OpenID parlance, there are two kinds of contributors to the OpenID ecosystem that benefit the end user. One type is called the relying parties (RPs), which are web sites that accept OpenID for authentication. The other type is the OpenID providers (OPs), which are organizations that issue OpenIDs to consumers, employees, or members to use on the web sites.

We have a property called MyOpenID.com, which is a hosted ASP service where you can come and get an OpenID. We also offer services to web site operators to enable their web sites for OpenID and to create better, more intuitive login user experiences.

And then we have solutions that we offer to OpenID providers who want to issue OpenIDs to their employees, their partners, or their customers with their own brand.

Scott: Obviously, people who are building sites that may accept OpenID are using a variety of technologies. What kinds of technologies do you support, in terms of the components that you provide and things like that?

Brian: There are open source OpenID libraries for about eight different platforms out there, including four or five major platforms.

Larry: We currently support three of those; we used to support six. Very early in the design process, we created open source reference implementations in a variety of languages. For OpenID V1, we wrote Ruby, Perl, Python, PHP, Java, and C# libraries, all with a common API, and tried to seed the market to get OpenID into the hands of developers.

Since that stage, we’ve brought a lot of companies into the fold, and it’s no longer necessary for us to support some of the more niche platforms. Right now, we provide open source libraries for OpenID V2 in Ruby, Python, and PHP.

OpenID fits together between the end user, the provider (which is often us), and the web sites through common web protocols.

Scott: As you mentioned, the idea behind OpenID is similar to what Microsoft was trying to accomplish with Passport. What are the advantages of OpenID versus a Passport implementation?

Mike: The crucial factor is that there are a lot of technical things inside of Passport that are very nice architecturally, but politically and socially, it’s just not an option.

The ecosystem will resist any company owning that space in that way, no matter how good the technology is. No one wants to be locked into a silo at that level of the stack.

One extremely valuable thing that OpenID offers is its decentralized nature. The trust and provider infrastructures are decentralized, so anyone can spin up their own OpenID server.

That doesn’t mean they have to be trusted or that anybody is going to trust them. But the trust circles and trust relationships arise organically out of that, meaning that if you use OpenID, you have portability and the ability to avoid lock-in. You can move to another provider using the same technology, and let them compete against each other in terms of serving the user.

That’s not the only difference, but it’s crucial that no one owns this, and the turf is not proprietary at the lowest level.

Scott: So since there are a variety of OpenID providers, people develop trust in specific ones, and the providers work to build their reputations.

Mike: Right. A lot of factors go into the decision of which provider you want to go with, and those factors can change over time. With OpenID, however you decide, you’re not tied into that provider.

With OpenID, if you read a headline that gets you concerned that your data is being disseminated somehow that you don’t approve of, or violates the EULA as you read it, etc., you have more options.

That doesn’t mean that it’s not a headache, but the architecture of the system is such that you can migrate to a new provider that abides by the policies or values that you demand in that relationship.

Sean: Would you say that what you are trying to do by federating identity is somewhat similar to the experience somebody gets when they can look at Debian versus Ubuntu?

That is, Ubuntu is Debian with some chrome and obvious other technical differences, but they come from the same lineage, and if I don’t like what Ubuntu is doing, I can go back to Debian, or CentOS, or elsewhere. Do you feel like OpenID shares some philosophical relationships to that approach?

Mike: Yeah. We understand that it gets away from the end-user retail-consumer mentality, but at its core, that ethos is very much the same as what drives the open source philosophy of OSs.

Linux is a very good analogy to use, because for instance, even if there are things you don’t like in a Debian architecture, many users can’t help themselves, but it’s always available. If you really need to change things, you have a path if you want to marshal the resources and effort to change and accommodate your interests, even on your own.

OpenID is similar, in the sense that it’s a roll-your-own architecture. If you’re sufficiently motivated and equipped, you can roll your own to provide biometrics support, or requirements for attribute exchanges that are very stringent or that otherwise meet your needs. And you aren’t beholden to anybody’s particular implementation.

Sean: What do you think the natural evolution of OpenID is down toward the desktop? Do you see people eventually weaving it into desktop scenarios to secure data in home and business settings, or do you think it’s predominately going to stay with service providers and data storage that you access in the cloud?

Mike: My personal view is that part of OpenID’s value is that it’s fairly narrowly cloud centric, and that’s the way it should be, even long term. There are lots of great existing desktop identity management and security solutions. OpenID’s primary design intent was for open Internet identity and access control.

We see OpenID as part of a mosaic or a stack that provides user-centric identity and the security and privacy around that, but I don’t think OpenID has the charter to try and reach from the cloud into the desktop.

We want to tailor things into a modular concept, where OpenID meets its goals very well and in a very lightweight, flexible way, and it defers to other tools that can be plugged in and out, depending on the context, to work with it.

I think it’s far beyond what OpenID needs to address in terms of desktop integration, because as you said, the Linux desktop is going to have a different set of tools than the Microsoft one. As you know, InfoCard is actually a pretty nice solution in this area, but it’s Microsoft-centric. And despite the difficulties of getting beyond Microsoft on the desktop, that’s going to be a sweet spot for the long term.

Sean: Let me ask you a question in a different area. How do you address the man in the middle question? Obviously, it’s a tough potential problem for any identity system, even if it depends on factors that are very hard to spoof, like biometric data.

Larry: As you know, there are general security issues with the web, and to some extent, the world has survived with those security issues. For instance, if I’ve been phished and I think I’m on Amazon but I’m not really on Amazon, bad things could happen.

Certainly, OpenID is living within the web framework, and we can’t do anything to remove existing global security issues, but we can add additional features onto the normal intercourse of the web that provide additional protection.

With our product, we’ve added security measures as options for the user. For instance, the user can use a client side cert.

We have an out-of-band, second-factor authentication solution,”Called VerifID,” which actually uses your cell phone in conjunction with a password to verify your identity. Every time you input your password, you get a phone call at that number, and you must hit a number on the keypad in order to continue.

We also allow InfoCard to be used, which is a sample-based public key encryption protocol. So all those options can provide additional layers of protection. Of course, we have to strike a balance between inconveniencing the user and securing their web transactions.

This is really the same story everyone can tell you–Amazon is making the tradeoff of just using a password versus a more secure option, which might cause more friction against people filling up their carts and checking out.

We are participating with other members of the community, since it’s not at all a unique problem to us, and we’re giving users tools so that they can secure their identity themselves, if they choose to.

Mike: A lot of this is anthropology, as you know. We have to be realistic about how much of this technology actually can solve. There are social relationships and social trust issues that play a major role. I worked at VeriSign for a long time, and they’re heavy into the mediating technology for trust relationships.

But even for a big, badass security company like that, they understand that, at some point, it happens above the technology layer. That being said, it doesn’t mean we’re not cognizant of it, but because we’re cognizant of it, we know that that’s not something that can be solved with just bits.

Scott: Another topic that tends to come up a lot when we discuss identity management is that there are a couple of choices about aggregating that information, and they both have issues associated with them. One option is to have a different password for every single place you go, and nobody really likes that. On the other hand, there’s obvious danger to having one password for everything–if it gets compromised, that’s really bad.

But as I understand it, I can have more than one OpenID, so I might have different identities for different purposes.

Brian: You may have one or you may have many. It’s up to you, and you’ll have different reasons why you’re using different identities, whether it’s a work identity, or a home identity, or an affinity group identity with your college, or some other interest area that you have. It could be an AARP identity. It could be an American Express identity.

The tradeoff with one or a few identities is that you can be a lot more vigilant and mindful of how you manage your identity. You may have 50 different username password combos out there on the open Internet, and most people do something which is called password reuse.

Your password is only as secure as the weakest place it’s used on the Internet, and if it gets phished at Billy Bob’s Phish and Tackle Shop, then it’s compromised, and it’s good anywhere else you’ve reused that password.

On the other hand, in the OpenID model you only share your password once, and that’s with your OpenID provider. And you can make sure that you trust that your OpenID provider is doing infrastructural things to protect it, and that you’re creating and managing strong passwords.

You can be reminded to reset your password with a given frequency, and you can be encouraged to implement multifactor authentication protocols–whether that’s an SSL certificate, an InfoCard, anti phishing site verification tools, out of band authentication with cell phones, or an RSA token. With this approach, you get to choose how much security you’re willing to layer on top of your authentication process, to balance between convenience and security.

That’s what it will always boil down to. Unless every device and every opportunity uses biometrics, and it’s your retinal scan or your thumbprint scan on every device where you would ever authenticate, then there’s going to be some inconvenience to you to do something more than username and password.

Actually, though, it turns out that if you have a good relationship with your OpenID provider, you get very easy login on any OpenID-enabled site – single click with no text entry required in many cases. You maintain a very trust-driven relationship with your OpenID provider, and that OpenID provider can manage all that infrastructure across one account as opposed to many.

If every web site had to implement the same multiple layers of protection for your identity that an OpenID provider did, I would posit that the whole ecosystem would not adopt the same level. But if you have a handful of OpenID providers who are really focused on managing your identity and providing multiple layers of protection, then you’re more likely to have a more secure experience.

Sean: What about the challenges of educating developers? Some technologies are just flat out sexier than others. That’s nothing against anybody in particular, but Scott and I have had the experience of educating people about some pretty dry stuff, and we appreciate how difficult it can be to engage people on it.

What do you do to evangelize to the broader open and closed source development communities about the importance of identity?

What are you trying to bring to market to make it easier for them? I have to imagine that, on one hand, everybody understands it’s important, but on the other hand, it might not be the thing that they want to pay as much attention to right out of the gate.

Brian: The OpenID Foundation along with the member companies are doing outreach and education on the features and benefits of OpenID. We run a web site called OpenIDEnabled.com that hosts resources for the development community. Additionally, we provide tools like ID Selector (www.idselector.com) to enable website operators to adopt and deploy “best demonstrated practice” implementations of OpenID login. We’re also working on white papers for OpenID providers, web site operators, and end users on how to get the most out of OpenID.

In addition to outreach and education to the development community, we’re providing more “turn key” solutions for OpenID providers, website operators, and end users. Just like Sugar CRM, VA Linux, RedHat, MySQL and other open source initiatives, you can get the open source libraries or you can get professionally managed services using the technology. We provide the widest range of innovative, fully featured solutions on the market.

Scott: Where are you seeing OpenID being most widely adopted? Who do you see as being the early adopters, and maybe the second wave of adoption? Who’s using it?

Larry: At the user-created points of the web, like blogs and wikis, it’s gaining tremendous ground and is actually a sort of thought leadership. It’s the right way to do things, and it’s thought of as very cool in lots of circles.

Beyond what you might call the technical early adopter part of the web, if you will, you have large players like Yahoo!, Microsoft, AOL, Google, MySpace, and VeriSign all coming out in support of OpenID. So really, the adoption is expanding. So the top end of the market has already either announced support or delivered production support.

It’s that middle of the Web that our eyes are really on, at this point, and it’s a large middle. That’s where future adoption is going to come from.

Scott: What do adopters point to as being advantages of the technology, to validate their decision to go that way?

Brian: As Larry mentioned, the early adopters have been either the large portals or the user generated content sites, like blogs, discussion groups, wiki’s, social networks, and sites like that, where the primary objective is to get registered users on their site as frictionlessly and as quickly as possible.

If users can show up at a site with an identity that they can sign in with, it’s much more likely that they’re going to become active participants. The big benefit to website operators is that the more people they have active on their sites, the better they can do with personalization, advertising, promotion, and cross selling. Also, for sites that sell advertising, they can get higher CPMs for profiled users than generic site visitors.

So the first measure of success is probably the conversion rate from “site visitors” to “registered users,” followed by the activity level on the site.

Back to your earlier question, I think that the next categories we will likely see come online will be content sites other than user generated content sites. They’ll be media properties, like newspapers, radio stations, TV stations, magazine sites, sports sites, gaming sites, and some affinity groups–organizations where they don’t control the members, so they can’t manage single sign on from an enterprise perspective, but they have members where they do want to provide access.

Alumni associations, community organizations, little league teams, AARP, Boy Scouts, and organizations like that are not necessarily transacting business, and they’re not asking for credit card numbers, but they do want to have registered users.

Another category would be all the customer supported sales and customer self help resources on product web sites, where they have blogs, discussion groups, or wikis to allow people to ask and answer questions and help each other.

If you go to one of those sites and you have to register to participate, and you’ve already got 50 user names and passwords, the likelihood you’re going to participate is lower than if it would accept an OpenID that you already have.

That’s the primary initial benefit for the web site operator–ease of registration and login, and ease of registration is important as well from the standpoint of being quick and error free.

OpenID has with it the ability to transfer personal data at the user’s discretion. Right now in simple registration, there are 10 demographic data fields, but it’s extensible (and we think longer term it will go that way) with a component of OpenID, which is called Attribute Exchange where you can share as many data attributes as you’re willing to publish about yourself. And you can share that information with a web site at registration.

So instead of having to fill out all those data fields, you just pass them to the web site operator in a machine readable format and pre-populate your application. As OpenID evolves, we’re going to get to the point where we’re going to get single-click login.

And that’s something our solutions actually support, so when you show up at a web site, you don’t even have to type in anything. It just remembers who you are, you click to authenticate, and it goes back to your OpenID provider to authenticate you seamlessly, and it comes back and logs you into the web site. So from a user’s perspective, the ability to register quickly and login easily is important.

Longer term, managing that data is going to be important. If you changed your phone number or email address and you had to go back to 50 different web sites and update your web site profile with that information, it would be a very tedious process.

If you have one digital identity, or maybe a handful that remember what sites are using that identity, and you can choose to pass that updated information to the site, that’s a benefit to you, and it’s a benefit to the site operator.

Scott: What are some of the misconceptions or common objections that you run into when you’re talking to people about using OpenID?

Brian: It tends to fall into one of three buckets.

The first is that they don’t know whether their customers want it or need it. The second is that they wonder how long and how much effort it’s going to take them to implement. And the third is that they want to know who among their peers and competitors are using it, because they may not want to be first, and they have to consider how this ranks among their other priorities.

We’re actually trying to systematically address each of those concerns with products and services that we’re offering to make it easier (a) for web sites to implement OpenID, (b) for them to be aware of the number of users who would like to use an OpenID on their web site, and (c) to get a few testimonial accounts in any given category to adopt OpenID.

You can envision the scenario that as soon a few major college alumni associations adopt it, others will follow. As soon as you get the Boy Scouts, you’ll get the Girl Scouts and the Campfire Girls and 4H and everybody else. As soon as you get United Airlines, you’ll get Delta and Northwest. And as soon as you get Hertz, you’ll get Avis and National and Enterprise.

So I think part of the challenge for us is getting those early adopters who see the benefit and want to be perceived as thought leaders. As soon as that happens, the others will follow.

Right now, of the 18,000 OpenID enabled web sites, a majority of them are in that “user generated content” category. Some blogging sites, web sites, and discussion group web sites that are just going live now are choosing to go entirely OpenID.

Some are retroactively going OpenID and converting all their users to OpenID. So in that category, the benefits are compelling, in terms of ease of adoption and deployment. Reduction in customer care costs is another thing we should talk about.

If you only have one or two user name passwords to memorize and maintain, the likelihood that you’re going to forget it on a site that you go to less frequently goes down dramatically.

And it turns out, according to Forrester and Gardner and Meta, forgotten passwords account for 30 to 50 percent of customer care support calls. So if you can drive that cost down dramatically, you drive the customer frustration down about going to a site that they’re only going to less frequently.

Scott: I guess the other advantage for the site operator is that it’s not their problem anymore, right? They just basically shuffle you off to your OpenID provider who handles all of that lost password, password reset kind of stuff.

Brian: There are actually a couple of benefits there. One, you do outsource that customer care cost to your OpenID provider. The other is you don’t need to maintain the passwords, so you don’t have liability in the event that a password is compromised.

Scott: I think there’s an interesting dynamic there, in terms of how the Internet has changed. I think originally, site operators saw it as a purely positive proposition to collect information on their users.

In the last couple of years, they have really started thinking about the fact that there’s liability associated with that data.

I know as a user, I really wonder when I go to a site whether they are really qualified to store my credit card information. Are they really competent to safeguard this information? And if they give me a check box that says: “We can forget your credit card and you have to enter it every time,” I always check that box.

Brian: That’s one of the reasons why PayPal has become compelling on smaller sites where you might not have that trust factor. There’s a way for you to abstract away your proprietary account information.

Sean: What do you think about the ability of something like OpenID to be born from a single closed source company or even a small gaggle of them, whether it was Google, who has tremendous credibility, or Apple, who’s also closed source but has a lot of positive brand equity?

Do you think something like this has to be created by an open source type of community?

Mike: I think that this could’ve been pushed out by one of the big companies. Google, or Yahoo!, or Microsoft has the girth and user footprint to push something like this out.

But the technology has to have some genetic features that those companies typically wouldn’t provide–namely, the ability to easily migrate right off that particular silo into another.

It’s practically achievable for someone like a Google or Microsoft or Yahoo! or AOL, but disruptive to their community base. One of the things that we’ve seen happen is that Yahoo!, for example, has been very proactive with OpenID, relative to some of the other big companies.

But there was a time not long ago that they hadn’t warmed to this idea any more than Facebook or some of the other more closed companies, because they wanted to preserve their proprietary silo. One of the things that OpenID or similar technologies will do is to flatten the namespace and make it easy to move things around and facilitate portable user-managed identity. Momentum for that paradigm is expanding beyond OpenID in areas such as OpenSocial, OAuth, Data Portability, Portable Contacts, etc.

Scott: We’re drawing near our time limit, so I’d like to ask whether there’s anything that we didn’t ask about that would be interesting in this space to discuss.

Brian: One area that we think longer term is going to be interesting as OpenID becomes more prevalent is the notion of consolidating and aggregating and more intelligently managing all the content and communication that happens on the Internet today.

Right now, you have silos like email, chat, discussion groups, blogs, and wikis, with different people, groups, and topics that you’re trying to manage in a more disaggregated way.

Imagine a time in the future when all of your communications and all of your content sites are OpenID enabled, so you can pull them all together in a more consolidated way. You could keep track of various discussion topics, blog entries, emails, and chats in the context of topics that are of interest to you from people and groups that are of interest to you. It would let you prioritize and organize information in ways that make sense to you, based on your stated priorities or what the technology can infer from your actions.

That’s something we think that in the longer term, OpenID will enable that no other solution or approach has done to date. So we’re excited about what it can do for registration and sign on today. We’re more excited about what it can do with digital identity, reputation, and content management long term.

Mike: One of the things that JanRain has to balance is the immediate, practical needs of equipping the industry to provide practical solutions, with the long range opportunity, which is huge. Once OpenID proliferates in a broad way, they will become the effective end points for all sorts of things. Communications is an obvious one, but commerce and trust and the things that spring from them are going to increasingly become factors, in terms of the basic building blocks that are OpenID end points.

For us, that’s an enormously motivating horizon to keep looking at, even if it’s not proper to talk about in this quarter or next, or even next year. But as time goes on, this is going to unleash a huge number of disruptive opportunities for new communication tools, new workflows for collaboration around content, new e-commerce trust mitigation, and e commerce payment flows, all using OpenIDs as the atoms building into molecules and organisms in a way that hasn’t happened before.

Scott: This has been a good conversation, and thanks for taking the time to chat with us. Identity and personal data are central to the work that lots of people are doing, and so it’s really nice to get your perspective on how OpenID fits in.

Bookmark this:

Interviewee: Gabor Cselle

In this interview we talk with Gabor Cselle. In specific, we talk about:

• Deciding between open and closed development models
• Strategies around handling QA challenges effectively
• Maintaining high usability and user security
• Business models and corporate customers as opposed to individual users
• The relationship between open-source and commercial software
• The value of avoiding feature creep

Sean Campbell: To get us started, could you please give us a little bit of background on Xobni, your role with the company, and a little bit of your work history before that?

Gabor Cselle: I’m the VP of engineering here at Xobni. I run our engineering team, manage QA and support, and was the first employee. Previously, I worked at Google as a software engineer, both in Zurich, Switzerland where they have a big engineering office and in Mountain View.

Xobni was started as a Y Combinator company. Y Combinator is a seed funding group that tries to encourage young college graduates to start companies, instead of working for “the man.”

The company was started by a graduate of MIT and a Penn State graduate in 2006. For about the first year, they were working on what we call Xobni Analytics today, which is a package that analyzes your Outlook emails just like Google Analytics analyzes your web traffic. It would chart out how many emails you got this month, how many people you emailed with, and so on.

After a while, we realized that people wouldn’t use that more than once or twice, and then they would let it fall by the wayside. We really wanted to build something that people would use for hours every day, and so we got the idea for today’s product, which is a sidebar for Outlook.

We got funding from Khosla, so after I joined, we built the current product in about six months, and launched the private data in September of 2007. Since then, we’ve been adding features and broadening the project’s scope, and we have some new exciting projects as well.

Sean: Given the overall open-source momentum these days, what was your thinking on making the project closed source from the start?

Obviously, some of it was driven by the fact that you are going to plug into Outlook, but that didn’t have to be a complete limiter, in and off itself.

Gabor: I don’t think it was a conscious decision in the very beginning. It was more about the fact that eventually we wanted to license this to enterprises, because we were working on the Analytics product back then. And obviously, Outlook is closed source–there’s the thin layer around Outlook of APIs that Microsoft makes public, but everything else is closed source.

I think we are just kind of imitating that model, and eventually we may have versions with extra features as something that you can purchase. I think that’s why it’s still closed source, and we’re probably going to continue to follow this route.

Sean: I guess the natural follow up to that is to ask what kind of push you have gotten from people to extend what you are doing with Outlook and put it in Evolution or Thunderbird?

Gabor: Actually, the day after we released, we had about 400 emails in our support queue that all said, “I’m not an Outlook user. I’m a Thunderbird user,” or “I’m a Gmail user and I really like the functionality,” etcetera.

So this is definitely something that we’ve looked at very, very intensely. We decided not to pursue the path of integrating with any other desktop clients, at least for now. We would need to revise a lot of the interface, but the main reason is really that Outlook is the number one email application in the world. There are 400 million people using it, and a lot of these people are in pain.

Outlook doesn’t do what they want it to do. They can’t search for messages quickly, especially with the older versions. We solve that one pain point really well for Outlook. It’s a huge addressable market, so that’s why we are sticking to it. The future might hold a version for other clients as well, though.

Sean: One of the things that I find interesting is how you have handled the breadth of QA questions you must have come up against. I mean, everybody uses Outlook, from the most computer illiterate to the most sophisticated users, so what they’re doing with their machines is radically different from each other. Notepad may be the only thing that gets more universal access on a Windows box.

What was your process around QA from the beginning, and how has it evolved? I’m sure you have to build something that is absolutely rock solid, and yet who knows what people are running alongside it?

Gabor: My first reaction is to say that this issue has been a lot of really hard work. When we were originally building the product and the team was just four people, we outsourced QA to a manual testing company in California. We had a good experience with that, but the problem is that the number of possible configurations and the number of things that can go wrong is very high.

I think the best decision I’ve ever made here is to bring QA in-house and to hire really good people and just try to get these issues taken care of by adding more bandwidth to them.

We hired an excellent QA lead and testers, and they’re not just testing the product–they’re testing the different integration scenarios with Outlook. They’re testing scenarios like how it works with Exchange, IMAP, POP, and various combinations of plugins.

That’s a lot of work, and it’s not particularly sexy, because everyone wants to work on a great new feature, rather than what happens when a particular ActiveDirectory setting is at one instead of zero. I’m very happy to say that we have QA engineers who are very passionate about getting these things right.

Scott Swigart: In a lot of open source projects, code’s checked in and people download stuff before it’s officially called a release, and they work with it. The model is often just to have good bug tracking, but not a lot of overt structure beyond that.

What do you think about the notion of handling bug fixes that way? What do you think would be the net result of a model like that in your case?

Gabor: That’s a very good question. I don’t think it would necessarily work for us–mainly because people don’t want to be exposed to bugs in their Outlook, and they don’t want to be the guinea pigs, because Outlook is their data store. It’s where all their personal data is, and they’re necessarily very sensitive about protecting that.

I think that our case requires in-house testing. That said, however, we originally did a staged private beta release of our software, to a limited population who had signed up on our web site.

That gave us a really strong test audience of people with a bunch of different configurations, and only once we were pretty comfortable with that did we go to the next order of magnitude.

There’s an effect at work that we call the “10X effect.” Every time you increase the number of users by a magnitude of 10X, new stuff starts popping up. For every 10X increase in user base, we get all sorts of weird scenarios that we haven’t seen before, and we go through these cases very methodically and just kind of adjust them one by one.

Scott: Another thing that open source does a lot is to use very distributed teams. You can have people working on one project from all over the globe and lots of different companies.

What kinds of advantages does it give you because you can pull everybody together in a conference room and you’ve got people working in the same building, versus what may be some of the challenges of using a highly decentralized distributed model?

Gabor: Everyone is in the office during work hours–no one really works from home on our current team. We have meetings about support, about QA, about where the product is headed, and about the current status of each of the projects that we’re currently working on.

We find that face time is very important, because if you want to describe a problem, you can just go up to the whiteboard and sketch it out. You don’t need to start up some virtual conferencing solution and try to drive your discussion through a window on the screen.

Really good ideas pop out from hallway conversations; we actually just had a conversation about how sometimes the Outlook UI would freeze when we were syncing data in the background:

We were able to fix that issue, but if we had not had that face-to-face conversation, we wouldn’t have been able to get a solid theory about why that bug was happening.

Scott: Another thing that springs to mind is that, aside from the architecture and what is going on behind the scenes, do you feel like a closed-source, centralized model facilitates getting where you want to be with the user experience and those kinds of things?

Gabor: To answer that, I should first take you on an excursion through history. The interface of Xobni has been highly praised, and some people call it “Star Trek”-like, because it looks like the computers on “Star Trek, The Next Generation.” The way that interface came about was basically that, in the very early days of Xobni, I just tried out different things in Photoshop. I came up with three basic designs: one looked just like Outlook, one looked like Outlook with bigger icons, and one I made as a joke about how far out we could take it. That is what we use now. And that’s how the basic look and feel of the interface came about.

It was basically arrived at with experimentation and talking to the other team members at the time about what they felt about the different designs. Lately, we’ve moved into a slightly heavier development model. We have a user experience expert and weekly usability meetings, we view mock ups, and we do hallway usability testing.

We also go to coffee shops. There’s a Starbucks we frequent a block from the office. We just ask people, “Do you understand this mockup? Do you want to try it out on our laptop and tell us whether it works for you?” One of us actually stands in line and says, “We’re going to pay for your latte if you have five minutes to look at some screen shots.”

Sean: Wow. Ethnographic research with a latte–it’s the grass roots ethos, for sure. [laughs]

Gabor: We’re in the financial district of San Francisco, so it’s a really good spot, as these are our target users. They all use Outlook, because they work at Morgan Stanley or Wells Fargo. They’re exactly the kind of people we want to talk to, and they’re exactly the kind of people that need to understand what our product does.

And so, back to the open source discussion. I don’t think that in a distributed team, these kinds of impromptu usability sessions would really be able to take place.

Sean: There’s the issue of reliability, and nobody wants to see the “Outlook is checking your inbox” message for the 5,000th time, just because you happen to shut it down. But at the same time, there’s a security issue, especially as you get into those large companies in the financial district and so on.

So what is your process around security? In the open source model, it’s well-known, and it boils down to “many eyeballs creates a more secure code.” I don’t mean this at all in a pejorative way, but closed source companies, by nature, need to protect their IP, and so they’re a smaller team that keeps it in-house.

What has Xobni done around the security aspects of the development model? Are you following a particular framework, whether externally developed or internally?

Gabor: Our basic premise is that we don’t ship any personal data back to us, including any part of your email, any part of your calendar, or any part of the data on your desktop. That’s been the case since the very beginning of Xobni, and it’s absolutely vital, because people are very sensitive about this issue.

Beginning in the very early versions, we shipped with something called “The Customer Experience Program,” which was an opt-in program, and it still exists, if you go into the options dialog. If you enable that, we ship back an anonymized data set about your usage patterns. It basically tells us that a user identified only by a randomly assigned number uses feature X and Y, but not Z. So, that is the one time where you can opt in and volunteer to give us some anonymized user data.

Sean: That’s an interesting bridge to another question that I have had in mind–what types of issues arise when you support a corporate enterprise sale? I imagine that there might be some concern on the part of IT organizations about supporting a broadly distributed plug-in like this.

Gabor: Actually, the first-line answer is that we’re a consumer company, and we want to speak directly to users and get them to download our products in order to address pain points that they are experiencing. We want them to be happy with our products and to be able to find the stuff in their inbox and to deal with it efficiently.

That’s much, much easier than trying to convince an IT department to get Xobni for their entire corporation.

That being said, a lot of the more progressive IT people have contacted us proactively and said, “We’re a company with 200 or a company with 5,000 people. We want Xobni on every single one of our desktops.” Of course, we encourage that, and may work out licensing that meets their needs.

I think in the future, we might have a support model specifically around this type of deployment, but that really hasn’t been worked out yet, mostly because the company is quite young and also because we are very focused on making a brilliant product that consumers really enjoy.

Scott: In the open source world, there’s the notion that the source code is free. People can download it, and companies have to figure out a way to monetize around that. They might sell support or have a more enterprise version that they charge for, with features that aren’t in the free one.

As you look at the landscape, what’s your impression around business models? Do you have a sense that the traditional model of having people download an eval version and then pay for the full version is still viable?

We run into the impression on the part of some people that everything’s moving to an open model and that open source software is built on a superior development methodology. On the other hand, there are new and innovative companies like yours that didn’t choose that route.

Do you feel that the traditional business models are alive and well, or do you see things moving to that “give it away and figure out how to charge for something” approach?

Gabor: People ask us this question a lot, and it basically comes down to, “How are you going to make money? Can you really charge for an add-in?” Our answer is that right now, we’re not focused on that. Right now, we are very focused on building a great product. We believe that if you build it, they will come.

And if you have millions of users that enjoy your product every day, somehow you will always be able to make money. Like Google. Google didn’t get started by saying, “Oh, we have search. Let’s find out how to monetize it.” They were, in the first few years, just focused on getting search better every single day, and later they found ways to monetize it.

There’s a story that the initial Google pitch stack was 70 pages long and explained how they would make money by selling search technology to enterprises, which was completely wrong at the time, but they later found a different way to monetize it, which has worked pretty well for them.

Our mindset, right now, is the same. I think we’re just building a great product and we’re going to find out how to make money eventually. People are asking us to buy pro versions. People are asking us whether they can pay if they want to install it into an enterprise.

We’re definitely looking at these models, but it’s really too early to take those conversations on in earnest, though–that still needs to wait six months or a year.

Sean: That’s fair. Let me ask you a different kind of question. There are obviously many, many good open source applications. At the same time, though, the little holy duopoly of Exchange and Outlook never seems to really find a strong open source competitor. I don’t think that people love it, but it’s the best that the world has at the moment.

Every time you try some other combination, whether it’s Zimbra or Evolution or Thunderbird or Sunbird or you try some other backend, like IMAP, POP, gmail, or whatever, they never seem to have that fidelity.

Why do you think the Exchange and Outlook combination never really sees the same strength of potential competitors as RedHat versus Windows, or MySQL versus SQL Server, or any of these other parts of, let’s say, the LAMP stack? Why is it that in that one spot, there doesn’t seem to be a parallel, even though there is almost everywhere else?

Gabor: I think you’re totally right, and I don’t know why that is. There are definitely great applications out there, such as the experience that Gmail provides. Also, Zimbra is great, and there’s just a fantastic variety of new applications in this field.

I think that Microsoft has the strength they do in the enterprise world right now because they know exactly what features they want, and they know exactly what these customers need and how to get into these enterprises. And once you have Exchange Server installed, it’s pretty hard to switch, because Exchange and Outlook have tremendous lock-in. I think that’s the key to their success and to their strength.

Sean: As a follow up to that then, since you came from Google, it’s an easy one to throw out there. What do you think about someone like Google’s relationship to open source? Or to give another example, what about Apple? Apple is seen as this amazingly open company, but they’re certainly not very open when it comes to their roadmap, or talking about other stuff like that.

With Google, you’ve got a company that builds a lot of stuff on essentially free open source, but yet they build a lot of commercialization on top of that, so there’s an interesting story there I think. Do you see any tension there?

Gabor: First, I have a really high opinion of Google, and I think that they’re excellent at software development. Also, I think their fundamental tenet of not being evil and doing good for the world is genuine, and they’re certainly sticking to it. They’ve open-sourced a lot of software already – such as Google Gears or protocol buffers, and I’m sure they want to be open and share the wealth with the world, but have to balance that with being a strong company that can generate profit.

Scott: During development, does open source fit in anything you do? I know as a developer that open source is really, really popular as developer tools for unit testing, automated builds, and for all kinds of different things. From the perspective of peeling back the covers of your organization, where do you find it useful or do you not really use it for much?

Gabor: We do use open source for build tools, release tools, unit tests, and so on. The infrastructure provided by the open source community is very useful to us, and I could imagine us contributing once we have a little higher level of resources, but so far our interaction has been very limited.

Scott: Would you mind giving a plug for some of the build tools and development tools that you really like?

Gabor: There’s always the wish list that you have. I think that NAnt, NUnit, or CruiseControl are really good, but I wish that they could each do this one more thing. There are other tools we use, all of which are an integral part of our development cycle.

Scott: One of the things that we have observed before is that open source really shines in projects that are ‘by developers, for developers,’ rather than software that targets a non-developer user base. In the case of developer tools, for example, the people building the project have an intimate understanding of what the product needs.

Open source projects tend to be very good at building a lot of infrastructure stuff–operating systems like Linux, web servers like Apache, and so forth. That seems to be where open source really innovates toward excellence. On the other hand, the really end-user-focused innovation like the iPhone, Google Maps, and the Xobni UI examples we have been talking about don’t tend to come from open-source projects.

I feel like open source is very concerned with architecture and performance and modularity, but when it comes to building a peak experience product, a lot of that’s still coming from the Apples and the Googles and the Microsofts and those kind of companies.

Do you feel like I’m on to something there? And if so, do you feel like there are fundamental reasons for that, based on how the different kinds of software are built? Or am I kind of just drawing a circle around a few things I have observed and calling it a trend?

Gabor: I have a very controversial opinion around this issue, which is that if you want to make a product with a peak experience like the iPhone for example, it’s not as much about what you build as what you take away, and about making those cuts at the right point.

For these types of decisions, you need very strong product people who understand exactly what the consumer wants and who understand exactly what the minimal feature set is that covers the maximum number of usage scenarios.

And this is something that I, as a developer, am not necessarily good at. There’s always the impulse to think of the next feature, and I always want to add, add, add, add. That’s why all of the open source tools have zillions of command-line flags and options and so on.

Whereas, if you want to make a consumer product like the iPhone, you have to make the cut somewhere, and make those things that are inside of the cut just outstandingly good instead of making a lot of things that are sort of a 75% solution.

Scott: That’s interesting from the standpoint of a lot of work I have done as a consultant for software companies. They say, “We’re going to add this feature to the product, because one important customer wants it.” And especially with smaller companies, a lot of times they can be pushed around by their bigger corporate customers to add things into the product.

Even something like MySQL does a certain amount of non-recurring engineering, and they add a certain amount of features, because a person wants it really bad for some specific scenario. I’ve never heard anyone explain it exactly the way you did, but that makes perfect sense. The iPhone’s good not because of what’s in it–it’s good because that’s all that’s in it. They really did just focus on making those things really good.

So how does a company like yours balance those two impulses? You must run into people who say, “We’d buy it if only you would add this feature.” How do you balance that with the philosophy of less is more?

Gabor: I think this question comes back to what I said earlier, which is that we’re really focused on consumers. We don’t, as a company, listen to the types of people who say, “If you just added this feature,” because right now we’re not even selling a product. So, the bottom line is that we can’t really consider a scenario where if we just added the foreign feature, we’d have $100, 000 more revenue.

And a related thought is that it’s been very painful for me personally, because I’m one of those “add, add, add” people, and other people on the engineering team are as well. Xobni used to have in the private data stage two tabs, which were “email” and “organize.” And one was about your email and the other one was about your calendars, your appointments, your outstanding tasks and so on.

We found through the customer experience improvement program that I talked about earlier, where we send back anonymized data about features if you opt in, that no one was using it.

We thought, “We built this great thing and no one is using it. Why? We should make it more prominent and maybe we should make the button blink or something.”

[laughter]

But then really, honestly, this is really and indication that people don’t want that functionality. And we decided to cut it out. It was probably one of the best decisions that we ever made, because we focused the product, and we made the things that people really do use even better. It’s worked great for us–it was just very painful at the time.

Scott: That reminds me of Firefox. We had Netscape, and then Internet Explorer became dominant, and it kind of became the browser with one more thing and one more button. And when Firefox first came out, it really made people sit back and ask what they really needed from a browser. They asked whether they really needed a lot of extraneous stuff, and a lot of them decided that Firefox really had everything they needed.

When it kind of caught fire, I think it was because it had that minimalist approach. I wonder if it depends on the size of your target, in the sense that if you target a niche, then you may need to be able to get pushed around by the niche a lot. And you may need to build a feature for a customer, because the niche is only so big.

On the other hand, if you are targeting millions of users like Xobni did, maybe you’re automatically freed from a lot of that. In essence, it seems like you asked where the pain point is for hundreds of millions of users, and that freed you from trying to be all things to all people. You just have to be the right thing for a bunch of people.

Gabor: Exactly. But finding that spot is really hard.

Scott: I can imagine.

Sean: Well, thanks for a great conversation. I’ve really enjoyed this.

Gabor: Absolutely. Thanks for setting it up.

Bookmark this:

Interviewee: Ben Chelf

In this interview we talk with Ben Chelf from Coverity. In specific, we talk about:

• Scanning the worldwide open source code base for vulnerabilities
• The necessity of automating defect identification
• Building scalability into code analysis
• Overcoming the limitations of automatic scanning
• The hubris of believing too much in your code
• The difficulty of appearing objective
• The changing face of code security and analysis

Sean Campbell: Hi, Ben. Could you start us off with some background of yourself and of Coverity?

Ben Chelf: Sure. I’m the CTO and one of the founders of Coverity. We started the company in late 2002, with technology that I worked on with a few colleagues while we were graduate students at Stanford.

The idea behind the technology is to make static source code analysis practical for everyday commercial software developers. It was a bit of a departure from previous academic attempts at static analysis, in the sense that our approach seeks to scale to millions of lines of code, so it can encompass systems that people are actually developing today.

To fine-tune the technology, we applied it to the Linux kernel, and the nature of the open source community made it fairly simple to work with developers to correct the defects we found. As more and more people found out about the defects we had been able to find automatically–as opposed to the old-fashioned “hard way”– a lot of buzz started to develop around the technology.

Some companies started approaching us, saying, “Obviously this stuff works because it’s being applied to a real system like Linux and getting some traction there, so can we try it on our code bases?” Eventually, a customer offered to pay us, and we decided that it was the right time to get the company started.

We saw a rapid adoption of our product, and we learned a lot in those first few months about bringing the raw technology into product form. We also started to address some of the other issues around making static source code analysis practical for commercial development, in terms of integrating it into existing build systems and making it so that defects are automatically distributed to the right developers when they’re discovered.

To encourage adoption, it was also important for us to work our technology into the existing workflow so that the defects we discovered automatically could be treated in a similar fashion to those discovered in the QA process and so forth.

Over the course of that learning process, we acquired a lot of customers–we have more than 400 now. Because the technology is so valuable, the company has been cash-flow-positive since our inception. And that’s a testament to the real value, I think, that we’re providing to folks in discovering defects earlier in the software development process with this new, disruptive kind of technology.

Scott Swigart: Coverity showed up in relatively recent news with a huge splash in making the announcement that certain open source projects are now in Coverity’s rung 2. Can you speak a bit to the significance of that announcement and what those different rungs signify?

Ben: Obviously, we started our work with the open source community when we were at Stanford, but under the Coverity umbrella, a new wave of work with the open source community really started about two years ago. We have been providing our technology as a service to open source developers and specifically as many of the widely known packages as possible through the site scan.coverity.com.

What led to this effort was that, a couple months earlier, we had been awarded a contract with the U.S. Department of Homeland Security, in conjunction with Stanford and Symantec, essentially to harden open source, since it is being used more and more in our nation’s infrastructure.

Software like Linux and Apache is everywhere, and as a matter of public security, it’s important to maximize the extent to which these systems are secure and reliable, work as they’re meant to, and are resistant to attack.

When we launched the site, we weren’t sure how it would be received. Being a proprietary software company, we sell a product. So we were a little bit nervous about how it would be received by the open source community.

So we tried to be very respectful of their code, and when we launched the site, I personally sent a note to the developer list saying, “OK. Here’s the deal. Here’s this site that we’re about to launch. I’m not letting anybody have access except for you. This is for the developers of these projects. This isn’t for press people to log in and troll through the bugs, or hackers to log in and look for vulnerabilities. This really is for the developers of the open source package.”

Today, we’ve kept that mantra of, “This is really for the developers.” Yes, we post statistics on the site, at a high level. But it’s really meant to try to improve the open source software as a development tool to help these folks.

I think it’s because of that cautious approach that the site has been well received. Developers started flocking to the site, registrations came flooding in, and defects started to get fixed as a result of the site.

In fact, to date, almost 8,000 defects have been fixed as a result of our scanning efforts through this site, across the board from open to commercial packages. We’re now seeing tens of millions of lines of code on a regular basis.

About the rung question, about a year after the site launch, we introduced the idea of the Scan ladder and the different rungs on it to incent the packages to use the product more. To acknowledge and reward those who have been diligently using the product and fixing all the defects that we find, we offer upgrades to the latest and greatest from Coverity, from a product and feature standpoint, as well as giving them additional access to us in terms of giving us their feedback.

There were 11 packages that met the criteria we set up for rung 2, in terms of cleaning out defects on a regular basis. We kind of stepped up our relationship with those packages and our commitment to them to help them get the latest and greatest out of the technology. We anticipate, over time, that we will add more and more rungs, as we have more and more capabilities in our product.

Scott: The press coverage seemed to stretch between two ends of a spectrum. One side seemed to assert that these projects that achieved rung two are bug free. Obviously, Coverity never suggested that–you were just rewarding projects that have made good use of your technology. On the other end of the spectrum, some people thought it pointed out inherent weakness of open source, because there were vulnerabilities to be addressed.

My impression is the truth lies somewhere in the middle. Large open source products like the Linux kernel or Apache have very high quality, and they’re worked on by very good developers. But I think it does show that certain jobs are best done by a machine. It stands to reason that using automation to scan code that never gets tired or bored makes sense.

Where do you think the limits are of what eyeballs can do in fixing defects? And as an extension of that idea, where do you think the technology can continue to go? What are some of the opportunities for things that could potentially be scanned for that maybe are still hard problems to solve today?

Ben: We started this project with the idea that we were not up to the task of debugging our own software; debugging is something that’s plagued developers for forever, right?

In 1949 Maurice Wilkes–a pioneer in creating the EDSAC–said that the first time he had to debug his software, he realized that most of the time he spent in programming would be on debugging the mistakes he made.

The observation, of course, is that the existing techniques are not sufficient. Testing can’t cover everything, and humans are subject to frailty, no matter how good we are. First, we’re not perfect. Even if I only had 1,000 lines of code and I could look at it day in and day out for months and months and months, I would still make mistakes.

Layer on top of that the fact that systems now are millions or tens of millions of lines of code, and a single developer has no chance of keeping the complexity of that kind of system in his or her head all at once. It’s just impossible.

We’re at the limit. There is no way to uncover these defects with the traditional software development process. So you look to automation technology that can cover all the different possibilities in code, and this is where static analysis shines, especially compared to testing or dynamic techniques. It isn’t tied to the execution of the program.

When you execute the program, maybe you get five percent code coverage, meaning that five percent of the lines of code are touched at least once in the execution of a test suite. And that doesn’t speak at all about the different contexts that those lines could be executed in.

So when you’re talking about real coverage, it’s a very miniscule percentage. Hundredths or thousandths or tens of thousandths of a percent in terms of the billions and trillions of different possible execution paths through the code.

Static analysis affords us the luxury of being able to try to cover all of that without running it. And so you can find those corner cases that every human and test case overlooked. Now, obviously the promise of that sounds great, but there are some very practical limitations, in terms of analysis.

Analysis is not easy, partly because the systems are so complicated. This is where a lot of the breakthrough comes in from our technology–addressing that scalability while retaining enough analysis heft to find interesting things in the code.

It’s one thing to analyze ten million lines of code, but if you’re not doing any deep analysis, you’re not going to find any of those real defects, those real security vulnerabilities that really resonate with developers. On the other hand, if you are doing deep analysis, it might take you the age of the universe to analyze a database of reasonable size.

Scott: [laughs] Right.

Ben: So that’s no good either. So there’s this very elusive sweet spot that for decades, literally, people didn’t quite have figured out in static analysis, and that’s what kept it from really delivering on the promise that it had ever since the days of Lint in the late 70s.

There are always tradeoffs that have to be made to get that scalability, and those tradeoffs manifest themselves as potential weaknesses, in terms of both false negatives and false positives.

False negatives, of course, are instances where static analysis did not report a defect that’s actually in the system. Of course, we’ll never have static analysis that can find every single defect, but if it doesn’t find anything, then it’s not going to be very useful. Therefore, in a nutshell, you need to find enough things to make the testing process valuable, in conjunction with your other mechanisms for testing.

The other issue is false positives, which tend to be a lot more deadly. When a static analysis tool reports to you something that is not actually a defect, it wastes a bit of the developer’s time. If you have too many false positives, then the developers will start ignoring even the good results. It’s the boy who cried wolf phenomenon.

If the tool cries wolf nine times in a row, even if that tenth one is a nasty bug, you’re probably going to gloss over it. I think the testament to the fact that we solved that problem was that open source developers gravitated to the product. They used it and they kept using it.

No manager told them, “You better go use this now,” or anything like that It was just out there and they kept using it. We’ve measured the false positive rate to be very low, and that’s an important part of what makes it practical–having enough analysis heft to keep the rates of false positive low, yet still having a high enough rate of real bugs found to make it interesting and valuable to the end customer.

All of that is really background to answer your question about where we take the product next. You want to add more add more analysis heft and still scale, so you can reduce that false positive rate even lower and find more and more and more bugs.

If you were to run Coverity 1.0 circa 2003 and compare it to now, we find double, triple, quadruple, or more number of defects on any given code base at that same low false positive rate, because we’re learning. We’re getting examples from our customers–they’re giving us more and more bugs, and asking us to find ways to uncover them.

We’ve now analyzed close to two billion lines of code between the open source community and all of our commercial customers. That’s an awful lot of great data for us in terms of learning about software systems, how they fail, and specifically in the code, what goes wrong that makes developers pull their hair out.

The obvious question is how to do that. How do you increase your bug-finding rate and make sure you’re doing it in a reasonable fashion? Recently, we’ve introduced a new kind of analysis technology based on Boolean satisfiability.

You take a Boolean formula, which only deals in values of true and false–variables that have true and false and the logical operators and “or” and “not”–and you try to figure out whether its’ possible for that formula to be satisfied. In other words, you try to determine whether there’s some assignment of variables that make the whole thing true.

This has actually been used in the hardware industry for a long time for the people who are making the tools for chip design. Of course, chips are very, very expensive to patch, if a defect hits in the field. We all remember that Intel Pentium 4 div bug–it cost Intel half a billion dollars.

Obviously, that’s a pretty big mistake, and it’s not surprising that the hardware guys generally apply more rigor on the analysis front than the software guys do. But the fact is, static analysis for hardware verification has been around for a long time. No one’s really translated this for software before, because there hasn’t been as much as a push for software verification.

Now we have introduced that kind of technology to complement our traditional dataflow analysis, or abstract interpretation, or lattice simulation–there are a number of different names for the notion of pushing down all the different possible paths through the code.

But we want to marry that with a bit-level representation of the software system. That helps use reduce false positives by reducing false paths, meaning paths that can never be executed, as well as paths that shouldn’t be analyzed.

It also helps us find new kinds of bugs like integer overflows and enhance the buffer overflow capabilities of the traditional static analysis technologies. So there are lots of different technologies out there that can do analysis of systems at various different levels.

But again, we always have an eye toward the scalability of the system and a low false positive rate, because that’s how you get the thing to be used. And of course that’s the name of the game. It doesn’t matter how good your analysis is, if developers don’t want to use it.

Scott: One of the things I did for a little homework was to look at the Linux kernel mailing list to see what the initial response was when Coverity started to show up on the list.

I remember seeing a thread where somebody was submitting a bunch of patches, and other people were arguing against checking them in. Their view was that they needed to actually look at why the tool reported a problem, and that it isn’t enough just to modify the code so that the tool doesn’t complain any more. They wanted to verify that there was really a vulnerability.

What are the safeguards, once the tool has output a lot things, to make sure they actually get fixed the right way? And do you think that there is the problem, or the potential of a problem, where people might just try to make the code pass through clean, but they don’t necessarily really spend the time to really do a root cause analysis? Is there a vulnerability here?

Ben: Yes, that’s always a danger with any kind of technology, in terms of a developer who isn’t quite in the right mindset, who maybe is unfortunately not careful enough about the changes they’re making. They might just try to either make the code compile without throwing errors or warnings, or make it run through the static analysis tool without reporting anything.

What we do to discourage that kind of behavior is to interface robustly to the defect. What we communicate is pretty much everything we learned about the defect in the analysis of that path that discovered the problem.

So, for example, if we observe one event on line 23, and a number of different conditions had to be true or false to get another key event to occur, say, on line 53, we try to step the developer from one condition to the next, outlining the path as exactly as possible.

We try to present all of the information that we have to encourage a deeper look into the issue, although of course you can lead the horse to water, but you can’t make him drink.

For instance, we might say, “OK, this is where you have a memory leak.” And they think, “Oh, memory leak. I should put a free in there.” Or no pointer dereference. “Oh, I guess I should just check that pointer to see if it’s null or not.”

Absolutely, you can end up introducing things that aren’t the right fix. And, of course, that’s a fact of life. It’s just as if a QA person told the developer, “Hey, I noticed that the system crashed.” Then the developer just made a bunch of arbitrary changes. Then it doesn’t crash anymore, but they never understood exactly why that crash happened in the first place. Chances are, they just masked the problem with maybe another problem that’s going to rear its head later on.

So the problem isn’t specific to static analysis, but certainly these tools can be susceptible to that.

Scott: We were talking to some people on Firefox, and they said the dynamic nature of their product, the way they use pointers, and so on causes Coverity to generate excessive false positives. Do you see that type of issue in other environments, and if so, how do you respond to that?

Ben: One of the things that we unfortunately don’t necessarily get to have with the open source community, as opposed to our commercial customers, is the implementation phase of the static analysis.

For the most part, users are going to get good scalability and a low rate of false positives right out of the box, but there are a lot of things you can do to tune it, as well. If you do happen to have a higher than average false positive rate, chances are good that we can address it by encapsulating just a couple of idioms in some configuration for the analysis engine.

For our commercial customers, part of licensing the technology includes our professional services, to come in and make sure everything looks exactly right for that environment. You need to tune things to the application that you’re analyzing, since every code base is a little different. Tuning the analysis engine can not only keep the false positive rate low, but it can also find more and more kinds of defects, if you add additional bits of knowledge.

Sean: When you looked at open source projects, I understand that PHP and some others came out as having particularly high rates of vulnerabilities. Was there any particular class of vulnerability that seemed to show up more often than others? And I guess this is really a separate question, but in cases like that, how do you provide the next level down of analysis for customers?

Ben: For the open source packages, we decided not to do that deeper kind of digging. Of course, the data’s there, and we could certainly look at the different kinds of vulnerabilities and defects that we discovered, but there was generally nothing that jumped out as so obvious that we thought it was either newsworthy or worth exposing in that way.

On the other hand, with X.org, there was one security vulnerability that caused them a great amount of potential grief. They were very glad that we found it, because they said it was literally one of their worst case scenarios–a root exploit that anybody could take advantage of.

It was actually just a simple matter of missing parentheses in a function call, where they meant to call a function but didn’t. That kind of issue is hard to categorize. It was just a simple coding error, but it led to a root exploit.

That’s another thing with static analysis: it’s sometimes hard to predict the impact of a coding flaw. For instance, we might discover a place where you’re going to de-reference a NULL pointer.” Well, that could translate into an innocuous crash that hardly ever happens, or a huge denial of service that every single attacker can take advantage of.

Sean: Once you’ve got a mistake, there’s no way to really know what the ramifications of that mistake are going to be. A curly brace in the wrong place, or, like you said, lack of parens could be nothing, or it could be a worst-case scenario.

Ben: Exactly. So we try to stay away from getting too sensational about claiming that a certain type of defects suggest a certain level of vulnerability. We prefer to let the data speak for itself, and we let the developers take care of the issues to get them fixed.

Sean: At this point, you guys have looked at a lot of code across commercial and open source projects, projects of different sizes, and projects of different ages.

Do you find any correlation between the number of defects that you find with the size or age of the project? Are there other things where there seems to be a correlation, or is it really not correlated to much of anything?

Ben: I would say it isn’t correlated to any of those obvious things like age or size. Certainly the bigger the project, the more susceptible it’ll be to the complexities of code interacting with other code, which leads to the kinds of things we find.

So there might be a slight correlation in terms of defects per thousand lines of code being a little bit higher on the larger code bases, but what I think dominates more than that is the mindset of the development organization.

Those who focus on quality, in terms of good processes and tools in that regard, have better code, and we find fewer bugs in that kind of code.

Sean: I’ll just go ahead and lay some personal bias out there for the reader to judge me on, but there seem to be a lot of organizations that have a lot of hubris around the security of their products because they haven’t necessarily had a lot of high profile exploits.

Not to throw stones at any one company, but I read an article just today about enterprise organizations that have a lot of concerns about the iPhone, because of vulnerabilities like buffer overruns from people loading unsupported apps on them.

You’ve been looking at this for a long time. Do you feel like companies are sometimes a little overconfident, so they haven’t necessarily really done the work to make it secure, and if they really started looking under the rug they’d find a lot of unpleasantries?

Ben: Obviously, I can’t comment on any particular companies, but I think software makers in general–whether commercial or open source–need to be paranoid.

They need to recognize that every system is attackable, and that there is no such thing as finding all the bugs. There is no such thing as finding all the security vulnerabilities, and that means you’re susceptible to malicious users.

I think software makers who have not been the victims of a lot of attacks should consider themselves lucky. It’s a game of probabilities, in many respects, and you can control some of the factors but not all of them.

You can control the degree to which you test your product, how good your secure coding practices are, whether you leverage the latest and greatest static analysis, find and fix all those defects, and do the penetration testing and so forth. All of that reduces the probability that malicious users will discover remaining security vulnerabilities to be attacked.

On the other hand, software builders don’t have direct control over factors like how many attackers are out there trying to break the system. I think the companies that have the highest profile security problems are the ones the attackers are really going after, because it’s their products that are on most of the hardware out there.

There are a lot of different mindsets for attackers, but one motivation is certainly fame or the pervasiveness of the exploit. If I have a choice of hacking something that a million people have or five people have, I’m going to pick the one that applies to a million different systems.

Sean: What do you think are some of the most common misconceptions that people have about announcements that you guys push out, whether it’s the homeland security one or whatever?

When Microsoft or Mozilla goes and puts out an announcement about bug counts, the other one comes and spanks it down. In the end, you’re always left wondering. I’ve been wanting to put a certain great Mark Twain phrase into an interview transcript, and this looks like the right place–”Lies, damn lies, and statistics.”

What are the common misconceptions you see about the meaning of statistics you publish?

Ben: The most common misconception, especially with the open source work we’re doing, is that we are somehow trying to make a judgment between the quality of open source versus the quality of proprietary software.

I think if you look at the debates and the Slashdot dialog and the comments to stories and so on, you’ll see people responding to report by saying, “Oh, is Coverity saying that open source is bad?”, “No, Coverity is saying open source is good!”, “No, Coverity is saying open source is bad!”

[laughter]

Sean: Your PR department must hate it when you put out an analysis, since they know they’re going to get hit by two sides. Both sides are going to pull the numbers and stretch it like taffy.

Ben: Exactly. I think that’s the number one thing that people misconstrue. I think the second misconception, unless people are really trying to understand what we’re doing with these projects and so on, is they think that we’re claiming that static analysis is the be-all and end-all to software testing and the ultimate yardstick for software quality.

We never make that claim, and we’ve never fed it when other people make it. The reason we put these reports out and publicize them is so that people are aware of technologies that can help make software better.

Static analysis hasn’t been around for a long time, and there isn’t necessarily awareness of what it can do, but in the last two years, 8,000 defects have been fixed in open source packages because of this new technology. That’s says something.

The fact that 400 customers have signed up, many of them across their enterprise, to have our defect detection as part of their development process also says something.

There are still many billions of lines of code out there that aren’t going through this new way of checking. To get this stuff implemented used to be hard, but not anymore. There’s no excuse.

When we make these announcements and talk to people about the technologies, we want them to realize that the development world has improved, in terms of the tools and technologies that help them make the quality of software better. We do that in part using living proof that open source developers are using it and benefiting from it.

Scott: For whatever reason, it’s hard to simply present data and have people believe that you’re not trying to subtly pass a judgment.

It must have been tremendously valuable to you that tens of millions of lines of open source code were available to throw at your product for analysis.

Ben: Absolutely. I wrote an article about two years ago, and the title was something on the order of, “Open Source Software and Static Source Code Analysis: A Perfect Match.”

The article was really about how the open source movement enabled this innovation in static analysis. Before that, not enough companies would have let us come in and play around with their code for months on end. Now that we have all this code out there, it’s a great playground.

Sean: It seems to me that what hackers do has sort of changed over the past five, six, seven years. It used to be, if you had a virus, worm, or whatever, you knew it, because your corporate network came to its knees.

Today, the hackers are far more subtle, because they’ve figured out that it’s a lot more beneficial for them to build a bot-net that nobody knows they’re a part of. Do changing hackers’ techniques affect the kind of scanning that you do?

Ben: I don’t think there’s too much of a correlation there. Not all hackers are even really ‘hacking’ so much, because there are so many social ways to take control of systems, just by getting people to install stuff.

I do think that the vulnerabilities that allow them to automatically have a worm or a bot spread across a network look the same as ever, though. Even if they hide their tracks a lot better than they used to, they’re attacking the same kinds of problems in the code.

Sean: Do you think that track-hiding helps perpetuate a false sense of security, as opposed to the old malware back in the day, like “Code Red,” “Nimda,” and “Slammer,” where you definitely knew you had them?

Ben: That’s a good question. Admittedly I don’t have my pulse on the consumer side of things, but my impression is that people are not as concerned about security, but they’re frustrated by a different aspect of the same issue. They buy a computer, and six months later it’s loaded up with all kinds of different services and applications that they have no idea about.

And some of those are probably malware, and some of them are legitimate, but I don’t think people see it as a security threat, they just see it as frustration with computing.

Scott: It seems that, even on the server side, nobody in the data center wants to install any more patches than they have to. By using static analysis in combination with other tools, like dynamic analysis and fuzz testing, on one hand the number of defects per line of code can go down, but on the other hand, the number of lines of code goes up. Do you see it as a zero sum game, or can the number of defects per line of code be driven down faster than the increase in the total number of lines of code?

Ben: You can definitely make progress, for the simple reason that people write code, and that’s slow, whereas analyzing code is fast. We can analyze code a lot faster than people can generate it, and when we innovate in terms of finding ways to analyze code and so on, we’re pushing that out to a billion and a half or two billion lines of code per year.

It doesn’t take very much innovation out of the Coverity Research and Development arm to drastically impact the quality of billions of lines of code, so I think we’re going to be able to stay ahead of the game in that regard.

It’s the exploding complexity in software that has people listening again to the messages of static analysis and disruptive technology, because they don’t have any other answers. They’re trying to make high-quality, secure code, but the volume is increasing geometrically. There are 10 million lines of code in a premium automobile today, where 10 years ago, there were almost none.

There’s code in everything; by the time you get to work in the morning, you’ve interacted with probably over 100 million lines of code, if you look at all the devices that you’ve interacted with. And since that code is coming from all over–outsourcing, interactions with third party libraries, new infrastructure frameworks, you name it–the complexity of code is just too great not to leverage these technologies.

Scott: Do you see any sort of a trend where when a company gets components from external sources, they use tools like yours as sort of a quality gate? For example, they might go out to some component vendor and say, “Look, you don’t have to open source your component, but I want to run some tools on your code base,” so they could compare the outcomes among competitors.

Ben: We’ve got a great case study on exactly that topic. One of our customers recently was trying to come to an agreement with their outsource vendor as to when some code should be accepted, in terms of whether it had been debugged sufficiently.

We didn’t suggest this to them, but they looked at Coverity as third-party arbitration, if you will, where they could say, “This is a gate, and we both can agree that when Coverity says there are no defects in it, we accept that development is complete.”

That approach cleaned up their relationship a lot, because they finally had some kind of commonly agreed upon measurement for acceptance. They also said that using the product in that capacity and also just fixing defects early in the development process let them shave a significant percentage off of the time it took them to release a product. Of course, that’s the ultimate ROI here–you want to get products out the door faster.

I think that’s something that we will see more and more of. People will finally have some way of making a reasonable gate for the code that they accept into their products.

Scott: Well, I want to be sensitive to the time, so do you have any final thoughts to add?

Ben: We have talked a lot about problems in software development and finding defects in code with static analysis, but we view this as just the beginning.

To do static analysis the right way, you have to understand how the software is built, how every file is compiled, and how it’s all assembled together. In the technology that we’ve generated over the last five years at Coverity, we have all of this information.

If you look at the marketing literature out there on Coverity, we talk about the software DNA map, the fundamental building blocks of everything in your software system–not just the source code. Versions, how every file was compiled, your build system, and all of that information is sitting right there.

The future for Coverity is to branch out from static analysis and to try to identify the places that are painful in the software development process, from build systems through the compilation process, and finding defects, both statically and dynamically.

All of those things where having a beautiful representation of the software system and a complete representation of exactly what’s going on could benefit the developers, the product managers, and the software development managers. We want to help them answer questions like, “What’s going in my code base? How is it changing? Am I ready for release? Is this code in good shape?”

Right now, no one knows how to answer those questions, so for us, the next few years will be spent exploring that. We’re really trying to find the tools and technologies that will enable high software integrity, and that encompasses much more than just finding a few bugs or a few security vulnerabilities–it means making sure that you’re confident in what you’re doing from a software development standpoint.

That’s how I’ll conclude–a little bit of a teaser for what’s coming next out of the pipe.

Scott: Excellent. Thanks for taking the time to talk with us today.

Ben: You’re welcome, and thank you.

Bookmark this:

Interviewee: Jamie Thingelstad

In this interview we talk with Jamie Thingelstad – CTO of the Wall Street Journal’s Digital Network. In specific, we talk about:

• Open and closed-source technology stacks at WSJ
• The technology evaluation process
• Intersections between open source and commercial entities
• Security considerations with open and closed platforms
• The value and liability of owning code

Sean Campbell: Jamie, thanks for taking the time to chat. Could you give us a little background on your role at Dow Jones?

Jamie Thingelstad: Sure. I’m chief technology officer for the Wall Street Journal Digital Network, which is comprised of WSJ.com, MarketWatch.com, Barrons.com, and a number of our smaller web properties. I’ve been in this role for about a year and a half. Prior to that, I was the CTO for MarketWatch.com inside Dow Jones, after we were acquired by Dow Jones.

Going back further, I was CTO and one of the early founders of Bigcharts.com, which is a premium charting website. BigCharts was acquired by MarketWatch in 1999, and I became CTO of MarketWatch after the acquisition. We acquired a few companies, and then Dow Jones acquired MarketWatch.

Scott Swigart: You mentioned a number of digital properties that Dow Jones has. Tell us a bit about the infrastructure behind those and the kind of volume that those properties have to handle.

Jamie: Since they’re all news sites, there’s a certain unpredictability to the traffic volume. For instance, the market can drop at any second, and that can drive huge amounts of volume to our web sites.

The two largest sites that we run, WSJ.com and MarketWatch.com, are fairly different. WSJ.com is built in a Java environment using WebSphere, making a slow migration to Tomcat. It’s built in a fairly rigid framework, but it is very robust and very scalable. It’s stable, but also not quite as flexible as we’d like it to be.

The MarketWatch platform is a bit different. MarketWatch is built in a Windows environment and uses .NET. It was originally built in ASP and has been migrated to .NET in bits and pieces. It follows a “hundreds and hundreds of servers” approach, where no one server is doing all that much, so it is a bit more agile, and we can typically deploy things a little bit faster. It also has a slightly different performance characteristic. It can have the equivalent of a brown out more often than say, WSJ.com.

Scott: That’s pretty fascinating that you’ve got several high volume, critical sites and yet their underpinnings are so different from each other. You mentioned moving the Wall Street Journal site to an Apache open source back end, but the MarketWatch site is on a proprietary Microsoft stack.

What can you tell us about the decision-making process as different organizations evaluate technology choices? It looks like these two groups took pretty different approaches to solve similar goals

Jamie: The lesson to be learned there is that you can build large-scale sites to be stable, reliable, and highly available in either of those platforms. There is nothing about one versus the other that inherently makes it unable to build and provide those kinds of solutions. Good developers working in either one can get the job done, and do it elegantly.

The historical context is that the Journal web site as it exists now is the descendent of an engagement with IBM Consulting many years ago, and they obviously wanted to build in a WebSphere environment.

All this predates my being part of Dow Jones, but my understanding is that the decision was more about who to partner with than how to get to a specific architecture. In this case, IBM came to the table with WebSphere. In the MarketWatch example, that’s the platform that we built over the course of many years starting with BigCharts, and the platform actually started in a very different place.

In the very early days, before it was MarketWatch, when it was the platform that ran BigCharts, it was Sybase. It was Perl and CGI in Apache. And it was an ANSI C++ program that we had written to serve charts, that could actually be compiled on both the Win32 library as well as in Unix, at the time Solaris. And so, we had this sort of cross-platform environment. We made a decision around 1996 or ‘97 that we were spending too much effort building plumbing code and decided that we needed to move to something else.

At the time, we evaluated CORBA and COM, and our conclusion was that COM was better suited to the type of solutions that we were trying to build. We rewrote the entire system into a COM model and then switched our front-end production environment from Solaris to Windows NT. The net benefit of all that was that we stopped writing a lot of code that was about glue and plumbing, and focused all of our code on business requirements and functions and features that our customers wanted.

Scott: Can you walk us through how you think about the technology evaluation process? And, as a follow-up, broadly speaking, what are your observations about the pluses and minuses of building on top of a community-supported open source stack, versus going with more of a partner, like IBM or Microsoft?

Jamie: I think you have to start with your team. If you’re able to build your own team, it’s less of an issue, but if you’re walking into an environment where the team is well-versed in one tool set or one part of the stack, you shouldn’t fool yourself into thinking that you can then choose some alien technology and that the team will just flow with it.

I don’t know many Java shops out there where you could just walk in and say, “Well, you know what? We’re going to do the next project in Windows” and have that be just fine. Independent of culture, independent of bigotry, I think it’s a matter of skills and capabilities, and some of those skills don’t transfer easily.

But if you take the team out of it, then I think it’s a pretty interesting angle. My view is that there really is little value that a proprietary platform like the Windows platform gives you. If I were going to start, green field, building my own shop, I would be heavily biased towards free platforms: some Linux distribution, likely something like Apache.

I would also be very biased towards an interpreted environment. I think there have been great advances in the compiled environments for the web, and .NET has made huge leaps in performance and capability. Similarly, so has Java. But it’s interesting that the sites that I see doing the most “out-there” stuff, that are the examples of innovation, are typically not running a compiled environment on the front end. WordPress, Twitter and Facebook come to mind.

I hate to suggest that there’s an either-or situation where either you are innovative and you’re on an interpreted environment, or you’re not and you run a compiled environment. I don’t think it’s quite that simple. But I do think that there is some real merit to the lightness that you get with something like PHP, with a robust framework, or Ruby on Rails.

In fact, we’ve tried to have our cake and eat it, too. One of the most recent products that we released on the MarketWatch site is a product called MarketWatch Community. And that project was actually built using the MonoRail framework from Castle Project, which is essentially the Rails framework laid on top of C#. It’s an interesting way to get the benefit of having a team that understands tools like Visual Studio, and wants to have that very robust debugging environment, but wants to also take advantage of the flexibility that Rails offers. We have been able to have it both ways, if you will.

Scott: I think there has been a resurgence in the last few years of dynamic languages or interpreted languages. That programming model seems to be making a pretty strong comeback. When I got started with Unix, way back in the day, they were used for a lot of administrative things, and there were things like Tcl, Tk and some of that kind of stuff. Then it went very compiled. Now it seems to be going back towards more of a dynamic language approach.

Jamie: It seems we always have the same problems, but we want to solve them in different ways. We try to find performance and optimization in different parts of the stacks. People have been arguing for decades: could you make an interpreted environment that’s as fast as a compiled environment just by doing JIT-based compilation?

Maybe you can, maybe you can’t–I don’t know. That’s pretty academic. At the end of the day, you can’t get away from the fact that Facebook is written on PHP. There are very large sites like Twitter that do an extreme amount of requests per second, and it was build on Rails. I don’t think that you can state emphatically anymore that those environments do not scale. Clearly, they can, even if it takes a different set of tools.

Scott: There was an article I read a couple of years ago that was called “The Hundred Year Language,” or something like that, where somebody was speculating about what we’re going to be using to program in 100 years from now. His conjecture was it will be just insanely inefficient, but it won’t really matter.

Because we would all write in assembly, if we wanted to wring the most performance out of it. Whether you have to buy 100 servers or 200 servers to run it, if it’s very agile in terms of being able to scale, then scale out and focus on supporting change rapidly and those kinds of things. That’s a lot more important than just wringing every drop out of the CPU.

Jamie: One of my professors in college, John Reidl, and this was the early ’90s so I think he was a little ahead of his time, would say that CPU, memory, bandwidth and disk all scale to infinity. The only thing that doesn’t is programmer time. At the end of the day, the most important thing is to understand your code and to be agile with your code. Hopefully, you can solve everything else through other innovations.

Sean: One thing I’m curious about is usability, whether that’s from a management perspective, a development environment perspective, or an end user perspective.

When you’re thinking about acquisitions is it less about scalability, and more about usability? Do you feel like the open source community is best to drive things that have a high degree of usability and utility? Or do you feel that that’s the place where maybe they’re striving to, or they have limits that are inherent in the model?

Jamie: Well, I guess it depends on what level you define that usability. Some of the most opaque projects around are open source projects. The code is all available, but unfortunately, you have to actually look at the code to figure out how to do anything with it.

And the result of any question is, “Well, you’re an idiot. Go look at the code.” That’s the extreme, but I think that’s one of the places that the open source model puts an additional obligation on somebody like a CTO, or a VP of engineering.

With proprietary software, you can somewhat base the viability of the product on its market acceptance, and you get the benefit of the fact that either these other people have bought it or they haven’t. And they have reasons why they bought it or they haven’t bought it. And bad products die naturally, out of lack of adoption. In the open source model, that’s not necessarily the case. You can have a product that is great and just doesn’t get picked up, or vice versa, so it requires a vetting that’s a little bit different. What’s the velocity of the project? How many people are truly committed to the project? How willing are you to commit to the project?

I couldn’t tell you how many corporate CTO types I’ve talked to that would love to use open-source projects, but would never give anything back. And, if that’s your position, I think you should question the ethics of using open source software and just being a parasite. If you’re not willing to give back your additions, which in some cases is the requirement, that’s not good.

The Mac also has an interesting dichotomy. I’m an old-school Mac guy, and I’ve recently returned to the platform when they switched to Intel. I have commented to many developers that they need to go get a Mac, because I think that, in many ways, the most innovative software that is being built right now is being built on the Mac. In some cases, it’s just wacky programs like WriteRoom.

A great example is QuickSilver. QuickSilver re-defines how you would interact with your computer. It’s clear to me that the most innovative development right now is happening on the Mac. Yet at the same time, the thing that bothers me about the Mac is that it is positioned as the computer for the rest of us. It’s portrayed as the people’s computer, when in fact; it’s the most closed platform out there. You can’t build one.

I have always felt that people’s computer needs to be a computer that the people can build. So there’s this dichotomy. But it’s a great ecosystem. Really great software is being built on Macs. I don’t know if I have ever seen a Rails screencast that was done on a PC. It’s very clear that Apple has successfully gained mindshare in that developer community, and they’ve gotten a lot of market share as a result.

I doubt you’ll ever see any volume of Macs in server rooms. But at the end of the day, it’s Unix underneath, and that’s what gives it the credibility to be able to do things like Rails development.

Sean: It seems like there are a lot of open source acquisitions and even some open source projects that have a single voice at the maintainer level. Sun gets criticized for this a lot. You see a lot of companies put an open-source bumper sticker on their product. The “free” version is open-source, but that’s just to get some market share and up-sell you to the paid version. Do you see a trend that direction, or do you see it going a different way?

Jamie: I think one of the great examples there is Red Hat. I don’t want to say that they’re not fully imbued with open source in their blood. They are. But Red Hat Enterprise is as expensive as Windows. You come to the table with this idea that it’s Linux, so it must be cheaper, but in fact, no it’s not. That proves out that technology teams still need support. No matter how good the teams are or how deep your team is, from time to time, most teams are going to need support from an expert. So there’s a market to be made there in providing that support.

It also highlights that the traditional software business model is still valid. Microsoft’s and Oracle’s models are valid, and are a proven way to run a business. These companies are successful doing that. One other thing concerns me–Sun buying MySQL makes me wonder about the future direction of MySQL. Is MySQL going to continue to be a viable platform for the kinds of projects that it has been, or are they going to try to move it into a different market space?

And even MySQL, prior to Sun buying it, is an interesting example. I’ve noticed that SQLite has really made a lot of headway. And what’s happening, from my perspective at least, is that SQLite is picking up where MySQL has abandoned. MySQL decided to go more up-market and tackle bigger problems, and build in some features that are going after the Oracles and the Microsoft SQLs of the world. And that left a space for a much lightweight SQL engine, so SQLite is filling that gap.

So you could argue that, maybe, that’s how it should work, that when a void is created, there’s a new open source project that will fill it. But I do think that this need for an enterprise version of these various packages is really just a reflection on the support needs of organizations.

The big thing that they sell all that stuff on is FUD. When there’s a security issue, you want the verified Red Hat patch, so you’re going to pay a thousand dollars a year per box, or something like that.

Scott: When you look at the spectrum of open source projects, you’ve got some projects that are very, very old school. You’ve got Linux. You’ve got Apache Web Server and some of that stuff. They’ve been around for a long time. Extremely mature communities, high-quality code, and ubiquitous.

And then you’ve got kind of this next-generation of open source, to some degree–PHP, MySQL, Xen–a lot of these kind of companies, where the main driver behind the code base is a single company. And some of them are really what I’ve heard people call an aquarium model–like MySQL–where they don’t really have outside contributors. You can look at the code, but unless you work for MySQL, you’re probably not a contributor. A lot of Sun’s stuff is that way, too, although I think they’re actually working to get more outside contributors.

And so the question is, if you’re going to launch an open source project today, how do you take the world by storm with it? Take a look at Ubuntu building on top of Debian. It seems like the ones that catch fire are the ones where there is a revenue model and there is a business around the code base. What are your thoughts on that?

Jamie: I think you’re right, although it’s hard to know what’s the cause and what’s the effect. Is it successful because there’s this revenue model, or is there this revenue model because of something else? Are there examples of very, very successful open source frameworks that are not built around a commercial endeavor? Could you say Rails? Rails is fundamentally the creation of 37signals, and they’re doing commercial things with it.

Scott: Debian comes to mind; they’re not commercial. But then again, Debian caught fire to some degree when Ubuntu used it as their basis. There seems to be a bit of an uneasy relationship between the two communities. I think Debian is thankful for the attention that Ubuntu brought, but at the same time, the philosophies of the projects are different. Going the other direction, Canonical is very happy to say, “we’re built on this rock solid distribution. It’s a distro you can trust, but we also have got our own ideas.”

One of the things that I look at is that we are in this first wave, so you’ve got PHP, which has the chief sponsor Zend. You have Xen, whose chief sponsor is XenSource. Red Hat acquires JBoss. And now as these acquisitions start to happen, I wonder what the landscape will look like in five years. MySQL gets acquired by Sun, which has promised to open source everything, so they are moving toward an open source model.

XenSource gets acquired by Citrix, which doesn’t have an open source history and ethos. JBoss gets acquired by Red Hat, that’s sort of open from the beginning. Then maybe some of those companies get acquired by other companies. To me, it will be interesting to see how much of the projects will remain tied to the companies.

And what happens if Citrix ends up not being a good steward of Xen? Will they end up having bought nothing because there will be a fork that emerges?

Jamie: The other challenge that presents itself in those cases is, how many open source contributors want to contribute to Citrix? Probably not a lot. Once you tie projects into these corporations, the culture may resist it and go off in a different direction.

I think it will be a true test over the course of the next two years to watch the legal pushes on the GPL and the various software agreements out there. Will companies be able to weasel themselves out of that, or just identify ways to break the requirements of the GPL or the Apache license?

Scott: It’s interesting too that MySQL had a way of unlocking the GPL, so if you didn’t want to buy into the GPL but you wanted to redistribute MySQL with your proprietary code, you could just pay MySQL for that right.

On the other hand, if you wanted to adopt a GPL license yourself, then you could redistribute without paying MySQL the licensing fee. It’s a very kind of Wild West feeling. Everybody is trying a different model and a different way. I wonder if that’s the way it will stay, or whether things will consolidate over the course of a decade. How much does this disparity of business models and licensing for open source projects weigh on your decision making?

Jamie: It weighs a lot. It concerns me greatly that we would be leveraging an open source framework that could potentially die or be acquired by the wrong company. Of course, the acquisition angle can happen no matter what. Products can be acquired by companies that you’re not really excited about, so I don’t think you’re protected from that in any model.

But the bigger issue, in my book, is the longevity and the velocity of the project. For example, I think Ruby on Rails has an incredible amount of velocity right now. There are a lot of people working on it, there’s a huge developer community out there. But I ask myself questions like, “Will I be able to find a Ruby developer in 10 years?” I don’t know. Will I be able to find a Ruby developer in a year? They’re not out there at the same volume as other skills.

So in that case, the fallback is, how hard is it to teach people? If it’s a fairly transferable thing and it’s something that an average engineer who knows a few languages could easily pick up, then I don’t think there’s a high amount of risk. But if it’s something very esoteric, I wouldn’t touch it. The other question to ask is, at what layer of your architecture does that system live?

There is a certain level of the architecture where I would be concerned about the ability of an open-source project to provide that level of always-on availability. I have to admit that that goes in the face of Linux itself, since in many cases, Linux is the underlying OS. But it’s at a little bit of a different layer in the stack. Knowing the complexity of that layer of development and knowing that it’s something that companies have a very hard time pulling off, I would be hesitant to use an open-source load balancer.

Sean: Given that a lot of the work you do is with financial data, I can only imagine you’re an interesting target for a variety of folks out there who have malevolent intent. You’ve got two stacks–one that’s predominately a closed-source base and one that’s predominately an open-source base. What do you think about security issues around those platforms? Lately, this is a really, really hot topic. It always is, but it seems to ebb and flow.

Jamie: Maybe I’m too much of a technologist to look for it, but whenever somebody says their platform is more secure than everybody else, to me it’s just a bunch of crap. There are really very few platforms that have truly iron-clad security. And by and large, those are the platforms that have been around for so long that they’ve just been exposed to many, many tests. When you think of the Unix kernels themselves, some of them have not seen significant modifications for a decade. And those are probably pretty secure.

But the reality is, if you’re regularly modifying and releasing code, you’re going to have security issues. So we protect all our systems with counter-measures in the front and make sure that we have compensating systems to defend against all of that. So I wouldn’t go one way or the other based on this promise of security. In fact, I think it’s a little slimy for Red Hat to use security as the push toward getting people to buy Red Hat Enterprise. I think you should be selling based on features and not fear, but I think it’s a wash between the various platforms.

Scott: There are organizations that are building things on top of Linux or Apache, and they’re not being hacked right and left. Like you said, it may not have that much to do with the platform itself. It may have to do with the expertise of the organization like yours. You actually do the implementation and know how to build these perimeters of defense and that kind of stuff. Do you have an opinion on the notion of many eyeballs as it relates to security?

Jamie: My general opinion is there is some difference between the number of the eyeballs and the quality of the eyeballs. Just having thousands of developers looking at your code certainly is not going to qualify it as secure unless they really are qualified to say it is secure. I do think there is a benefit to that. At the end of the day it gets to these bigger issues, like what is the right way to develop source?

We know that in medicine, for example, it is not good for one doctor to just do everything themselves. They should consult and review their findings with other doctors and come to some conclusions. We certainly would not let one engineer build a bridge without ever having another engineer look at the code. In technology, we have this belief that it is our intellectual capital. The source code is the family jewels, and that is where the intellectual capital is located.

I don’t tend to really follow that idea. I tend to believe more that the intellectual capital is the engineers that are writing that code. I have actually gone on record many times saying that your source code repository is not intellectual capital, it is a liability. If you look at it on your balance sheet, you should be putting it on your balance sheet as a liability. The only thing you have to offset it is the intellectual capital you have in your engineering team. All code is just waiting to break. It is just a matter of time.

Scott: I have heard people say that code is like maintaining inventory. You have a warehouse full of stuff.

Jamie: Every line has a cost. In the future, is the fact that your code is open source really going to matter much to your business model?

You could argue that companies like Red Hat and MySQL are showing that it’s not the case–that you can have a very normal business model and have an open source code base underneath that.

Sean: Right. Someone would have to treat MySQL very, very badly for somebody else to do a meaningful fork, because nobody else is going to staff 200 developers on it.

I think Sun is going through that transition, where they’re realizing the value of Solaris is not necessarily the code itself. It is the fact that they have x-hundred number of Solaris developers that know how to work on it. I think Microsoft could do the same thing, if they open-sourced Windows.

Jamie: You could argue, what would happen if Microsoft just open-sourced Windows tomorrow?

Sean: I don’t think it would change very much at all. Again, nobody is going to fork that in a way that Dow Jones would trust this competitor to enhance, patch, and build the next version of Windows more than they would trust Microsoft to keep going.

The expertise is in the people’s expertise around that code base. That seems to be fundamentally what a company is acquiring. As long as they are reasonably good stewards of the code, I don’t think companies have to worry about losing control of a project just because it is open source.

Jamie: It is interesting, because I am a believer in this kind of long view of computer science. What I mean by that is it is still an incredibly immature field. It is much more art than it is science. We have not figured out the basics to our craft to the extent that people who build bridges have figured it out. We may find 20 years from now that the best practice is for all the code to just be out there and that is just how you do it.

Part of the benefit of that is the many eyeballs. Or part of the benefit is automated agents that are scanning code on the behalf of third parties, looking for things. It is hard to know exactly what that would be. In general, I do think this belief that your code is a huge asset you have to keep locked in a big safe is terminal. I do not think that is going to be the path forever.

Bookmark this:

Interviewee: Justin Erenkrantz

In this second part of a two part interview with Justin Erenkrantz we talked to him about:

• How the Apache project ensures good collaboration.
• The Apache Foundation’s philosphy of having no single person as the leader.
• Apache’s security committee.
• The process of removing someone from a position of responsibility within the Apache Foundation.
• What would make someone want to be part of the Apache Foundation’s group of projects.

Scott Swigart:  In this part of the interview, we wanted to dig into some of the tenets, if you would call it that, of the Apache way. And the first of those of course is collaborative software development.

So talk a little bit, if you would, about how Apache does collaborative software development. I’m sure some things are very traditional and similar to the way that other open source projects might do it, and there are probably things that also might just be a little bit unique to Apache. So how do you try to insure good collaboration?

Justin Erenkrantz:  So the main center point of all of the collaborative software development that we do in Apache is the mailing list. That’s where pretty much everything happens. As one of the guys mentioned before, the maxim has been: "If it didn’t happen on the mailing list, it didn’t happen." And that generally tends to be true.

Basically, if you follow the mailing list for a particular project, then we’re expecting that you should know what’s going on within the project. Within that, those are pretty much all public lists. Everybody can subscribe, even people who are committers, people who are just users, people who just work on another project that may consume the Web server or maybe PHP modules or something like that.

It’s pretty much an open forum. Anybody can voice their ideas. Generally though, just the way things work, most of the traffic tends to be from the core developers who are active at that time. The traffic patterns of the list change, and you get an idea of how much discussion. You generally see peaks and valleys for the mailing list discussion. Things get really heated or things are just chugging along and there’s not much traffic on the list.

So that’s where all the discussion should happen. And then of course there is the source code repository. And in Apache we always had the thought of having shared repository. A lot of projects now are starting to ‑‑ Git and Mercurial and all of these distributed version control systems to be centralized version control systems.

And that’s something that, within Apache, that goes against what our thoughts are, because we want to all be agreeing on, "This is the Apache version."

Scott:  Sure, no problem.

Justin:  So there’s no leader within the ASF. There’s no person, if you look at say, Linux and you say, "This is Linux’ tree," or this is Andrew’s tree or this is Alan’s tree. Instead there is just the Apache tree. So there’s really no concept of, "This is Justin’s tree," or someone else.

Scott:  Is the feeling then that essentially there should just be intensive discussion before something gets checked in? So when you’ve reached a point of consensus, it should get checked in rather than the way other projects work where different things get checked into different people’s trees. And then when it’s time to build a release, you have to pull stuff from these different sources to figure out what’s going to be in the release and what isn’t.

Justin:  Right. So, generally, what tends to happen is there are two states a tree can be in within Apache. One of them is "commit then review." And then there is "review then commit." And you’ll see some projects differ on particular trees.

For example for the HTTP Server, the trunk ‑‑ which is the main development ‑‑ is usually always under the "commit then review," which basically means that anybody who has commit access can feel free to go and make changes and they basically have the benefit of the doubt that the change is going to be good. And there’s generally an implied threshold: that if you’re going to make a really big change, go discuss it on the list. But if it’s a minor change or adding a little feature, that’s probably not going to be controversial, go ahead and commit that to trunk.

But for our stable releases, generally things that have already been released and we’re doing maintenance on those, are under the "review then commit" model. So that’s going to be RTC, and that means that any change to those trees has to be pre‑approved. That means you need to get three binding votes from other committers to say, "Yes, this is good change and no one has vetoed it." Technically you would use a file called STATUS, just a plain text file that some projects will use, that basically tracks all of the things that are under discussion to be back‑ported or added into this tree.

Scott:  Got you. Like any open source project, and this one is democratic, there is a vote to commit. Well, there is a vote if it is a release product. If it’s not a release product and people later decide it wasn’t a good thing to do, it can be reverted out.

And you mentioned before that part of your governance is that out of these 60‑something maintainers, one of them can essentially veto a change if they want to. So talk a little bit about how conflict resolution usually works. For example, when you have people who are—and I understand it doesn’t happen often—adamant one way and another person who’s adamant a different way. How do you see it play out that those eventually get resolved and things move forward?

Justin:  Generally what happens is, like before, the veto just tends to be a last resort. So let’s just say that someone makes a change to the trunk and I don’t like it. I might just say, "You know, we should talk about this change," or, "Here’s the problem with this." Generally, the person who has committed that says, "Oh, yeah. Here’s why I did it this way," and comes up with an explanation, and then through a process on the mailing list, figures out and resolves those conflicts.

That’s what you tend to see happen. And it’s all tends to be, for the most part, "I’m sorry, I forgot about that particular corner case or that." And everything tends to get resolved very naturally.

The veto tends to be when someone says, "No. I have to do it this way," and someone else says, "No, that’s wrong." That’s basically when the community is at risk of breaking down. But generally it doesn’t get to that point. Everybody is, "We’re all going to go in this direction; this is the right direction for us to go in. We want to add this feature. Let’s work through whatever issue you may have about this particular commit."

Scott:  One last thing before we move on to the next point. Other than the fact that everybody has their own private tree, is there anything else about the collaborative nature that you think is somewhat unique to Apache?

Justin:  We tend to do a roll call before the release. So at this point there’s been a review of everything. But then let’s just say that I want to release the next Apache 2.4 or whatever. Then basically what I will do as a release manager is say, "OK, I’m marking this as 2.4," and I produce all the artifacts. I produce the tarballs, [inaudible]generated files, whatever. And then I send it to the list and say, "Hey, is everybody happy with that?" And then that goes to a voting process.

There’s review at all stages, but there’s also a review at the point where you do a release, and you have to get, at least, three people to approve a release. One thing that’s different is that it’s not possible to veto a release. It’s strictly majority rule on the release.

Scott:  So in other words, there is veto capability on the individual check‑ins. But, as you said, when it comes down to doing a release it is a majority rule vote.

Justin:  Yes. You’ll tend to see someone voting ‘no’ on a release if it doesn’t work on Linux or something, and generally you’ll see it get stalled and it then gets fixed. But there have been a couple of cases when we did the release even though we knew that it didn’t work on a particular platform, and so we made a release note. But the veto does not apply to releases.

Scott:  Interesting. So moving on to licensing, one of the key things with Apache is the commercial‑friendly standard license. Talk a little bit about what that means.

Justin:  Basically within Apache, we like to have a big tent where everybody can come in and play with us. We think that the community that we developed within Apache is going to be the motivation for you to stay involved. For example, within the HTTP Server community, you have all these experts and Web servers, and if you’re part of this community, you get the benefits from them. And so there’s an incentive to play within the community, so that I don’t have to hire five guys and do a whole team; I can leverage the other people within the community.

But in turn, all those people who are part of the community say, "Whatever you want to do with the code is fine. We’re not going to get hung up if you make it a commercial product or an open source project. We created it and it served our needs, and if it serves your needs, that’s great."

Scott:  What springs to mind are things like GPL. Am I seeing it right in that Apache is more commercial‑friendly than GPL, V2 or V3?

Justin:  There are companies built around GPL licensed software. But what we tend to see are two classes of GPL products: In one, there is a real community, maybe within Linux, and they’re all happy to make all their changes available to everybody, and that’s a very good community. The other community you tend to see has a single stakeholder that has a prevailing interest in the GPL product, and they basically have an unfair share.

You can see this with some of the GPL projects that require copyright assignment. In order to participate, you license your changes in the GPL and you have to give a copyright assignment to the principal stakeholder. Now they are then free to release the commercial closed source based on your work because they have the copyright or whatever legal mechanism. There is an imbalance there when you look at those two.

Generally when you think about the GPL, you’re divided, with broad strokes, into those two groups. This is a real community but the other groups are aware and want to be clear about which one has a dominant role, and that’s one thing within Apache we don’t like to see. As our projects go through incubation and get added to the Foundation, one of the things we do is make sure that the community is diverse. In fact, there is not a single dominant stakeholder that can direct the project in any untoward way.

Scott:  Right. I can make changes to it, distribute it as part of a commercial product, and I would not be required to contribute those changes back to Apache. But under a GPL license, any modifications made require you to make the source code available. You cannot have closed source proprietary extensions or modifications of it. Any modifications you make, you have to open source and it has to be under the same license. So that’s the key differentiator?

Justin:  Yeah, our philosophy is that the community is what’s going to bring you and keep you there, and that’s why you’re going to stick around. If you released a commercial product around one of our projects it’s going to be to your benefit, to basically keep your commercial project as close to whatever we’re releasing.

Scott:  Right.

Justin:  You can pick up all the bug fixes and whatever improvements; you get those as a free rider. But in a sense, you are contributing whatever changes you’re making voluntarily back into the greater community.

Scott:  Yeah, that makes sense. Do you have examples of companies that have used different Apache projects because of the commercial‑friendly licensing, where they probably wouldn’t have it if the license weren’t so commercial‑friendly? Is that a topic that comes up?

Justin:  Absolutely. You see companies like IBM that release their versions of the Apache Web server or Geronimo under different names, but in the core, they are Apache projects. We’ll see that even with smaller companies such as Covalent that does commercial support. Basically, they added in a couple of extra things that provide support to their users.

One thing that the Apache community really does not focus on providing is 7/24 support. Covalent goes in with their business model and provides the support and training around these particular Apache projects. You will see businesses like JBoss using Tomcat. So you see all of these commercial companies using things that are Apache projects under the covers.

Scott:  Right, so that freedom has led people to be a lot more creative about how they structure their business. They have a lot more options in how they participate with the different Apache projects, how they contribute back and how they structure their own products. What is the relationship between the Apache Software Foundation and the Free Software Foundation? Is there any or are those fairly separate endeavors?

Justin:  There is no formal relationship. I’ve never had a conversation with Richard Stallman, but I’ve had conversations with Bradley Kuhn who used to be, at that time, the Executive Director of the Free Software Foundation. So there’s an informal get‑together of foundations to compare notes, and that’s generally a very good thing. How do we keep our ears open to what Mozilla’s doing? If they’re doing this new technique, then we can give them a call and ask, "What are you doing? We’d like to follow on it."

One thing we’ve been doing with the Eclipse is a joint conference. There’s going to be a conference in Asia that’s now scheduled for 2008. So it’s a way for us to get the communities talking to each other.

Scott:  Sure. So basically, if I can summarize, you guys get together around joint events and joint things where it makes sense. You share information because you’re all part of the open source community. Philosophically you may agree to disagree in terms of the details of licensing, commercial friendly, and that kind of stuff.

Justin:  Well and you ought to be using more…projects have different circumstances. Apache‑‑we have a very vocal membership and we have this and you compare that to let’s say the Mozilla which has a completely different governance structure. But if you look at Brian Behlendorf, he’s been on the Mozilla board for a very long time and he was one of the founders of Apache.

Scott:  Gotcha.

Justin:  So you have this intermingling of the communities. So someone like Brian Behlendorf who was brought in through the Mozilla and says here’s how we did things within Apache, and here’s his expertise and his experience that he got, he can share that with the other people within Mozilla.

Scott:  Gotcha, gotcha.

Sean:  Let me refer to one of the other tenants of the Apache way. I’m curious about this just because I was thinking about the conversation we were having. To state the obvious, you’re focused on producing software. I notice that you have a security committee that—if I’m reading it right and for lack of a better phrase—provides a service to all of the projects that are part of the foundation. And it looks like those projects can turn to the security committee and ask security related questions or possibly look for guidance from them, regardless of whether they’re Tomcat or some other piece of the foundation? Is that accurate or is that not accurate?

Justin:  It’s somewhat so. We have a security team, which I believe is currently a Board committee. But basically what they’re responsible for doing is ensuring our security at Apache.org mailing address gets responded to. And these are generally people who are very security savvy.

But there tends to be some people from Tomcat, from the HTTP Web server, from a higher profile project on this internal mailing list. So let’s say that, to give you an example, let’s say there was a security vulnerability in Derby and they could parse to those reps and say, "Hey, we have a security vulnerability. What do we do?" And so there’s expertise and, "OK. Here’s what you do. Here is your administrative contact. Make sure your mailing list… Go talk to…" Kind of a shared resource. But we’re not getting the focus on producing the fixes for the project but it’ll be "OK, here’s the responsible disclosure policy and an attribution policy." So that’s generally what their role is.

Sean:   Are they providing fairly prescriptive guidance but just not down to the ‘I’m going to change you’re code’ level because they don’t know the individual projects at that level? Would that group essentially be the center for discussions around a security development lifecycle for the Apache Software Foundation? And an attempt to pull those best practices together?

Justin:  Yeah. I think basically our project concern…we have something. What’s the process? What do we do? And that’s as an advisory role. OK, here’s the process and the procedures to follow.

Sean:  But it’s purely advisory, right? I mean one of the projects where they feel that their code is "secure enough", or they’ve looked at it long enough or they feel that they’ve handled it.  Then the advisory committee comes back and says, "Well, we really think you could take a look at this again." That’s where the communication would stop and it would be up to the individual project whether they want to take that under advisement or not.

Justin:  Yeah. I think so. I think record security, can you maybe at that point write back to your original reporter and say, "We looked at it and we don’t feel there’s a security vulnerability here." That may be…that has happened where we look at things and we say, "No. This is not an issue." But generally, really the security team is more of a reactive. So they’re not proactively performing security analysis on our code or anything like that.

Scott:  I just want to clarify. It sounds like they have a little bit of an all‑up policy for somebody sending an email to that address; somebody reports what they perceive as vulnerability or reports some kind of issue. They do a little air traffic control. They route it to the project.

Justin:  Exactly, exactly.

Scott:  There’s a general process that the different projects would follow. Basically that sort of happens and that’s one of the things that the security group advises the other projects on. Well this is generally a ‘way we do it’ sort of thing.

Sean:  Let’s go to a different piece of the Apache way, the emphasis on a technical‑based interaction. One of the things that I find fascinating about open‑source projects is the way that they exorcise community members that maybe aren’t following those rules.

[laughter]

Sean:  Because we got some interesting responses when we talked to people about it. It’s like, fine. We understand that everybody is an adult. We understand that everybody will try to handle themselves in an appropriate manner. But if anybody’s worked on a software project they know that not everybody does, right? So…

[laughter]

Sean:  Considering that you can learn a lot from a story…if you’ve got a story or two about either fully pulling the ejection handle on somebody that would be interesting to hear? Or a scenario where it just took serious counseling to get somebody pointed in the right direction.

I would be curious to see how you guys handle that. You obviously have procedures in place. But at times you have to go beyond those with some amount of intervention and I’m just curious how that played out.

Justin:  Yeah. So there’s one case that comes into mind but I’m trying to reserve the right to figure out how much of this has been disclosed.

Sean:  Yeah, Sure

Justin:  So I’ll tell the story and then leave people’s names out of it but I have to go back and see how much of this has been told. So recently within one of our larger, well‑known projects, there was a bout, to use the word, between two committers. And they basically ended up vetoing each other on everything. It’s like no, no, no and tempers got flared. And it got into a very unhealthy situation. They’re two very strong‑willed individuals.

And basically what happened is the PMC, so it was the PMC responsible for this particular activity, basically had to step in and say, "OK; we need to come up with some policy or come up with some new rules to get everybody back to ground zero. So it wasn’t a matter of ejecting anybody. It was never really an option that…basically what happened was they said, "Here are the ground rules. Here is…if you’re going to go do this you have to follow this set of rules. If you’re going to go do that, you’ve got to go follow this set of rules." Basically the community agreed to say we’re going to go and we’re going to voluntarily adopt these rules. But as a settlement process; lots of flames and a lot of innocent people getting…

Sean:  Right.

Justin:  …accused of things and that process. The other one that’s in our history was a project called Avalon. And this is one that’s definitely well known so this won’t be any issue about this one. Avalon was a container framework. And what happened was two individuals just did not get along. And they were ending up in what we would call a commit war, where they would basically be reverting each other’s changes as soon as they came in. And it is just this whole really poisonous environment and basically in that case, the community wasn’t able to deal with it. And so basically what happened there was they fractured.

And so that one of the individuals went off and he took his code and you know, we wished him luck and said have a nice life and he went off and then it was kind of some other people came along and they did a project called Excalibur, which was basically the remnants of this whole Avalon project. You will just see if you look at the mailing list traffic, you will just see this giant peak and then this sudden nothingness because the project got shut down because no one could play well with each other.

Sean:  Right.

Justin:  By the way what it is interesting is some of the veterans of Avalon, they really got involved in the greater Apache community, one of our new directors this year. He was in the middle of all of this, but during that whole experience, he was one of the people trying to keep things level and stuck around and this year he got elected to the board of directors.

Sean:  Since Apache is a large foundation, I’m curious about a different point. If you were a closed source company and you feel you are short on testers or short of security experts, you simply go to HR and put out a requisition and hopefully some good folks come back.

So has the foundation had to answer requests from projects where they say, "OK, look, we think we are geniuses on nine out of 10 of the things we need to do, but this one thing we really need people to help." How does the foundation help with staffing up a project in this type of case?

Justin:  It is more bottom‑up than that I think in the sense of the culture that we have. You will see some overlap between the HTTP Server committers and the Tomcat committers because you will see that ‑‑ sometimes the Tomcat committers came over and they say, "Hey, we need some help with HTTP." Well lucky us, we have some of the world’s foremost experts in HTTP server, and that basically got them within the communities.

So I think our community’s diverse enough to, "Hey I am looking for a person who knows SQL." OK, I am going to go on the Derby mailing list and say, "Hey, I need some help with SQL" or for something for build systems, I’ll go to Ant community. And there will be some of the people who are the foremost experts in that. That’s actually one thing of having such a large diverse community is that you can pretty much find someone who understands something about something somewhere within the Foundation.

Sean:   I figure that’s one of the advantages. You’ve got a massive talent base but at the same time it is segmented into the project, so the Foundation can help orient a little bit of that knowledge of where the talent base is.

Justin:  Yeah and as I said if you look out the social graph and it is a weird mix of people who are committers on Cocoon, maybe committers on Mina, then maybe on Gump and so you will see that developers themselves, the committers aren’t necessarily staying in their silo, there are some who do, but there are also just as many who will go to other communities and work on other projects.

Sean:   Well I have couple more questions but Scott can go ahead. I want to give you an opportunity to jump back in.

Scott:  So talk a little bit about standards because one of the other tenets or pieces of philosophy is faithful implementation of standards. Talk a little bit about what that means for Apache?

Justin:  Right. So this was initially when we started off there was HTTP and there was the IETF standards and that was when you have editor of the HTTP standard is one of the people behind the code base, there is the knowledge is going both ways in a sense is that we are that able to influence the standards process. But at the same point we also have some of us involved with the standard process and feeding those changes back and to the development and supporting those standards.

But since then, the initial, you see our participation within some of the key web server specifications, then probably most importantly our participation in the Java Community Process and that is, as you know, we have so many Java projects and that so many of our projects are implementing some JSR specification and our involvement within the JCP has been to ensure that we can implement the specifications and we have projects have representation on these expert groups that device these standards.

Scott:  Right. So it isn’t just like the standard shows up and then you figure out how to implement it. There is this two‑way street where you are shaping the standard because you guys have such a big, real world implementation. And meanwhile, the standard is telling what you guys do because they have their own stakeholders, but they are considering your recommendations and you want to conform to what they eventually approve.

Justin:  Correct.

Scott:  Yeah, I don’t know. I don’t have anything else specific. Sean, do you?

Sean:  No, not right now. I think this led us into some new stuff and from our end, we really enjoyed chatting about it. Justin, do you have anything you’d want to add or things you think we should address overall based on the theme of where we were going?

Justin:  No, I mean you did a good job of asking me the questions.

Scott:  I guess there is one final question. When somebody is starting or has an open source project, they can pick the license they want  they can do what they want for their community, things like that. What makes people want to be, in your mind, an Apache project? What is the draw I guess?

Justin:  Well, I think from my perspective, the draw is we handle a lot of the mundane governing structures and all this and the infrastructure and the licenses. And that is all essentially managed and I think you see a lot of open source projects like, "Oh we need to go get a foundation."

And that’s a lot of overhead, there is a lot of overhead to create a corporation, handle donations and handle essential infrastructure and that is what our goal at the Foundation at the broadest level is to provide support. So that these people who are working on all these different projects all they have to worry about is doing code. They don’t have to worry about, "Oh I need to go and buy a new server, how we are doing to deal with this donation or this tax policy." We try to deal with all of that. So I think there is a critical mass that works in our favor.

Scott:  Right, right and let them focus on the piece of it that they really enjoy, which is whatever this project is that they have come up with. They have a passion, like you said, for not having to worry about all the housekeeping stuff.

Bookmark this:

Interviewee: David Campbell

In this interview with David Campbell we talked to him about:

• His background with the SQL Server team and Microsoft
• What about planning, implementation, testing, delivery, and servicing do you think are unique to a product as high profile and critical as SQL Server.
• The Security Development Lifecycle and SQL Server.
• Usability and SQL Server’s Development.
• His experience with OpenSource databases.
• How much the experiences of mySQL and other open source databases have affected SQL Server’s development.

Sean: Tell us a bit about your role at Microsoft with the SQL Server Team and as a Microsoft Fellow.

David: I have been working on SQL Server for roughly 13 years in a variety of roles but my passions are in systems and product development. Currently, I am heading up a team we call “Strategy, Infrastructure and Architecture” – SIA for short but I’ve worked on SQL Server as an SDE (Microsoft lingo: SDE = Software Development Engineer), Development Manager, Product Unit Manager and General Manager overseeing a number of component groups within the product.

Sean: SQL is a product upon which enterprises bet their business. So much of what’s mission critical depends on SQL Server. What about planning, implementation, testing, delivery, and servicing do you think are unique to a product as high profile and critical as SQL Server?

David: In the business world, enterprise database products define “mission critical”. A crash in many line of business applications may result in a disruption of service until the server or application is restarted but a severe crash in a database server resulting in data loss can cripple a business. I learned this lesson the hard way before coming to Microsoft when I worked on database systems at Digital Equipment Corporation (DEC). One day I received a page from our support staff and found that a product defect had resulted in a production line shutdown for a major semiconductor manufacturer. They had to send folks home; some of the physical equipment suffered damage as a result of stopping the production with partially completed batches of material and they reminded me that they were losing roughly a million dollars an hour in business while things were stopped.

Getting to your question might require a little background on SQL Server. Some folks might know that Microsoft hired a bunch of people from the database industry in the early to mid 1990’s to work on SQL Server as Microsoft tried to become a player in enterprise database systems. We acquired a source license for Sybase 4.2 and shipped two versions, SQL Server 6.0 and SQL Server 6.5, on that architecture. We hired a number of query processing experts and they started building a completely new query processor from a clean sheet of paper. I worked on the storage engine team and SQL Server was getting beaten up in the market around this time since we didn’t support row level locking as the Sybase architecture we inherited only supported page locking. I was responsible for the design of the row level locking feature for SQL Server 7.0 and the more I dug into the Sybase architecture the more challenging the design became since the entire Sybase transaction and recovery system was predicated on page level locking and it was very difficult to do a clean row locking design without a number of major compromises. After a number of sleepless nights and arduous design meetings we ultimately came to the conclusion that we would have to rewrite much of the Sybase storage engine to do this feature correctly. As part of this major architecture shift we made a decision to change the on disk format for SQL Server 7.0. This meant customers would have to unload and reload all their data as part of the upgrade to the new release.

So, now we have the context to start answering your original question. We started what would become SQL Server 7.0 with 2 strikes against us: It was really going to be a V1 product with a V7.0 name and we would require every customer to completely unload an reload their data to migrate to the new product. We knew that poor quality would mean strike three. Since we had a number of people that had experience building enterprise database systems we weren’t lacking in design knowledge so the real success factors for the release came down to doing a great job of architecture, implementation and validation. On the validation front we did a number of interesting things that could probably fill a book but I’ll highlight a couple of them here.

Given that we were going to make everyone migrate their data we knew that we need to make the data migration process both highly performance and rock solid. We engaged our sales force to ask our customers to help us by sharing their databases so we could convert them to the new format in the lab. We called this the 1,000 DB challenge and we wanted to get 1,000 real customer databases of all different sizes and complexity so we could run them through a database conversion lab that we created. The other interesting thing we did in this space was to write a playback system. This consisted of a capture utility that would log the customer activity on a production database and then a playback utility that would allow us to play back the actual customer workload against the customer database in our lab. We could play the workload back in “real time” which included the dwell and think time between queries, or “compressed” where we just jammed the queries into the server as fast as we could. In this mode we could stuff a day’s worth of work into the system in an hour or so and really stress the server. We’d ask customers to take a backup of the database that corresponded to the workload and to capture the actual work using the capture utility and then send us the database and capture log. This data allowed us to test the conversion of the database from the old version of SQL Server to the new version and also let us test the new version of SQL Server by replaying the actual customer workload on the new system. Later on we started to capture query performance of replay on the old vs. new version of the product to find performance bugs.

The next interesting thing we did on the validation front came from Don Slutz, a long time database veteran that was working in Microsoft Research at the time. He wrote a program he called RAGS that was really a model based testing system that used the SQL language grammar and the schema of an existing database to generate bizarre, but syntactically legal, SQL statements and feed them into the query processor. Basically, he married the state domain from an existing database schema with that specified by the SQL grammar and was able to probe all the dark corners of the search space programmatically. He wrote an MSR technical report that is available on the Microsoft Research we site. The way this played out was pretty interesting. At first it was pretty easy for Don to crash the query processor. So he filed a bunch of bugs, the developers fixed a bunch of bugs then Don ran it again, etc. After a couple of iterations of this cycle RAGS needed to generate some pretty ugly queries to crash things. You’d wind up with these 5 page SQL queries that looked like random gibberish but was legal SQL syntax and the query processing team would spend a bunch of time figuring out what the query was supposed to do and then figure out why it crashed the system. Later on Don used RAGS against different versions of SQL Server and other database products generating queries over equivalent schemas and comparing both the results and the performance of the different products across a wide range complex queries.

I could list more if you guys want to write a book…

Sean: It’s hard to imagine a product that has higher security requirements than a database server. It has to talk on the network, and it has to be able to store sensitive information. Since the introduction of the SDL, SQL has seen a dramatic reduction in vulnerabilities. How does the SDL play out on a day to day basis? How does it affect your architecture?

David: Great question! To fully appreciate the change you have to understand that 10-20 years ago there was very little widespread security knowledge in the software development world. In some sense the environment didn’t require it; most systems weren’t interconnected and remotely accessible. This meant that security breaches required typically physical breaches and people had a model in their head for physical security. It was easier to understand locks on doors than buffer overruns. What is interesting about the SDL and SQL’s evolution is that we’ve gone from a period in which security review and validation was done mostly after the coding was done to today’s world where we formally design security threat models as part of the design process before writing code. We also have a great training program in place that everyone touching the code needs to take. We also have refreshers that developers must take for emerging threats. Additionally, we’ve developed a great set of static code analysis and run-time tools to avoid and detect potential security issues. In 2002 we had a “security push” where the entire development team stood down and reviewed every line of code in the product. We did a smaller push in 2004 for SQL Server 2005 and with our current release we have truly integrated the security best practices into the development process and don’t need a separate security push as security is simply part of our day to day process. One challenge in the security space is that the threats are constantly evolving so we can’t rest and, as long as the bad guys are learning new tricks, we need to up our game and having a process in place for rapid mitigation in the event of a new threat or vulnerability.

In terms of how security has affected our architecture we are much more mindful of the threat environment each line of code is executing in. For example, there is a small amount of code that performs the initial client authorization before allowing a connection into the server. Since a remote client executes this code pre-authorization, any remote code that can access the port over the network can execute this portion of code. Architecturally, we strive to keep this code to a minimum and it is very thoroughly reviewed. Similarly, the security base is designed in a layered fashion so you have smallish amounts of thoroughly reviewed code providing services that other aspects of the security system are built upon.

Sean: How would you respond to a statement like, “If you really wanted to make SQL secure, you’d forget about the SDL and just open-source it so that many eyeballs could look at the code.”

David: I think there is some benefit to having more eyes on a piece of code. There is benefit from each person’s fresh perspective, benefit from varied knowledge, etc. However, eyeballs alone are not nearly enough. The SDL process evolution has been really interesting in that we have changed the culture, habits, and processes of every Microsoft developer in a way that is much more effective than having many otherwise competent programmer’s looking at the code. With SDL we have many eyes looking at threat models, many looking a the design of a security feature and ultimately, many looking at the code following a pretty rigorous and proven process. I think there’s sufficient independent objective evidence to say that SDL is working for us. If you have a few minutes go to the National Vulnerability Database at nist.gov and check out the vulnerability reports for SQL Server vs. other databases. I’ll warn you, if you try to search for Oracle flaws they are hard to find since every major and minor release of the Oracle database server is listed separately so you need to do a little aggregation to get the real picture.

Sean: It’s one thing to design powerful functionality, it’s another to make it easy to use. Talk about how Microsoft insures not just functionality, but usability.

David: OK. Now you’ve hit on one of my personal passions. I believe that many technologies go through stages of evolution as they mature. I define three major phases – nascent, developing, and refined. There are many examples that can serve as a lesson for software developers.

Consider televisions; thirty to forty years ago when TVs were a nascent technology you almost needed to be a technician to own one. Certainly, you needed to know how to take the back off and pull the tubes out so you could go to Radio Shack and test them when one of them went bad. Furthermore, there were knobs on televisions that were there solely because the technology hadn’t matured to the point where they weren’t necessary. I often ask audiences how many people miss the horizontal and vertical control knobs on their TV and we’re getting to the point where many in the audience don’t know that these knobs even existed to keep the picture from rolling and waving in early TVs.

Twenty years ago TVs entered the developing age as they became fully solid state. They were much more reliable and the technology developed to the point where many of the knobs disappeared. In this phase they were good enough for the masses. You didn’t need to be a technician to own one but it was nice to have one in the neighborhood. Frankly, I sort of think this is where PCs are today.

Today’s TVs are refined in that the user’s control surface captures the user’s intent rather than exposing the control surface of the underlying technology. For example, instead of fiddling directly with color temperature, saturation and hue to adjust the picture, my TV has a control that asks me if I want to watch sports, movies, or regular programming and adjusts the color parameters accordingly. Furthermore, some TVs are aware of the operating environment such as ambient light and compensate for that. You can do this same thought experiment on automobiles, microwave ovens, etc. When viewed through this perspective, most system software still has a long way to go to become a refined technology. Of course, there are cases where these maturity phases ripple and repeat through a single technology where advances happen in waves. I think my new Smartphone is an example of that. It’s way more capable than my previous cell phone but I never had to reboot my earlier one.

So, how are we doing on SQL Server? One simple example is the work that we did in the database engine in SQL Server 7.0 when we got rid of many of the knobs and made many of them self-tuning. SQL Server 6.5 had roughly 100 knobs whereas SQL Server 7.0, which was much more complex in many ways, had roughly 20. Many people thought that more knobs meant more control but, in reality, we found many systems were performing poorly in the field due to mis-configuration. We classified the knobs into those that should simply take care of themselves – things like the number of locks the server could allocate when it booted, the number of hash buckets in the cache manager, etc. These were our “horizontal and vertical control knobs”. For other knobs we set them up where the database server managed them by default but if an administrator wanted to impose constraints he could. The amount of memory allocated to the server was in this category; by default, SQL Server manages memory in cooperation with the demands in the operating system but you can set low and high watermarks if you want to. In other areas we used control theory and feedback loops to have the system adapt dynamically to the environment and control things based upon instantaneous system response. The adaptive systems work we did was really interesting in that existing control knobs were often static; a good example is “sort memory”. In many database systems before SQL Server 7.0 you set aside a portion of memory for sorting to perform queries and build indexes, etc. During times where you weren’t building indexes or didn’t have queries that required a sort that reserved memory was just laying fallow and wasted. Further, if you had a sort that needed a little more space than what you had reserved it would spill to disk because the sort wouldn’t fit in the reserved space – even if there plenty of memory unused elsewhere in the system. In SQL Server 7.0 we created an internal memory broker that could use server memory for whatever purpose made the most sense over time so things like the procedure cache, workspace, sort, and the buffer pool all cooperated to use memory in the most efficient manner over time. The result was fewer knobs and a more efficient system.

This was great work but we ran into a situation where we were ahead of the market in many respects. DBAs were worried we were going to put them out of a job. They thought they were getting paid, in some part, to respond to their pager at 2:00 AM because a big batch job failed because they hadn’t allocated enough lock blocks for the server. Our competitors also used this maturation against us – things like, “How can SQL Server be a real enterprise database system – it only has 20 knobs and our system has 500!”. What’s interesting is that if you look at the major database systems they have all made major investments in self-tuning and ease of use.

Market dynamics also demand that we make these systems easier to use and self managing. As an industry we’ve expanded database deployment from an installed base that was likely measured in the small 100,000s of units 20 years ago to one that is likely measured in 100,000,000s of units today. They had better be easier to use than 20 years ago!

Scott: How do customer needs and requirements make it into the planning process? How do you handle situations where the customer is asking for the wrong feature? (The customer asks for a setting so they can tune X, and you realize that if a subsystem was redesigned, they wouldn’t need to tune X)

David: We have had many situations where customers have asked for features that they have seen in other systems that didn’t make sense for SQL Server. Often customers ask for performance features that other products have that may provide a large advantage in those products but, given the way that SQL Server is architected, these same features may provide little to no benefit on our architecture. One example is raw device and partitioning support. 20 years ago many UNIX systems didn’t have advanced I/O features such as the ability to avoid the file cache, scatter/gather I/O or great asynchronous I/O. In fact, many early UNIX file systems had 32 bit file offsets so the maximum size of an individual database file could only be 2 or 4 GB in size. NTFS, in contrast, supported 64 bit file addressing, great asynchronous I/O, and the ability to do unbuffered I/O from the start. So, whereas other products needed separate partitions over multiple files with an I/O thread per file to simulate asynchronous I/O – SQL Server didn’t need any of this. This didn’t prevent customers from asking though. Of course, once you get up to very large databases, partitioning makes sense for a number of reasons such as the ability to physically manage large tables in index in smaller pieces but the point is that SQL Server didn’t need partitioning to get I/O parallelism in the same way some other systems did. We had to educate many customers on these points so they understood how our architecture achieved the performance that other systems did but through a different architecture.

I’m also mindful that control doesn’t always represent progress and often simplicity is the best approach – especially when it affords an opportunity for the software to do a better job. I think one good example involves a feature known as “tempdb in RAM”. Prior to SQL Server 7.0 you could allocate a region of RAM to hold the temporary database which is used for scratch tables and intermediate query results. In certain environments, placing “tempdb in RAM” could provide a significant benefit given the way that SQL Server 6.x was architected. Unfortunately, it was easy to under or over allocate the amount of memory and wind up spilling to disk or paging excessively. Another point is that the amount of RAM allocated to tempdb was statically determined; once set you needed to reboot the server to change it. Since the amount of space needed for tempdb varies depending on the workload, the optimal caching strategy for tempdb is dynamic. We removed “tempdb in RAM” in SQL Server 7.0 and did a number of other optimizations under the cover to better manage tempdb pages in memory so the actual customer result was much, much better across a wide range of scenarios for SQL Server 7.0 but customers who had improved their performance 2-3x by using the tempdb in RAM feature in SQL Server 6.5 screamed loudly. I finally wrote a long mail and included some experimental results that proved that the new approach was better but I still received hate mail for several years after that decision.

Scott: Areas like the developer division have strived for greater transparency. In open source, all development and decisions are transparent. Talk about how SQL Server views transparency during development, and how you have to balance expectations from customers who want full transparency, vs. not shooting yourself in the foot by disclosing early and giving a closed source competitor like Oracle an advantage?

David: You touch upon a very real challenge. SQL Server has matured to the point where we do a very good job on the fundamentals and, as a result, it’s more important that we listen to and work with our customers to continue to produce a product that helps them better run their business. We’ve talked mostly about the core relational database engine thus far but today’s SQL Server includes data analysis, data mining, reporting, enterprise class ETL, etc. The solutions we deliver in this space touch a broader range of customers and the pace of innovation is much faster than that seen in the core relational database engine. As a result, we need to be much more in tune with our customers to produce the right product. We’re making some real progress on including key customers and experts in our design process. For some of our more complex SQL Server 2008 improvements we’ve done joint design with key customers and MVPs and they’ve helped us make many of the tough scenario and design tradeoffs required to deliver the right feature. Done well, this reduces the need for iterative field testing to get solid feedback on new features.

Disclosing early is a real risk and, yes, our competitors do listen and respond. It’s funny, they pay much more attention to what we say now than they did 10 years ago.

Sean: What experience do you have with Open Source Databases?

David: Yes, I have experience in open source and certainly follow the key open source databases from the perspective of technology evolution and adoption. I don’t look at any source code from the open source databases to avoid any potential IP issues.

Sean: What do you think are some things that are easy to accomplish in a closed source model that would be challenging in open source?

David: I think one thing is something I would call “consistency in the large”. This isn’t necessarily easy in the closed source world but in a coordinated engineering environment you can align things in a way that lead to a degree of consistency which creates customer value. For example, the products within Microsoft’s Server and Tools business have a set of “Common Engineering Criteria” (CEC) that we all follow. The criteria include things such as common processes and UE content that lead to a more uniform customer experience and aligned features, such as a Best Practices Analyzer or having a management pack for our management tools when we RTM. This sort of broad consistency doesn’t happen organically and is one of the challenges that large scale open source efforts have to wrestle with.

: How much is SQL development and features informed by things like MySQL and DB2? Are there features in SQL that were inspired by these products?

David: Certainly; and it goes both ways. Things like persisted views have been done by all major database players. I can’t recall whether Oracle or IBM did it first but now SQL Server, Oracle, and DB2 all have a form of persisted views with varying degrees of updatability and query matching sophistication. Oracle was the last major database player to have a credible fully cost based optimizer. I would say that SQL Server led in the advancement of real self-tuning and ease of use. Unfortunately we did it in 1998 and tried to sell it to a market that didn’t understand its value and IBM did a great thing (for IBM) later on by coining the phrase “autonomous computing” and selling the value. I don’t think MySQL has contributed too much yet to the big 3 but I do like MySQL’s notion of installable storage engines with implementations optimized for various scenarios. What’s interesting is that SQL Server 7.0 was architected to support this concept with the OLE-DB interface between the relational and storage engine but MySQL has really gotten value from the concept.

Sean: If you had the opportunity to borrow more from the development model of the open source community when it comes to SQL Server what are the top couple of things you would like to bring over to SQL Server development that you see out there today in the Open Source community?

Scott: In general I like the notion of tapping into a large development community’s collective energy. Different people have different passions and, if you can harness the energy effectively, it can lead to some interesting results. For example, if someone gets excited about adding a specific feature or fixing a particular bug they can often just make it happen. Of course, one challenge in the open source world is maintaining architectural and feature consistency. Smart people don’t always agree on what’s important for the customer or know how a particular feature should be implemented in a way that maintains the system’s design tenets. I think a key aspect of great design is in providing the most value with the simplest and most intuitive implementation and user model. The open source project maintainers have a very tough job keeping this in check.

Another aspect of the open source world which is powerful is the ability to generate an ecosystem of tools and add-ons around a core technology. This is an ecosystem effect that may or may not require open source access to the core technology. For example, imagine if MySQL were a closed source project but that it had a vibrant open source community building installable storage engines for various scenarios. We are starting to do some of this with SQL Server through Codeplex.

Sean: How much do you think SQL Server’s development has been impacted by Open Source development efforts in terms of how they build community around various database products?

David: Enabling the community to help the community and giving the user base a direct voice to the development team is one of the most exciting and powerful concepts to come out of the open source development model. We have learned a lot from this and in our product review meetings we regularly review input from our community including how they vote on design change requests. Rather than us guessing or asking a small sample we can now reach out to our community quickly and efficiently. Frankly the Internet and access to our community have enabled us to produce a much better product. Things such as Watson, SQM, and our community product provide us direct feedback that allows us to respond in ways we couldn’t even imagine 10 years ago.

Scott: Give us some understanding of the structure of the SQL Server development team in terms of some high level estimates in terms of the number of testers, developers, number of machines in the test lab, length of time it takes test suites to run, etc. Just so folks can put the SQL Server development effort in perspective to other efforts they are familiar with.

I do not have exact numbers in my head but let’s just say we have 100’s of developers, 100’s of testers and probably 10,000 machines in various test labs. We literally have millions of individual test cases and the test system is highly automated. Many of our test machines are in offsite data centers and we’re able to connect to and control them via IP KVM. We have evolved our test methodology to include much more model based testing rather than individual test case generation… One interesting thing is that we completely revamped our development methodology between SQL Server 2005 and the current release. This was a huge cultural and process change and we have learned a lot from this experience. Perhaps we can chat a bit out this next time.

Bookmark this:

If you are looking for subjective opinion, I recommend looking through the interviews we’ve done here. At the risk of sounding like a Microsoft fan-boy, the Microsoft interviews (in my opinion) demonstrate a company where secure coding is “in the water”. Code goes through threat modeling, risky function calls have simply been banned, code goes through automated and human inspection, and vulnerabilities that do slip through feedback into the process to determine how to prevent them in the future.

I simply don’t get the same feeling from the open-source people we’ve talked to. When we’ve brought the subject up, the response is almost universally “many eyeballs,” and faith (without data) that “many eyeballs” is effective.

Am I completely off base? Do things like the Linux kernel and Apache go through rigorous security reviews? Is there proof that “many eyeballs” in open source is at least as good as something like the Security Development Lifecycle in Microsoft? If you’re in a position to know, let’s chat!

Bookmark this:

ComputerWorld just did an interview with Microsoft’s Scott Charney, which provides more insight into their efforts to produce secure products.

Bookmark this:

Interviewee: Ryan Waite

Ryan Waite

In this interview, we talk with Ryan Waite, Group Program Manager for High-Performance Computing at Microsoft.  We talk about:

• Microsoft’s entry into the High Performance Computing area.
• Microsoft’s support of the open source Message Passing Interface (MPI).
• MPI has become bloated, and there’s no usability data (like there is with Microsoft Office) to indicate what should be deprecated.
• At Microsoft, select customers help shape the features for the next product release.
• Microsoft has always focused on ease of use, simplified administration, and uncomplicated setup. HPC is no different.
• Instrumentation helps find reliability issues. If partners are affecting reliability (with unstable drivers, for example) certification helps raise the bar.
• Parallel programming is now the only way to do more computation in the same amount of time. Processors just are getting faster like they used to.
• It’s easier for open source to be incremental. Proprietary software has to make radical changes.
• Hard drives + overnight shipping = really high bandwidth.

Scott Swigart: We are conducting a series of interviews to explore the premise that open source software and closed source proprietary software are built differently — that the software development life-cycle is different, and that difference is likely to manifest itself in concrete ways in the final product.

That’s not to say that one’s better than the other, but simply that there are certain expectations you can have from either open source or closed source software because of the way it’s built, as well as certain inherent limitations.

In the past, ISVs pretty much just wrote closed source proprietary software, but today they have a little more of a choice in how they build a product, and we want to look at some of the decision points around that. And if you’re looking at bringing software into your organization, and you have a choice between something open source and something closed source proprietary, again, what are some of the common decision points or expectations around that?

To get us started, tell us a little about yourself and what you do, to give us a little bit of context.

Ryan Waite: I’m the Group Program Manager for High-Performance Computing at Microsoft. What that means is that my team is responsible for writing the technical specifications for our products; we design which features are going to be necessary for the product, then we write up how those features should work.

We work in close partnership with the software development and software test teams. The development team, as you can imagine, writes the code itself, and the test team tests that code. I should also mention that my product actually combines both closed and open source software — both Microsoft-designed as well as open source software.

My staff is comprised mostly of computer scientists or computer engineers of one kind or another. I’ve been at Microsoft for over 15 years, and I’ve worked mostly on server products, including things like Exchange Server, BackOffice Server, Windows Server, and Small Business Server — all sorts of different pieces.

And the product I work on right now is called Compute Cluster Server. As an HPC product, Compute Cluster Server allows you to bind 50 or 100 or 1,000 machines together to work on a single computational problem or a single set of closely related computational problems. So it’s basically a category of supercomputing.

The idea is to take very large computational problems, like proteomics problems, genomic problems, problems in mechanical engineering, weather simulations, and computational finance; and you can then run these against this huge set of machines. And the great thing about these kinds of clusters is that they cost a fraction of the cost of traditional supercomputers. But clusters are very powerful, and they’re very well suited to certain types of programming problems. These kinds of clusters are based on the concept of what are called Beowulf clusters, developed back in the early ’90s at NASA. The idea is to bind together a bunch of commodity servers with a high-speed network, and you can run some very interesting computational problems on them.

Most problems can be designed to work on a cluster, although there is still a category of supercomputing problems that you really do need a traditional supercomputer for.

Scott: Thanks. You mentioned that you have closed source and open source solutions; can you talk a little bit about what you do on each side?

Ryan: Loosely speaking, you can think of our product as having three major components. One is what’s called a job scheduler, which schedules work across this big cluster of resources. If you have 1,000 machines, it’s not like every job takes 1,000 machines; some take 16 machines, some need 64 machines, some need 132 machines. And then they have time limits for how long they want the machines to go — for example, they might need eight hours of computational time. So we have a job scheduler.

One of the gentlemen on my team was an architect on the LSF job scheduler, at a company called Platform Computing — he’s a rocket scientist. The second thing we’ve built is systems management functionality, and this allows us to manage this big cluster of machines. There’s a woman named Cathy Palmer on my team who was at Cray and Terra Computing for 11 years and has a PhD in job scheduling; she’s a rocket scientist, too. These are some of the best people that ever I worked with, so I’m just going to glow a little bit about how great they are.

The third area is about high-speed networking, and the parallel programming model itself. That’s headed up by another gentleman on my team named Eric Lantz. Part of that is high-speed networking. On the high-speed networking end of things, you need to enable these machines to talk to each other very quickly, because sometimes the speed of your computation is dependent on how fast the nodes can communicate with each other as they’re running their computations. It’s not just about the CPU inside that machine, but sometimes the I/O, or the communication speed between those machines.

For high speed, tightly coupled communication there’s something called MPI. MPI stands for the Message Passing Interface. This is basically the communication interface used for people to write these kinds of massively parallel applications that run on clusters. There are some standards for how MPI is implemented, but the reference design comes from Argonne National Labs. And Argonne National Labs distributes that code under a BSD-style open source license.

So Microsoft is a licensee for that code. We’ve taken that open source code from Argonne, and we’ve made modifications to it and we ship it as part of our product. The folks at Argonne are genius UNIX and Linux developers — really great UNIX developers. They’ve been doing this kind of stuff for years and years. And message passing is a tremendously complicated area of computer science, and they’ve done a great job at building this code. They call it MPICH and MPICH2.

The problem is that they’re not Windows developers. They don’t have a lot of experience with that. They have one person over there that is their Windows developer, but we have a lot of experience programming Windows and can certainly help. And Windows and Linux, are different operating systems, so you write programs for them in slightly different ways. Not radically different ways; in both cases you use C or Java or C# or whatever to write your applications.

But the thing to keep in mind with them is that you have different efficiencies in different kinds of ways. And so one of the things we’ve done is to implement those efficiencies in the Windows version of MPICH. All of this allows us to have a very high-performance solution.

We’re about to contribute those changes back. We’ve just been working through some of the final issues with how we transfer the code back. It’s just logistical stuff. In the next month or two, you’ll see a press announcement go out that talks about our contributions to this code, these modifications that we’ve made back to Argonne. As far as I can tell, this will be the largest contribution Microsoft has ever made to the open source community.

With this particular code, we want anybody running Windows to have access to a high-performance MPI library. If they buy it through my product, that’s great; if they are just running Windows and they want it, then they can get it from Argonne.

Scott: You’re in kind of an interesting position, because you’re one of the few Microsoft people developing both closed source proprietary and open source projects. What are some of the things that stand out to you as key differences between how those two processes work?

Ryan: What’s interesting about MPI is that there are a large number of features that have been driven because somebody in a standards body or open source group thought it was an interesting feature for what they were doing. They’ve implemented particular APIs.

Now, that API may never be used by anybody else except them. And yet it becomes part of this code base. The really great thing about that is that if you need to do a very specialized API for the application that you’re writing, you can go ahead and contribute it to Argonne — and if anybody else in the world has that same kind of problem, they might be able to use that same piece of code that you wrote.

The disadvantage of that is that this is now a very large code set with over 200 APIs. It’s a bit of a joke, and I don’t have any proof that this is true, but the joke is most developers use six or eight of the MPI commands out of that 200 for the bulk of their programming, so there’s a whole bunch of stuff that just isn’t used. And part of the problem is that you have to maintain all of that code as it marches forward, making sure all those APIs work.

They’re all documented, they’ve all been shipped, and ideally, they all continue to be well-supported. I think that’s the struggle with that kind of model — the code has exploded pretty quickly. It exploded because everybody gets to contribute the little things that they need into that code.

Scott: That’s interesting, because in other open source projects, the maintainers tend to be pretty strict and conservative about what they’ll let in.

Ryan: There is no single person for MPI saying, “this is OK, this is not OK.” You don’t really have the same kind of maintainer making things cross a very high bar before they get included. And those bars can take many forms, other than just the quality of the code.

Sometimes people establish a quality bar for when code gets included and sometimes it’s a customer requirement bar — you might want to make sure that if people are going to use something, it works properly. We all have to have different reasons for what we decide to include in software and what we decide not to.

I think it’s just the nature of these projects and the way software projects mature. They get bigger. Take the Linux kernel — it’s huge compared to what it was five or ten years ago.

As these products become more and more general purpose, they have to be able to service the needs of a greater and greater community. And we have an enormous amount of experience with this at Microsoft with products like Office, for example.

We have more usability data about how people use Office than you can imagine. Because we shipped instrumented versions of Office during the beta period, we know exactly which features people use and we know exactly how they use them.

So, even thought there may be a thousand features inside of Office, we know that every one of them is used. Everybody may use a different 20 percent, but we know that all those features are well-used. And features that aren’t used, we deprecate or remove from the product. We have very concrete usability data to tell us that.

Scott: Contrast that with the closed source work that you do. It seems like Microsoft goes through a process where there’s a big wish list of features, and people start coding on things as the project marches toward release. Things get cut, and developer resources get put on the more critical things.

How do you see the closed source work that you do comparing with the way you’ve seen the open source projects developed, in terms of how features are spec’d, how they’re built, how features are cut, and how things make it into the release product?

Ryan: That’s a hard question to answer, because it’s so different depending on which open source group you’re working with. I’d say for us, the first step early on in a project is to figure out whether a product is going to be more driven by a release date, or more driven by a feature set.

We do both at Microsoft. If you’re more driven by a release date, you really make sure that you scope that product so you’re going to do well at hitting that release date. And Office is a good example of a product that really drives to release dates.

The video game industry is an even better example. If you look at Xbox or you look at any of the game title providers that are building stuff for Xbox, they have to hit Christmas. No ifs, ands or buts. Otherwise, they’re not going to recoup any of their development costs, and those companies could fail. It’s really important for those organizations to scope their features to a particular date.

For other projects, you might look at customer requirements and pick a date, but you’re really going to drive to implementing that set of features and making sure that they hit the appropriate level of quality. I think that as those projects move on, that’s where you may begin to see some features get cut in order to hit a release date, or just because the team realizes that they’re less important.

We spend a lot of time sitting with customers. We have a customer advisory board called the Technology Adoption Program, and they come out here for a couple of days. These are big technology decision makers at government national labs, biotech companies, and financial institutions like banks and stock trading houses. We spend two days presenting to them on what we’re planning to build, and they give us feedback about what they’d like to see.

We release public betas. In the Technology Adoption Program, people actually deploy beta code and run it in production prior to the release of our software. That lets us know which features are being used and which ones aren’t.

I also spend a lot of time with people asking them which things they think are dumb or badly implemented, with an eye toward cutting those and focusing on other stuff instead.

The overall process takes the entire universe of things that you could possibly build, and you continually cut, as you move through the software development process, until you have what is really important at the end.

Scott: Again, this varies from open source project to open source project, but a lot of them are built by developers for developers, so the way a feature gets into an open source product is that somebody just sits down and writes it. Their first interaction with the process is by submitting code for a feature.

If they can’t code it, then that feature isn’t going to get included, so the downside is that you only have coders to some degree, spec’ing features. The upside is that nobody writes something that they aren’t personally going to use.

At a company like Microsoft, when you’re building the next version of a product, you’re doing a certain amount of work that’s fairly speculative. You’re building what you think people are going to need — you’re really building what you think people are going to use, and sometimes that’s more successful than others.

I mean, sometimes Microsoft delivers stuff that’s wildly popular and everybody uses it, and some things are dead on arrival, even though there was a lot of effort put into building them.

Ryan: Right — do you remember the “Bob” operating system? It’s not a successful product.

[laughter]

Commercial software companies have to determine what people want, and what will be successful. That’s why we spend a lot time doing things like customer research, usability research — really trying to figure out what we should build. I talk a lot about market opportunity documents, which is a fancy title for “what do people want?”

It tells us first of all, what set of users we are going to help. We segment the market, and we figure out where we’re going to prioritize. That’s the first round of cuts — we’re not building something for everybody; we’re building something for a particular group.

Then we ask, “What do those people really want?” and we visit customers to try to answer that. One of the first questions I asked is an open-ended one, which is, “What drives you most crazy about your HPC System?” Windows or Linux — it doesn’t matter — what’s the thing that drives you most nuts?

If you have some really open-ended questions, you hear some really interesting things come back from customers. Remember that HPC was entirely a UNIX market in the past. A lot has moved to Linux, but there’s still a fair amount of Unix out there, and some of them are beginning to adopt Windows.

And because so much of it is driven by the open source people, you have a whole set of computer scientists really driving at issues that they think are important in high performance computing.

But when you go talk to the system administrators, they tell you things like, “I’m struggling with power and cooling — the systems just run too hot.” The second problem that they talk about is, “It’s so hard to get the cluster up and running in the first place.”

Sean Campbell: Do you think that either closed source or open source software reaches out a little more effectively to some of those audiences and meets their needs?

Ryan: Take systems administration software for example. How do I get servers up and running, and deployed, and get the whole system going? We do really well with that, and it’s a good thing to do well at since a lot of people complain about it.

This is where things get a little bit touchy, because there are a lot of UNIX administrators who are very passionate about UNIX or Linux. And Windows administrators like to talk about how great Windows is. People throw mud all the time about this. I’m on a bunch of user group mailing lists, and people there say, “Well, the problem with Windows administrators is that they’re inexpensive, and they’re all the same.”

On the other hand, I have heard people say, “Every time we get a call from a Unix customer, or a Linux customer, their whole network is set up in a unique way. Every time I get a call from one of our Windows customers they’re easier to support because all Windows networks are traditionally set up the same.” It’s true that there are better prescriptions of how to do things in Windows than there are in the Linux world, or the Unix world, and so that has some benefit.

I think we do a good job of servicing the needs of the system administrators. Certainly in the database space there are a lot of debates about it. I think MySQL has actually come a long way, and is doing pretty well. I’m not a big database expert, but SQL Server really changed the database market.

I don’t know if you guys have ever set up a Oracle Database before, but it’s hard [laughs]. The first one I set up was about 10 years ago on a Solaris box, and oh my God. I consider myself to be a pretty technically competent guy. I’m not a database expert, but I spent a weekend just trying to get through the ten volumes of manuals about how to get the thing up and running and performing correctly.

With SQL Server, you walk through a wizard for setup, and you’re pretty much up and running. So, I think we do really well at servicing particular user needs, and servicing a broad general market as well, even though it may not be as technically savvy as people that are making contributions into the open source community.

Scott: Another area that I’d like to dig into involves some of the differences around identification of bugs, and support, and so forth. I’d guess that when you’re dealing with high performance computing and different architectures, you see all kinds of strange things that might be hard to reproduce.

On the closed source side, how do you handle that? What is the process for a customer to come to you with a problem and get a resolution, whether it’s just sending knowledge back to the customer, or whether it’s an actual change to the product to resolve it?

Ryan: One of the things that’s important for my group to keep in mind is that we’re brand new in this space. Windows is eight to ten percent of the HPC market right now, which is really pretty good. But if somebody has an HPC cluster, it’s probably a Linux or Unix cluster, so what has been important for us is that we understand the way that people are using their existing clusters, and that we fit into the model.

We provide support through our traditional Microsoft product support services, and we also have newsgroups that we support. But we also behave a lot more like more open source organizations might behave.

We have a site called WindowsHPC.net, and there are a set of bulletin boards up there that people can search to look for problems that they run into. Our development team here in Redmond reads those groups all the time.

That helps us plug in to that type of traditional model for how people support these systems. And that’s how people can report bugs as well. We also get bug reports from a lot of our partners. In the HPC market, it’s not like building a packaged product like Office, where you kind of ship it out there and you’re done.

I have to get OEMs like Dell and HP and IBM and SGI all lined up to ship our software. We have network hardware vendors, software vendors, critical applications like Mathematica and MatLab, that are all integrated in with our products. We have to get all of that put together.

So, we get bugs from those customers or partners as well. And those come, usually, directly in to us. They talk to somebody that’s kind of an account representative for them at Microsoft, and we get bugs filed directly. We get on the phone with them, and partners get a really close level of support in that way. When end users run into problems, they post to the newsgroups, and we help them out there.

So, the interesting thing here is that we do both. We do stuff that’s a lot more like the open source community as far as finding out about bugs as well as stuff that is more traditional.

The bug reporting service built into Windows gives us really great, qualitative data about where things crash, like the fact that most crashes in Windows occurred because of video device drivers. I forget the exact numbers, but something like 50 percent of crashes in Windows used to be because of video device drivers.

And then we have something called WHQL, Windows Hardware Qualification Lab, and they run tests on drivers. In order for a driver to get WHQL certification, it has to pass a battery of tests. So, we would look at the ways these different video drivers were crashing, and create tests that we would then put into WHQL, and that would improve the quality of those drivers continuously. Because even if it’s a particular driver that crashes, you still blame Microsoft, because Windows crashed. And so, it behooves us to really go through making sure that all of that code is performing correctly.

Scott: In other words, you’re able to regulate the ecosystem to some degree by saying, “If you want to be certified for Windows, submit your driver to our lab.” And then if your lab is running tests and they’re getting the driver to crash, you go back to the vendor and tell them to fix it?

Ryan: Actually, the qualification labs are run by a separate agency. They can contact us to get a waiver for a particular company, and under some circumstances that’s worth doing, to grant a waiver if they’re not able to pass a particular test.

It could also be, as hardware continually evolves, that the tests may not always keep up. If a hardware vendor implements a totally new piece of hardware, the older tests may not work correctly, so you have to be flexible, of course. But everyone wants to go through that qualification process, because the OEMs — like Dell and HP and IBM, etcetera — all want WHQL-qualified drivers on their systems. Otherwise, they’re going to be getting support calls as well.

Scott: One of the things that Microsoft’s been really focused on for about the last five years is security — just increasing the security of the stuff that you guys ship. Is any of that included in your certification process?

Ryan: There are parts. We’ve been releasing a lot of our security tools to the community, for people to use the same security tools that we use in their own products as they build their own software. We have a lot of very sophisticated software analysis tools that help identify security bugs before they even pop up. We published a book called “Writing Secure Code.”

Scott: We interviewed Michael Howard, so we’ve got a pretty good background on that.

Ryan: Actually, Michael and I have worked really closely. As he was doing the first round of the security development life-cycle, I was responsible for security on mobile devices like Pocket PC and Smartphone.

I would say that we do really well on the security end of things. And if you talked to Michael, you know that the issue is that if you miss one thing, there’s some hacker out there that has years to try and find that one bug that somebody missed.

Scott: He also pointed out that as Microsoft hardens their products, hackers are focusing more on the ecosystem because they’re softer targets.

Ryan: I’ve read that same data and I think that’s fascinating, the way that these kinds of ecosystems change over time.

Scott: So, the third party that does the certification runs tests, but I’m guessing it hasn’t evolved to the point where they say to a driver vendor, “Well, submit your threat models.” Do you see things evolving in that direction, or is that more intrusive into how an organization does what it does than Microsoft would want to get?

Ryan: I would say it leans toward your second point, that when you hand over your threat models, you’re basically handing over the architecture of your product. And I’m not sure that a lot of people want to do that.

Scott: One of the things that Michael also pointed out is that certain APIs have just been banned, things as simple as strcpy(), because they’re vulnerable to buffer overruns.

It seems like you guys would be in a difficult position because high-performance computing, on one hand, is all about speed, but I’m guessing that there’s a security aspect to it, too. So, how do you balance security and performance, or do you see that there’s any tension between those at all?

Ryan: I don’t think it’s black and white. I think there’s a little bit of tension, but not a lot. Here’s what people traditionally do with HPC systems: you have a cluster — let’s say you have 500 machines — and you have what’s called a head node. And a head node is where you submit computational jobs.

That head node will be sitting on the public network, and when you submit your job and your job is ready to run against all the compute nodes in the cluster, those compute nodes are sitting behind a firewall on a private network.

In a sense, all of those compute nodes can have a very open relationship with each other. They may not be hardened in the same way that every other system on the network is. And that’s OK, because they’re all behind a firewall. And all the jobs, when they’re submitted, are validated. You know what user submitted them, and they have permission to submit jobs. And if they don’t have permission, the job is rejected, and so forth. Those jobs are submitted into that cluster and they run on the cluster at that point.

There’s a pretty well-established security model for how clusters run computational jobs already. In the military installations that we’ve talked to, they actually hook them up in separate rooms. They’re not plugged into any kind of public network at all.

Scott: I would assume that just for performance reasons if nothing else, you don’t want your cluster sharing a network, and it’s also probably on a much higher speed network than the rest of your organization. I would assume that it’s pretty isolated.

Ryan: Here’s where it gets really interesting. The thing that my group is really pushing for is moving high-performance computing into more mainstream areas. We don’t focus on share-shift of moving people from Linux to Windows in the existing market; our job is really to grow the HPC market. And we expect that there’s going to be a little bit of share-shift, and that’s great for us, but we’re really focused on growing that market in new spaces.

One of the areas that’s very interesting is departmental and workgroup clusters. Those people may set up the whole cluster on a public network. In those systems we need to make sure that that system is secure by default when it comes up.

Scott: Do you ever see a day where every computer in the organization is potentially also part of a high performance cluster? Or do you think there will always be dedicated machines that are separate from machines running the user-workloads?

Ryan: Are you talking about a situation where my computer in front of me would actually be plugged into a cluster on the back-end?

Scott: Right. Kind of like the SETI@home thing, where my machine is idle, so it starts working on this job.

Ryan: In some cases, I do see that happening. People call it “cycle stealing,” or “workstation clusters.”

Those systems only work well for a particular set of computing problems, though. For certain kinds of computational problems that have a high number of cycles that provide data, those kinds of clusters work great. So SETI@home is a great example; there’s also a protein folding project which works this way. When your screensaver kicks on, you start doing your computation as a part of the screensaver. There’s a whole set of other problems that don’t actually work that well. Seismic processing is a great example of a problem that doesn’t work very well, because they have very low number of cycles of computation per byte of data. I talked to one of the companies that does this kind of work on a 10,000 node cluster, and when they get customer data in from an oil company, it’s a whole pallet of tapes.

The whole thing gets loaded into a gigantic tape array, and then they farm it out across a 10,000-node cluster for computational analysis. They’re dealing with just ridiculous amounts of data, and that stuff wouldn’t work well with cycle stealing. When you really get these systems working in the enterprise phase, you’ve got to figure out how to make sure that data that’s residing in London doesn’t get shuffled off to China for computation, because it’s so inefficient to move it all the way around the world. You may actually spend more time moving data than you do computing it.

Scott: So, in other words, ‘high volume of data, low computation per byte’ is bad for that cycle stealing scenario. ‘Low volume of data, high computation per byte’ is good for cycle stealing, because you ship a little bit of data out to a machine, and it just crunches on it for quite some time and submits, again, a fairly small result set back.

Ryan: Yes. Here’s what I think is a more interesting direction, in terms of your question about what’s happening with my PC and my cluster in the background. We’ve hit a point where CPUs aren’t going faster any longer. 4GHz is pretty fast, and the amount of heat generated by a processor running at 4GHz is enormous. The reason we can’t go faster is they just get too hot.

So the silicon vendors have started moving in a multi-core direction, and so we see four cores right now, we’re going to see eight cores, and pretty soon a bazillion cores are going to be on these machines.

And this is an issue for the software industry as well as for Microsoft. What’s been so great about the software industry is, as processors get faster, we can add more and more features, right? The Linux kernel gets bigger, the Windows operating system gets bigger, we have richer features, we can do more sophisticated stuff than ever before. But now that processors have hit kind of their Gigahertz speed limit, we can’t just willy-nilly add features into our products any longer.

The bad thing that could happen is that the software industry would become very similar to the white goods industry. You don’t buy a new washing machine because ooh wow, this one has this great new spin cycle! No, you wait until your washing machine breaks, and then you buy another one.

What has to happen in the software industry is that all software development needs to start moving in the direction of being aware of concurrency.

That concurrency could be across multiple cores on your local machine, or it could be across multiple machines running in a cluster. And so, the thing that I predict we’re going to see in the future, is that programming models and the kind of programs that are being written will not only be better at taking advantage of multiple cores, but those programs will be better able to run on a cluster as well.

Scott: I’m sure that this is an area of just absolutely intense research, but right now, it’s just too hard for the average developer to write a good parallel program.

It’s too hard to write a multi-threaded program; it’s too hard to write a program that’s going to run on a cluster. Event driven programming became easy when it was just something the languages natively knew about. And right now, the languages don’t really natively know about threads, and concurrency, and all that stuff. It’s done through libraries; it’s not really done through core concepts in the language. You have to write the thread synchronization logic, and you can’t really just declare a synchronized integer, for example.

What do you see as the future of the tools, so that it’s not so insanely difficult to do it right? Do you see the way it’s done, with high performance computing, where you’re sending off batches of work and getting results back, do you see that as being a viable model for a multi-core system? Or do you think there’s some new paradigm on the horizon that might unify both?

Ryan: For general purpose programming, it’s the latter. We start moving to new programming models — not radically new programming models, because we can’t retrain everybody in totally new programming constructs, but you’ll see subtle shifts. Languages like C# will continue to evolve and understand better concurrency, and it will become simple for developers.

I think we have to get there, because I agree with you, trying to teach average software developers about how to do concurrent programming just isn’t going to work, as it stands right now. And if you think that’s hard, you should look at the MPI programming that people write. This is akin to Assembly level programming — it’s just outrageously complicated.

You get this 500-node cluster, and the communications are not handled automatically. One node switches to receive mode, the other node switches to send mode, and then it sends data to the machine that’s in receive mode. If they both switch to send simultaneously, or they both switch to receive simultaneously, it deadlocks! And your whole 500-node job halts.

Scott: Wow. Getting back on some of the differences between closed source and open source, to follow upon your statement that “you can’t just change everything because it’s hard to train everybody to do things differently,” open source seems to move very incrementally, because of the way it’s built, because things are submitted as 100 lines of code to a mailing list where they get scrutinized, and maybe it makes it in and maybe it doesn’t. And a lot of times, things are worked on in fairly small pieces.

It seems like it’s fairly impossible for an open source project to make a radical leap. There seem to only be two ways that happens: one is that somebody forks the source and does a fairly major change to it, or a whole new thing comes out of nowhere; you know, Ruby on Rails shows up out of nowhere, and it’s all the rage for building web stuff. People look at it, and they have to decide whether to stick with PHP or to learn a whole new thing.

In closed source, it seems to be a little different. It seems that there’s a lot higher barrier to just releasing something completely new that has no connection to an existing product. There’s a lower barrier to taking an existing product and maybe doing something a little bit more radical with it. Like in Office, they came out with the Ribbon, a whole new UI paradigm. I don’t know that something like OpenOffice would have ever evolved something like that.

Vista is obviously built on the same code base as previous operating systems, but there were some pretty radical departures there that had some impacts on application compatibility and issues like that. And again, it would be kind of a stretch for open source to do that. But on the other hand, you might come out with a whole new shell that competes with Gnome or KDE or their peers.

Talk about that a little bit. Do you think I’m onto something, or do you see things that run counter to my observation?

Ryan: I think it’s a very good point, and it’s something that I’ve also thought about and wondered about. Part of what’s powerful about the open source community is that it is incremental. And as they make their incremental changes, if they’re good, people use them; if they don’t, nobody uses them, and they eventually are discarded. A good example is user interface models. We have one, Apple has one, and those are very popular user interface models.

The X-Windows model was never really that popular. It’s popular among a particular group of people, but not like the Macintosh and Windows paradigms were. And they’ve continually competed and evolved with each other. And I think you see Gnome and KDE coming in and filing the best of what they like out of those models and evolving that, and maybe keeping a little bit of some of the X-Windows stuff that they like as well.

Those are highly evolutionary systems. Things like OpenOffice can really pick at what has been successful–like Office, for example–and take that and put it into their products. At the same time — and I think this will be true for a lot of commercial software companies — because we spend so much time trying to figure out what people would pay for, if we think something is going to be paid for, we’ll invest the money there.

Scott: It seems like there’s almost a pressure not to do things that are incremental, because unless it’s a big enough delta between this version and the previous version, people won’t pay for it.

Ryan: That’s right. We’re constantly in this business of having to generate value on a huge scale, because it’s not like we can build something that’s interesting for a hundred or a thousand people. It has to be interesting for millions of people. And so that’s why we can spend so much time figuring out the next big shift that’s going to come along.

Sometimes that means that we look at stuff that’s been popular in niche markets and figure out how to make that go really big. And sometimes it’s radical new things. In some ways, for example, you can say that the original Windows NT operating system was an evolution of traditional systems, based on work that was done at DEC, and work that was done at some other software companies, to create this new kind of kernel.

But NT was a fascinating concept. If we go back and think about it, the idea was that you could write a kernel that would span from the smallest laptop computer up to gigantic SMP systems.

Scott: The whole Hardware Abstraction Layer was an unproven theory and a pretty radical idea.

Ryan: Right, and it’s now the foundation for Vista.

You know, HPC is a really interesting market to study from an open source point of view, because it’s a full ecosystem, and it’s still pretty heavily driven by the academic community, which is, by nature, fairly open source-oriented.

We fund work at something like 10 universities and research labs around the world, like Jiaotong University in China, and Pacific Northwest National Labs, and the University of Tennessee where Jack Dongarra does all of his HPC research. And we do that because it’s still critical to really be working well with the academic community, in order to be successful in this market.

Scott: They’re the ones running the climate simulations, things like that. They’re the ones who, largely, are running the kind of models that need high-performance computing. One of the things that’s kind of interesting — it’s not really high-performance computing — but Amazon has in beta what they call their Enterprise Compute Cluster.

Ryan: Yeah. My friend Marvin Theimer is an architect at Amazon.

Scott: Do you see a future for hosted high-performance computing? Do you see a future for maybe ‘high-performance computing lite,’ where it is more tailored to the kind of jobs where you’re not moving enormous quantities of data, the computation per byte is higher, and you can have simplified communication between the nodes?

I wouldn’t have to have 500 nodes, but I could push a job up to some hosted environment that could put 10 nodes or 10,000 nodes on it. Does that kind of thing exist today?

Ryan: It does exist today. Sun has been doing this for a few years now; it’s at least two years. They have this model where they charge on a CPU-hour basis. We talk to those guys, and they’re very open about what they’ve been up to, or at least they used to be. And the way that those jobs work is you actually ship all the disks over there.

Jim Gray did some really interesting research here at Microsoft. He computed how much it costs to move a certain amount of data and how long it would take to move that much data over a particular kind of pipe, because the question is: how long does it take you to get a terabyte of data from one site to another?

Scott: DVDs and FedEx have an incredibly high bandwidth.

Ryan: Yeah. That was the result of the research, years ago: “Just put it on DVDs and ship it that way.” And so what Sun does is that you actually ship them a cabinet of hard drives, and they plug that into the cluster and use it to compute.

Your point is very good though, that a lot of customers will typically do small computational jobs and occasionally they’ll need a really big cluster. I think that is true going forward, that we’ll see more and more of that, and people will occasionally need to run a big job.

If part of what we believe is true, that this revolution will happen in workgroup and departmental clusters, it’s because workgroups and departments will need those smaller clusters for some of their work, but every now and then they’re going to want to hand jobs to a huge cluster for computation.

And that may be handled using grid standards that are generated by the open grid forum. That may be handled by just handing off to a place like Amazon or Sun.

I also think that, in closing on the discussion of open source and the HPC market and the commercial software that we’re building for HPC, I’m really hoping that what’s going to happen is that our group at Microsoft will help pave the way for how commercial groups at Microsoft are better able to work with open source projects.

Scott: Thanks, Ryan, for all your insight. This was a great conversation.

Ryan: Thanks.

Bookmark this: