• Deciding on the open source approach to development and choosing a license
• Simplifying the creation of multi-tiered applications for the cloud
• Fitting WaveMaker into the broader open source and proprietary ecosystem
• Engaging with the community and deciding what goes into the paid edition
• On the importance of community managers, and how to choose a good one

Sean Campbell: To start, why don’t you take a minute and introduce yourself and your company?

Chris Keene: I’m the CEO of WaveMaker. Our company makes an open source cloud development platform, which you can think of as being like force.com without the proprietary lock in.

We introduced WaveMaker about two years ago as an open source Apache project, and there’s also a commercial version of that same product that has additional features, around security primarily.

At the end of last year, we moved that development platform into the cloud and really started to realize the full vision, similar to PowerBuilder or Microsoft Access for the cloud.

Sean: Tell us a little bit about the open source project and the choice to develop using community involvement.

Chris: I should say first that my last company, Persistence Software, was a proprietary company, and we took that public and then sold it, so I kind of know the drill on the proprietary side. When I came to the open source world, I found that everything I thought I knew was wrong. And so the main way we’ve figured out how to do things in open source has been by doing them wrong first and then fixing our mistakes as we went along.

I think the upside is that if you’re willing to fix your mistakes, open source has an incredible adoption curve. In a little over a year, we gotten more than 15,000 registered developers in our community. For comparison, it took the SpringSource community over six years to get roughly three times as many developers.

We’ve had hundreds of thousands of downloads, but what we really care about is how many people have downloaded, installed, come back to sign up for our community, and then started to participate. To build that kind of a user base in a little over a year is just incredible.

Still, along the way, we made a lot of mistakes. We started out, for example, with an AGPL license, and we found that the open source community didn’t really trust that license for software that was going to be embedded into their own applications, which caused us to switch over to the Apache license.

We’ve ended up tweaking and changing almost every element of our business model, because we’re constantly getting feedback from our community about what they’d like to see and what’s most valuable for them.

Sean: With the Apache license, people can embed it in proprietary software. So the GPL license makes things compatible with bundling with a Linux distro, for example, but the Apache license makes it possible to carry open source in a proprietary product without having to make the whole thing open source. Is that the motivation?

Chris: That’s exactly right. Almost by definition, anything that somebody builds with a cloud development platform is going to embed, at a minimum, your libraries. Because of that, people didn’t feel comfortable under the AGPL license.

Part of the reason for the difference in the community’s reaction is that AGPL is fairly new, whereas Apache has been around for a while, and people really understand and trust it. I think if you are an open source company, you need to be willing to follow the lead of your own community; otherwise your community is basically just a marketing exercise.

Sean: That makes sense, and in general, open source communities vote with their feet fairly rapidly. Generally, they have a certain number of choices in any given space, and it’s instructive to consider, for example, that MySQL and Apache took off even though they obviously weren’t the only options available.

It seems like there is a sense of getting things just right in order to have one project versus another one become a center of gravity that attracts a lot of people.

Chris: I think that’s true, even if a feature-by-feature comparison might suggest that it’s a toss-up. One thing I will say for MySQL, though, is that they absolutely got the install right, and it’s still much easier, in my opinion, to install MySQL than many other of the open source databases.

I think that ease of use and approachability really help drive rapid adoption. That’s a key characteristic of MySQL, Ruby on Rails, and frankly of WaveMaker as well. We think that it really helps drive success when people don’t have to invest months of their lives to figure out whether your stuff works or not.

Sean: Before the cloud, people were using hosting providers to build websites. Hosting providers started to get into virtual private servers, so instead of having a dedicated server that was co-located, they could spin up a virtual instance. Amazon took that to a different level, with utility pricing and letting you spin up hundreds of servers with a command line or an API call.

How does WaveMaker fit into that spectrum, from the dedicated, co-located server to the virtual private server with Rackspace or whoever, to running in a cloud where you’ve got options for real elasticity?

Chris: The cloud itself is just a concept from my perspective; it’s really just a data center with an API. If I’ve got an application and I need to figure out where to put it, the question is, “What steps do I need to go through to get my application running someplace where people can get at it with their browsers?”

Of course, in the co-lo world and the hosting world, I have to call people, I have to put contracts in place, I have to pick up the phone and send emails to make something happen. In the cloud space, I just make a set of API calls.

Not only has the cost and complexity of making something available on the Web gone down dramatically, but I can now start to imagine businesses that previously wouldn’t have tried to put things up on the Web wanting to do so.

Cloud computing really lowers the bar for deployment, in terms of both systems administration and economics. But cloud computing on its own doesn’t really do anything to lower the difficulty of creating that cloud application in the first place. That’s really where WaveMaker comes in.

If cloud computing is all about democratization, utility computing, and allowing more people to put applications on the Web, then we also need a new set of development platforms to facilitate that. In fact, we’d argue the need not just for development platforms but actually for application templates that people can take down and customize for their own purposes.

That set of templates has an intent very similar to what SharePoint templates have provided: a simple means of managing projects, time, assets, and issues to allow a whole new class of developers to build and deploy web-based applications.

Sean: A lot of open source development, on core products, involves very low-level C programming, or those kinds of things. Then there’s another level where people are doing Java, .NET, or PHP programming.

Those who are using a higher level language might be using open source libraries, test and development tools, or development environments.

How receptive have you found the open source community to be to the notion of 4GL languages in general? Also, do the people who are actually building stuff with WaveMaker even really know or care that it’s open source, or to them is it just a really productive way to get stuff built?

Chris: I think WaveMaker has pushed the envelope in terms of the type of developers we’re trying to reach out to. The typical stereotype open source developer is an alpha geek who has never gotten past EMACS as their development environment and eats other coders for breakfast.

The reality is that there are very few people, even very sophisticated developers, who have the full range of skills needed to build a web application. For example, a person might need to build a Java application, put an Ajax front end on it, build a database and database schema, and then deploy that into some sort of a hosting environment.

We’ve found that the target user base for WaveMaker consists of developers who may be very skilled but don’t have full teams where they can afford to put four to six developers with different skill sets on a project to get it through.

We tend to work with very sophisticated developers, but oftentimes they’re in the business units. Gartner calls these people “citizen developers” or “non-expert” developers, but it’s really the notion of people trying to get a lot done on their own.

Frankly, I think that’s the same crew that Ruby on Rails has appealed to. One of the interesting things for WaveMaker is that our biggest core set of customers is Macintosh developers, because they really like visual development. They tend to be very sophisticated, and they’re incredibly pragmatic; they just want to get the job done, and they want the best and most productive tool for getting that to happen.

Sean: My understanding is that WaveMaker is that it’s really optimized for the “forms over data” types of applications. In other words, when NBC is trying to figure out how to build its site for streaming the Olympics, WaveMaker is maybe not the best choice. On the other hand, if somebody is building an expense-reporting application, or maybe even a whole CRM system from scratch, that’s more the niche that WaveMaker fits in. Does that ring true?

Chris: Absolutely. WaveMaker really fits into the classic category of enterprise solutions that help business developers build basic business applications. We’re not trying to focus on the multimedia side of the Web, nor are we trying to focus on the complex transaction processing part of core IT.

We’re really about the broad middle: the 80 percent of applications in the business that are building similar functionality over and over again but customizing it slightly, which is where we think that combining a development platform with application templates is an incredibly powerful way to drive productivity broadly throughout the organization.

Sean: So, if it’s running up on Amazon or you’re running it in your own local data center, it’s running on a server that has some kind of an OS, and it’s running on some kind of a Web server. What is the stack under WaveMaker? Is this a Linux/Apache sort of thing? Does it run on Windows?

Chris: The stack underneath WaveMaker is a Spring/Hibernate stack. We made a decision very early on that not only were we going to release our software under open source, but we were also going to eat our own dog food, in the sense that we were going to actually build our stack on top of an open source stack.

That means Spring and Hibernate on the back end, and it means JFON and Dojo on the front end. What’s great about building a platform on that stack is that it’s incredibly flexible. We can run in any Java server, we can work with any database through Hibernate, and with any security mechanism through Asiji. We can to talk to any Web service through JAX WS.

On the front end, we’re able to support accessibility and internationalization requirements through Dojo. So effectively, we’re standing on the shoulders of giants.

It’s very interesting that in the cloud world, and in the cloud development platform world in particular, companies have gone back to inventing their own wheels. For example, with force.com, every single element of that architecture is proprietary.

Not only do they have to invest a tremendous amount of resources to build the same quality of tools, but by definition, they don’t have the breadth of integration points they would get working with an open source stack.

Sean: If you’re dealing with a Java shop, this stuff probably fits right in. For more of a Microsoft .NET shop, it sounds like this would still integrate pretty well, because of the technologies that you’ve talked about. Do you have any examples you can cite about this being used in a heterogeneous environment where interoperability was important?

Chris: Absolutely. WaveMaker as a platform talks to any kind of a Web service, whether it’s SOAP or RESTful or RSS. We have customers who very easily integrate functionality from WaveMaker with functionality from the .NET environment.

That’s a big part of what’s out there, so we have to be able to build on whatever intellectual property the enterprise has already developed, and that includes .NET.

Sean: Can you talk a little more about the corresponding story for shops that do Java, PHP, and that kind of stuff?

Chris: We’ve got a number of examples where companies have been able to effect a real transformation. For example, KANA is a company that produces call center software. They supply call centers to about half of the Fortune 100, and they have effectively rebuilt their entire stack based on DB2, WebSphere, and WaveMaker.

They created an SOA bus where all of the functionality around these very complex multi-channel call centers is wrapped up into a series of web services. WaveMaker calls into those web services and provides the user interface, the ability to customize that user interface, and the ability to make changes to the back-end system.

Sean: If people are using WaveMaker to build their web front-end, how does it scale up and scale out?

Chris: The work we’ve done with KANA has been targeted toward some of the largest enterprises on the planet, and they’re looking at things like 20,000 concurrent users and sub-second response rates, so we feel very confident that our solution scales.

Here, again, there’s a huge benefit from open source. For example, if we found that our back-end wasn’t scaling the way that we wanted it to, there are a lot of Spring developers out there who could help us make it better. We’ve been doing work with KANA to make our Dojo front end perform better, and again, there are many people with expertise in building very high-performance Dojo clients.

So we’re able to leverage not only our own community of 15,000 developers, but the Spring community, the Dojo community, and the Hibernate community to tweak out the performance and basically learn from best practices.

Open source gives us a huge benefit in this area, as opposed to building our own proprietary back-end and then having to go through all the painful learning process of how to get it to perform without the community’s help.

Sean: I’d like to hear more about how you guys engage with the open source community, at a tactical or business level. You have obviously articulated a lot of good, strong benefits for having an open source development model, but sometimes inherent tensions can creep in.

For example, there are issues around having X features in the community edition and Y features in the paid edition. You don’t have a static product—it’s obviously evolving—so how do determine what goes into the full-blown edition versus the community edition? How do you determine the levels of support?

These questions seem to especially get complicated when the community would really like to see something in the community edition but you’re not yet offering it.

Chris: Those are hard questions, and I don’t think you ever get them completely right, even though you need to get them right enough. One thing that probably doesn’t get enough attention is that the open source community is not just developers. There’s also a very interesting level of cooperation and collaboration between open source CEOs.

I’ve had the benefit of being able to spend a lot of time with Marten Mickos of MySQL, Brian Gentile of JasperSoft, Rod Johnson of SpringSource, and a number of other CEOs. Those relationships have convinced me that open source is as much an attitude as it is a business model. If you really embrace the open source approach, what you get is a set of CEOs and companies that are wired to collaborate and cooperate and share best practices and learnings.

We found some rules of thumb for how to separate our open source and enterprise products. I think that the most important one is that the commercial features should have more to do with robustness and scalability of deployment than they should with fundamental things that you can or cannot do within development.

If you look across commercial open source companies, including WaveMaker, SpringSource, MySQL, and others, you’ll find trends in the things that tend to separate the commercial versions from the community versions. They tend to be clustered around management, monitoring, scalability, security, and other things that you really start to be concerned about when you take an application and deploy it into a robust enterprise deployment.

I think the most important thing is that the point of having a community edition versus an enterprise edition is not to make as many people as possible pay for your product. The point, to me, is that you’re trying to make the people who are willing to pay want to.

Therefore, you don’t pick a feature that could trip people up so that they get started with your product and then you somehow catch them because they want to do something else. The process involves sitting down in advance to determine who would never pay for the product anyway. The first step is to figure out a product that will make them happy and make them good marketers and developers for the project.

Next, you need to determine who would be likely to pay for the product, because they’re using commercial security, commercial databases, or commercial Web servers, or because they need higher levels of security and performance. You need to make sure that those people fall into the bucket where, in fact, there is a payment required.

If you think about it that way, it makes sense, and in fact, I would go one step further. You want to engage with that group of people who are never going to pay for your product and encourage them to invest time in your community. That’s the payment that you want from them.

For the group of people who are paying for your product, you want to provide as close to a commercial software experience as possible.

Sean: Is tracking sales and trying to convert those community members over to paid members even a part of your business model?

Chris: Certainly we track sales, but I have to say (and a number of other open source CEOs are starting to feel more this way as well) that people either have money in their pocket when they come into your community or they don’t.

If somebody doesn’t have money in their pocket, you’re not ever going to convert them. It doesn’t matter how compelling your for-pay product is, because they don’t have any money. The more you try to put the screws to them, the more you fritter away the goodwill that you would otherwise have with them.

Similarly, there are people that come into your community with money in their pocket, and if they find that they’re more productive, they’re getting better results, or their boss thinks they’re doing a great job, then they’ll give you some of that money relatively happily.

I think the whole notion is wrong that you should regard every single person in your community as a candidate for giving you their credit card.

Sean: I agree. At the same time, how do you go about encouraging the non-buyers to incent other potential paying customers? I realize there’s an ethos around it, but at the end of the day there are lights to keep on, bills to pay, and things like that.

Chris: From those people, we’d like to get good karma, because they like our stuff and benefit from it. Then we need to find ways in which they can give back that are non-monetary. I’ll admit that we’re not fabulous at this, although certainly the community is one place to do that.

Whenever we help somebody out in the community, we try to back-channel to them to make sure that things went well. If they’re excited, we usually suggest to them that, if they feel like the community helped them out, they should consider helping somebody else out by giving back to the community.

It’s basically karmic marketing. As people in the community get more confident in their skills, just as in any open source community, people get more involved. In addition to contributing code, for example, we’ve had people create communities in various different languages.

We want everybody to pay for the value that they receive, but some of the people pay in money and other people pay with their time. At some level, we’re indifferent to which they choose, because we think the more people who pay with their time, the more people they’re going to influence. Some percentage of the people they influence, as you said, are going to be looking for a commercial application of what we’ve got.

Sean: Because of the size and the acceleration rate you guys have, do you have a community head or liaison who deals with issues like that as their full-time role? If not, do you have some kind of plan to staff up that area?

Chris: I think open source is definitely a business model where you go hard or you go home, so if you’re not willing to pursue all the implications of open source, you really shouldn’t start down the path. And to me, the core of open source is the community, in the sense that if you don’t have a community, you really have nothing.

If I had two only employees in an open-source company, one of those people would be the community manager. That’s how important I think that position is.

In our company, we have a full time person who manages the community, and his background is technical. My view is that if you have a marketing person managing your community, you’ve said something pretty profound to the community, and I think there’s a spectacular mismatch between somebody with a marketing background and the kinds of questions and concerns that the community is needing to get addressed.

As your community grows, you may need to put a layer of management up there, but I think first and foremost, a community is about dialogue, and you need somebody who has the technical depth to be able to answer the questions of the people who come to the community. Otherwise, guess what? They’re not going to come.

So day one, we put a community manager in place. That is absolutely what drove us to 15,000 registered developers in our community in a little over a year. That’s absolutely what got the attention of companies like IBM, Amazon, Rackspace, and all the others we’re working with, and our community is our crown jewel.

Sean: How do you go about finding somebody to be your community evangelist? In a lot of ways, you can’t measure them using standard marketing metrics. They may even be out of the building, virtually or physically, more often than a traditional marketing manager would be. On top of it, by design they’re bringing you feedback that runs, sometimes, upstream of where you want to go, but by definition you have to listen to it.

Your story might be a personal one, for all I know; maybe it’s someone you’ve known for years. But if you could extrapolate, what should other projects look for when they’re about to hire a community manager?

Chris: The first key is that they need whatever technical depth is required to engage with specific questions in their communities. Second, you need somebody with some kind of management skills. Maybe they’re not going to be managing lots of people, but they need to be able to manage their time and their communities.

They need to be able to ask for help when it’s necessary. They need to be able to recognize when somebody’s really angry and be able to handle those situations. At the same time, they need to be able to recognize that somebody is really happy and take advantage of that as well, for instance by taking the situation to marketing to see what they can do with it.

You also need somebody with an incredible customer focus, because day in and day out, they will need to defuse people who have been disappointed in some way. People are bound to use your product for a while and then have it cause problems for them (or give them that perception). Some of those people will inevitably come back to your community yelling and screaming.

I tend to think that the combination of those skills is most likely to be found in a customer-support operation. People who have done technical support for software companies for a while know the job and how to keep their cool, and they tend have a good perspective where they can see trouble brewing, and similarly they can see opportunities.

That, to me, is the ideal. I think you could get that person from a number of different places, but technical customer support is probably the best place to look.

Sean: Well, we’re almost out of time, so are there any closing thoughts from your end?

Chris: I think we’re very early in the trajectory of the open source industry, and it’s been amazing to me to see over just the last two years, the kinds of changes in attitudes and understanding of what it takes to build a real ecosystem.

I think cloud computing is going to take that up another order of magnitude. Ecosystem is the killer app for the cloud, and companies that have learned the lessons of partnering and working with communities are going to be at a huge advantage as the cloud computing revolution takes off.

Sean: That’s a nice place to wrap up. Thanks for taking the time to chat.

Chris: Absolutely. Thank you.

Bookmark this:

• The creation of the Selenium web-testing tool
• The Sauce Labs startup around Selenium to run testing on cloud systems
• Automated web testing with Selenium
• Taking Selenium beyond testing into software documentation and marketing
• The revenue and business models of Sauce Labs
• The maturation of Selenium beyond predecessors and into the emerging cloud ecosystem

Scott Swigart: To start us off, could you introduce yourself and your company?

Jason Huggins:I’m the founder of Sauce Labs. I think my unofficial title is CTO, but the official one on my business card is “Executive Software Chef.” I’m a fan of silly titles, so that’s my silly title. I’m also the creator of the Selenium web-testing tool, and Sauce Labs is built around the things that we wanted to do, as a startup, with Selenium.

A lot of interesting things happened in mid-2004: Django and Rails came out. And a little bit later in 2005, Jesse James Garrett coined the word AJAX; he looked at what Google was doing with Gmail and Maps, and how they were using JavaScript to create a tremendously dynamic website.

In what seems like overnight, JavaScript as a technology to use in web-app development went from being taboo to becoming something that everyone had to do. Right around that time, I was also coming to the conclusion that JavaScript was cool and useful, independently of the AJAX hype wave.

I was the tech lead of a team at ThoughtWorks that was writing an in-house time-expense system. Unlike most corporate entities, we couldn’t dictate by fiat that everyone should use the same browser. We had a bunch of alpha geeks there, and people used whatever they wanted to.

The classic thing you were supposed to do in web applications at the time was to have everything done on the server and present a kind of static HTML page to the client. This JavaScript stuff made the site more lively and more fun, though, so we decided to use that. The problem was that, even as JavaScript was getting better and Google was showing ways to help it work across different browsers, there were still some rough edges there.

As the project continued, the same series of events would keep happening. We would have a feature implemented in JavaScript that would be working in IE but broken in Mozilla. We would file a bug, and our developer would fix it in Mozilla, but forget to test it in IE. So the next week, the bug was still there, but now it would be broken in IE, instead.

That was frustrating, and I decided that we needed to put a stop to it. We decided to come up with a way to write one test and run it across all of the browsers we cared about. That was the genesis of the Selenium tool.

Scott: How did the tool progress from that point to become an open source project?

Jason: I thought I was going to be famous with this time-expenses thing we were building. It turned out, though, that as I was showing people what we had done to test these two browsers, they said they wanted to use it on their own projects.

Those developers wanted access to the code, so we made a formal request to open-source it. One thing led to another, and it kind of took off. We never actually open-sourced the time-expense system, but Selenium definitely took off.

Scott: What was the trajectory from that point? Did the project grow primarily by word of mouth?

Jason: It grew by word of mouth, one project at a time. At one of the last projects I worked on at ThoughtWorks, the developers were frustrated that running tests with Selenium was taking too long–about 30 minutes to run all tests end-to-end. In a classic agile project, you want to run everything often–ideally, at least every time you check a file in and everything done in 10 minutes or less. If the whole process of build and test takes too long, you’ll skip running those tests.

Every project has a different conception of how long is too long, but usually five to 10 minutes is kind of the ideal. On this particular project, total test time was hovering at about 25 minutes. Mutiny was afoot, and I had the idea of getting a couple of extra machines. Using four machines, I wrote some code to split the tests across the four machines, running concurrently, and we got the testing time back to a good level.

A couple of months later, I heard through the grapevine that Google was also using Selenium, and that same contact asked me if I had an updated resume. Of course, that’s the offer you can’t refuse, to be courted by Google, so I applied. It turns out they were building a Selenium farm, and I arrived on the scene right when they were doing that.

It turns out that the Writely team, which is now Google Docs, had the same kind of an epiphany, if you will. There were also using Selenium at the time, and it was taking an hour to run their test. They ran the testing on four machines, and the time went down to 15 minutes.

They started telling other groups at Google, and it became a virtuous cycle where more teams started using this farm, which enabled them to get approval to buy more machines. Having more machines let them divide all of their tests across those machines and make everything go faster. Selenium eventually started being used for Google Apps and Gmail, Maps, Calendar, and lots of apps that I can’t talk about.

Scott: How did Sauce Labs grow out of that experience?

Jason: I was at Google for about a year and a half, but I was looking to get back to Chicago. Meanwhile, I met John Dunham, who is now Sauce Labs CEO, through Elisabeth Hendrickson, an expert in agile testing, and a mutual friend of ours. John wanted to get back into the testing space, and he sought me out just to have lunch with a guy interested in the same field.

I revealed to him I might be looking to do a start up, too. I was the idea guy, but I didn’t have all the business sense to be able to actually run a company. The initial “cocktail napkin” business plan was to prove the viability of throwing hardware at the software testing problem.

We had proved it on a small scale with my little project at ThoughtWorks and also on a big scale at Google. The next question was to determine whether there was an actual market for it, and we started Sauce Labs to answer that question.

It didn’t hurt that cloud computing was the most hyped story of 2009. The type of testing we’re doing at Sauce is the perfect use of cloud technology. We use disposable machines to run a whole bunch of stuff in parallel and run web browsers on all those cloud machines. Customers then drive those machines through our API to execute their tests.

It’s a classic problem where a long, slow process needs to go faster, and we just need a lot of disposable machines to run it against. Sauce Labs wouldn’t be really feasible without cloud technology.

Scott: Do I understand correctly that Selenium provides a plug-in that you use with your browser of choice that lets you record what you do? My understanding is that it emits a script that can then drive the other browsers through those same actions.

It can actually video record what’s on the screen as it’s driving them through those tasks, and it’s got logic to verify that the right thing shows up on the screen. Do I have that right?

Jason: There are several tools in the Selenium toolbox, but the tool you’re thinking of is a Firefox extension called “Selenium IDE”. The video recording is not included in Selenium out-of-the box–that’s something Sauce Labs has added to our online-based service to extend the capability and usefulness of running Selenium tests in the cloud.

Taking a step back, though, Selenium specifically is a functional testing tool (also called “acceptance testing”) that can verify features by interacting with a website exactly as a user would. Think of Selenium as a programmable software robot that knows how to use a web browser, and does so in all browsers, not just in IE or Firefox. We use IE, Firefox, Safari, Opera, and Google Chrome, on Windows, Linux, and Mac OS. We have APIs for Java, C#, Python, Ruby, Perl, and PHP.

The core minimum Selenium API is a collection of simple commands: “open”, “type”, “click”, and maybe “verify”. Open a web page, type text into a field, click a couple of buttons, and then ask the page if some text or another element is present or whether some state has changed. That’s the minimum essence of what Selenium does.

Scott: My sense, too, is that it’s not just simulating the browser. It looks like one of the reasons it seems suited to the cloud is that it’s basically starting up a VM with specific OS and browser builds, driving it through certain things, collecting screen shots, video, and whatever, and then tearing it down.

Jason: That’s actually what makes us unique as a cloud vendor. Everyone else is putting servers in the clouds, but we’re actually booting up the desktop environments and are running what you would otherwise call “client applications” in the cloud on those servers.

One of the more cryptic tag lines we had was, “Web browsers as a service.” That probably doesn’t play in Peoria, but it kind of gives you a sense of what we’re doing. We’re launching these desktops that you can connect to remotely and see that they are starting up IE or Firefox. It really is starting up the browser and you get to see everything a user would see.

Since you mentioned video, I will add that it’s been really important to me to add that functionality, to see directly what happens when a browser running in a cloud crashes, for example.

It was important to me in creating Selenium, since I am very visually oriented, to see what the browser was doing. I want to see whether it looks good or ugly, and I don’t really believe the application’s working until it actually runs in IE and Firefox. And by default, when you run things in the cloud, you can’t see what’s going on, so we put in the effort to add back that visual feedback when customers use our service. Video recording of every test has been a pretty popular feature with our users. They don’t just have some log files to look at. They can actually view the video, share it with their coworkers, and so on.

Scott: I would imagine that as the web becomes more interactive, testing gets an different or expanded focus, as well. For example, with a traditional shopping cart app, you need to make sure data is handled the right way, and that at the end, the “your purchase has been completed” string shows up on the final page.

On the other hand, to test the functionality of dragging a map or something like that, there’s not a string to test for. In some of these cases, the only way you’re actually going to know if it really worked is to look and see whether the map been dragged to the right position or whatever.

That need for video suggests that you are always stuck with a certain number of tests that require kind of some human subjectivity, even if you can automate it to some degree.

Jason: That’s true, although one of the “maybe someday,” crazy ideas I have is to add in some kind of “Mechanical Turk” functionality. For instance, once the video is created, perhaps you can automate answering the question of whether the functionality worked, even if it takes a human to answer the follow-up question of whether it looked ugly.

In other words, even if there are still questions that only a human could answer, it would be easier for them to do that by watching a movie and answering a couple of multiple-choice questions, instead of requiring them to actually perform the entire workflow manually themselves.

I read a blog post recently by Hugh McCleod, where the author suggested that, if you’re doing a start-up for any kind of company, there needs to be some kind of social object: some conversational aspect of what you do or produce.

For us, that’s this video that we can pass around, which can include assertions about whether it is an example of a passing test or failing test. If the test failed, it becomes a social thing that you can pass around to the other developers to say, “Look at the one-minute mark; you can see where the bug appears and the test fails.”

Ideally, you have a test for every feature in a website. That might be sending a message in Gmail, doing a product search on eBay, or buying something on Amazon.

Scott: It seems like that could have implications beyond app testing. What kinds of things have you considered elsewhere in the life cycle?

Jason: I would like to see some of the lines erased between documentation, marketing, and functional testing. My best example is an iPhone commercial, and even Android now these days. All they do is zoom in on the phone, show you opening an app, clicking buttons, seeing how cool it is.

They just show one feature, and how well it’s working, which is actually an ideal use case for Selenium and Sauce Labs. People write these tests so they show off one little feature, then there’s a video for it. Ideally, if you are a software developer, and you’re coding up a project, you add more features over time to the thing you are building.

You could have this continually growing little check-list of the features that you have written, and you can drill down and look at the recorded video of that feature working on latest version of your code.

You can see the history of that feature working, but specifically you can actually see it working over the life of the project. People are doing screencasts now to help sell their software, putting up little YouTube videos on their site.

The old approach was to let people try before they buy, downloading a trial before they bought it. With the unfocused attention of most people these days, now they want to watch the video before they even try it.

I’d like to make it so these videos get recorded all the time with all of your little features. You can use them either as-is or as your prototype for making little feature screencasts you can put up on your website.

Scott: It also potentially solves the problem of keeping the docs in sync, especially with software as a service solutions that might roll out little changes daily, weekly, or every couple of weeks.

Keeping the videos in sync could become fairly automatic, since if it’s been tested it’s also documented, and the only stuff that’s undocumented is the stuff that’s untested.

Jason: Actually, I’d like to see testing move away from being so focused on the tests themselves, and toward looking at the tests as more like screenplays, with the real focus being on the videos they produce. Those would be the actual artifacts that you care about.

As recently as five years ago, project testing was usually kind of a late process activity, done by one or two people in the corner. You called on testers and they used really expensive commercial tools, in isolation from everybody else.

Testing with Selenium follows the growing adoption of agile methodolgies where testing is something that developers do all the time. By analogy, I see a lot of those marketing screencasts being done very late in the process, and not by the developers. Screencasting today is where testing was five years ago.

I believe screencasting could benefit greatly by being pulled into the development process using the video capture functionality from Sauce Labs or similar tools. I think that the key glue there is automation, as well as encouraging testers and developers to think of themselves as screenwriters showing off their applications, rather than being so focused on acquiring the pass or fail data point.

Scott: There’s Selenium, which is an open source project, and then as you said, wrapped around it is Sauce Labs. Talk a bit about what Sauce Labs does. What do you add beyond the open source project? What’s your revenue model?

Jason: We started with the single focus of Selenium and cloud computing. As a request comes in, we either route that request to an existing running machine, or we boot more machines from a cloud vendor. There’s a lot of code needed that’s not in Selenium to manage this cloud environment and scale the infrastructure.

That’s the bulk of what we do, but we also collect log files, record video of each test, and make all that data accessible through our website. We also add support for this notion of multi-tenancy, in the sense that we have to make it work with lots of customers, so we have to set up and tear down cleanly between customer uses. That’s more code we have to write and maintain.

Along the lines of those security considerations, that’s how I found my other co-founder, Steve Hazel. He created a site called codepad.org, and you actually can paste in code from pretty much any language, and it’ll run it on his servers in EC2.

In addition to the actual security code itself, and managing how to properly sandbox running code in the cloud, Steve also provides that mindset of being security conscious and vigilant about how people can use and abuse your site.

Along those lines, we’ve expanded our mindset beyond just providing cloud execution to become the Selenium support company more generally. Because Selenium’s already been out there for five and a half years, a lot of people who have a pretty big installed base may not want to use our cloud service, but they do want a phone number or an email address where they can get support, bug fixes, updates for the latest browsers, and things like that.

We’ve expanded the scope of the company to a classic open source model of being the obvious company that you go to for help when you use this particular tool. Blended with that, we also buy cloud resources at wholesale rates by the hour and sell them by the minute.

Scott: I think you mentioned that the cloud provider behind this is Amazon EC2. It would seem that there are going to be a lot more choices as time goes on. I could envision you using multiple cloud providers simultaneously, transparently to the user, because it’s at a lower abstraction level where the VM is actually running, versus the data that the user’s getting back.

Jason: Right. There’s also an argument to be made that, because we’re telling browsers to go to some website, ideally those browsers should be as close as possible to the active servers that we’re testing.

It’s a simple matter of the speed of light. If the server we’re hitting is in Japan, and our test lab is in Virginia, latency is going to be a problem. Therefore, we’d like to be wherever our customers are, in the sense that if they’re inclined to host their application on EC2, our browsers will be there, too, so the network obviously is really fast.

There are lots of good reasons to be cloud-vendor neutral.

Scott: Obviously, if you’re in the same geography as somebody that helps, but if you’re in the same data center, that’s even better.

Jason: If nothing else, your bandwidth is free when it’s inside the data center.

Scott: On another topic, in getting ready to chat with you, we went out and did a little bit of homework and came across Adobe BrowserLab, which seems to provide some services that are similar to Selenium. Can you talk a little bit about some of things that are unique about Selenium?

Jason: I’m glad you asked that, because it’s where I think the original synapses fired and I was like, “Hey, maybe there’s something here with Selenium as a service.” The word “cloud” didn’t exist a couple of years ago, but the phrase “as a service” was in fairly broad usage.

What was out there was very similar to that Adobe site. There was something called BrowserCam, which sounds shady, but it’s not.

[laughter]

Jason: Basically, BrowserCam’s a screen-shotting service, and there are a bunch of others, like BrowserShots and Litmus.

The limitation there was that all of them were what I might characterize as just single-page snapshot services. They specifically market themselves to designers who just want to see whether a page that they’re developing renders right in all these different browsers. They don’t really target developers who need deeper functionality than that.

Also, at the same time, five years ago, virtual machines were not as commonly used, so if you only had a Windows machine and you wanted to see what a site looked like on Safari on Mac, you would submit your URL and get a screen shot back.

The most obvious limitation is that you can only give it one URL, and that’s it. They can’t test a multi-step workflow behind a login. So if you had, let’s say, a shopping cart, what you might really be interested in is to go from the front page, add something to the shopping cart, click “submit” and get all the way to the end of the purchasing process, then take a screen shot.

None of those screen-shot services could do that, unless you add some kind of automation mechanism to be able to navigate through the site to get to that point and then take a screen shot.

The genesis of what I originally wanted to do in the cloud was to take a classic screen-shot service and combine it with Selenium. Of course, I kind of took it to 11 and said, “Hey, why not, instead of just screen shots, record a movie?” And then you’ve got, theoretically, 30 screen shots per second that you can play with.

Scott: Is the size of the movies prohibitive at all? Does it take screen shots at set points, such as when it knows that a response is finished rendering, or is it really full-motion video of the browser?

Jason: Right now, it’s full-motion, in the sense that we’re running VNC as the protocol, remote desktop. VNC and RDP are the two main industry-standard protocols. VNC is kind of the open-source lowest common denominator, and RDP, which is the default one for Windows, is the next step up.

We just have a VNC server running on all of our test machines, and instead of connecting as a client application, we have a software library that I added some improvements to. The open source of that library is something called Castro. Instead of a user watching the remote screen, you dump the VNC traffic to what ultimately becomes a Flash video file.

Of course, we have done tweaking here and there, although it is inefficient if you do 30 frames per second. At five or even three frames per second, it actually works out well enough, though.

We’ve made some other improvements as well, although there are still more to be made. Specifically, I would love to bookmark the moments in the video where the interesting things happen, synchronizing it with the command you’re giving to the browser.

To your point, I could see where, instead of watching a video to get to those little bookmarked items, you could end up just getting something that’s a little bit more static. Using the raw video, for instance, you could go to each point of interest, pull out the video frame, and produce a sort of photo book or slideshow of just the relevant frames.

Scott: It would be similar in some ways to apps that detect when a sufficient number of pixels change, and so they know that some kind of transition happened.

Jason: Right. Perhaps it could be streamlined down to a slide show of only the key parts of the work flow, making reviewing the results go faster. Since we have recorded all of the video, we have all of the data, and we can then edit it in any number of ways.

Of course, people still come to us just expecting screen shots, which we will do for them. I think the next step is to synchronize the API traffic, the clicking, the typing, and so on, to make it easier to watch that flow of pictures, rather than having to explicitly take a screen shot of it the whole time.

Scott: Well, I think we’ve covered most of the stuff that I had planned to talk about. Any closing thoughts on your end?

Jason: The only thing that comes to mind is that we didn’t really get into the differences between Selenium user groups. There are developers on the one hand, using programming languages like Java, C#, Python, Perl, and Ruby. On the other hand, there are other Selenium users who don’t really do a lot of programming, but they use Selenium IDE specifically.

Sauce originally started out just to target those people who are using Selenium Remote Control. As we become more of a general Selenium company, we’re also embracing those users who are a little bit newer to automated testing, who are using Selenium IDE. We are branching out to do more things for those people as well. For example, we have a version of Selenium IDE, called Sauce IDE, that integrated with our Sauce OnDemand cloud service. So in a few seconds after creating a test, you can quickly start using our cloud service from inside the IDE.

Scott: Thanks for taking the time to chat.

Jason: Thank you.

Bookmark this:

• The evolution of various data models in databases
• Advantages of object-oriented databases in terms of horizontal scaling
• Handling issues when columns are added or deleted between application versions
• Suitability of MongoDB for the cloud
• OS support and usage with MongoDB
• The company (10gen) wrapped around MongoDB
• Community involvement in the development of MongoDB

Scott Swigart: To start, can you introduce yourself, 10gen, and the role of open source behind it?

Eliot Horowitz:Sure. I’m the CTO and co-founder of 10gen, which was really created to build MongoDB, an open-source, document-based database. We started 10gen and started working on MongoDB mostly out of our own frustrations after building scalable web infrastructure for a dozen years.

It really just came to a point where we were constantly fighting the same battles and building one-off solutions to handle functional gaps in relational databases. We started building MongoDB about two years ago, and we launched it earlier this year. People are getting very excited about it.

It’s very much targeted toward the standard web developer’s needs, anywhere from a simple one-guy website, all the way up to a Facebook-level, highly scalable, highly distributed system.

Scott: Building a storage engine is no trivial task, and beyond that, for a long time the bread and butter way of doing it was with relational databases. There was Oracle on one end of the spectrum and MySQL on the other, with a lot of other stuff in between, but people were used to throwing select statements at a database, joining tables together.

Then companies like Google came along, and over time, they’ve started to stored data differently. It’s sharded; it’s a schema-less store. You see things like SimpleDB up on Amazon, and I think Microsoft’s doing something similar and offering a data store. Talk a little bit about where MongoDB fits in the spectrum of databases.

Eliot: I think the first question you have to ask about any database these days is, “What’s the data model?” First, there’s the relational data model, which includes Oracle, MySQL, and all those. Then you have key-value, which has been around for awhile and includes things like MemcacheD and Dynamo from Amazon, where it’s really just “get input” type operations.

Obviously, those have been used for a long time, and they’re great for many cases. When you need simple gets and posts, they’re highly scalable, highly distributable, and very fast. They also have a very simple data model, although on the farthest possible extreme from SQL.

And then you’ve got a bunch of other data models in the middle: graph databases, tabular databases like BigTable, and document databases like Mongo.

There are a few different ways that you can think about document databases. One of the nice things about document databases is that they’re closely mapped to how most developers are writing code, whereas SQL databases were designed for accounting and banking 30 or 40 years ago, prior to the advent of web applications and the rise of object-oriented programming.

Now, when people are writing in PHP, Ruby, or Python, and they’re writing with data objects instead of rich classes (like user profiles). Trying to map that to relational data models is complex and slow. You can see the evidence of that in the number of different object-relational mapping libraries (ORMs) out there.

There are probably 1,000 different ORMs in the world, so clearly, it’s an unsolved problem, and every week you hear about a new one coming out. That whole class of problems exists because there’s a very clunky mapping from objects to relational databases. With document databases, that mapping becomes much simpler.

In many cases, it enables a direct mapping. You can take your object and direct it toward the database. And when you do need to do a mapping, the whole class of problems is massively simplified.

Another aspect of the same issue is that, when you loosen some of the versional constraints, you don’t have joins, because the data model is more suited to rich objects. For example, if you want to store tags for a blog post, you can store them as an embedded array rather than having a separate table that you have to join against. Once you get rid of joins, scaling horizontally becomes much simpler, which is a great benefit.

A bunch of the new databases coming out right now are trying to completely change the database paradigms, in terms of issues like how replication works or how you query the database. One of the things we’re trying to do with MongoDB is to keep it close to what people are used to. So, when moving from a relational database to MongoDB, there aren’t too many surprises.

You still have collections that are basically equivalent to tables, you still have different databases, and you still create indexes. You can still do an ‘explain’ to see what kind of query plan you’re going to get, and you can do ad hoc queries.

That means that if you’re coming from MySQL, it’s a pretty easy transition to MongoDB. The data model changes, but a lot of the things you’re used to are still going to be the same, mostly because we don’t think there’s any need to change them.

Scott: There was a guy a number of years ago who said that object-relational mapping is the Vietnam of our industry. It’s the thing that people just keep trying over and over to do, and it always ends up being excessively complicated. Back in my coding days, I spent as much time writing the mapping as I would have spent just writing the code to hydrate and dehydrate the objects.

Still, there have been object-oriented databases in the past. Talk to me about some of the things that make MongoDB unique or interesting compared to preceding object-oriented databases.

Eliot: The first object databases came out in the early ’90s, and there were a bunch of problems. For one thing, a lot of code was being written in C++ then, where they sped up the types, and the mapping layer really didn’t work all that well.

You could take a C++ object and save it, but not all the people were doing that. And then you had other languages where you didn’t really have objects the same way you do now. Remember that it was still very early in the trajectory of object-oriented programming at that point, so in some ways it was ahead of the curve.

The applications people were writing then were often more suited to relational databases, I think. Now, web scalability is a lot more important, because things can get big pretty quickly. I think people want much richer information from their databases, and they’re trying to use them in very different ways than they used to.

If you’re building a banking system or an e-commerce system, relational databases work pretty well. It’s really when you get into Web 2.0 sites that relational databases really start having troubles. If you look at Twitter, Facebook, or those kinds of websites, the needs have changed.

Really, though, I think the biggest thing is that the programming languages have changed a lot. No one’s writing web applications in C. Most web applications are now being written in PHP, Ruby Python, and some Java, all of which map very nicely to a document database. You can interact with any one of the languages and it all works pretty well.

I also think, going back to what we mentioned before, that the object databases before were actually more closely related to current graph databases than to document databases. The document database is really just taking MySQL, and instead of having a row, you have a document. So I think it’s a much simpler transition and it’s actually much closer to MySQL than a lot of people might think.

Scott: I’m calling it object-oriented, and you’re calling it document-oriented. Can you unpack that for me a little bit?

Eliot: In our case, a document is a BSON document, which you can think of as similar to a JSON document. You can have numbers, strings, dates, embedded arrays, and embedded objects.

If you have a user profile, instead of having first name, last name, then street, city, zip, you can have an address sub-object; you can have an address, and that can have fields. Then you can have multiple addresses, so instead of having address1 and address2 fields, you can have an array of addresses, and it’s already much simpler.

In your code, you can map that very cleanly. You have an address object that maps to this object, and you’ve got an array of them. You could have none, or you can have as many as you want. The database doesn’t really have a clunky interface, and you’re not joining, so there’s no huge performance hit. It’s all stored in the same place on disk.

Scott: Is a document basically a language-agnostic object, for lack of a better term? In other words, it’s not tied to PHP or Ruby or any particular programming language, but it basically has object concepts?

Eliot: Right; it’s basically like JSON, which is used pretty heavily in transporting data across the web now. We use BSON, which is basically like JSON except that it’s binary and has more types. JSON doesn’t have a native date type, and there are more types you need for web infrastructure.

Rather than hacking JSON, we just created a binary version that’s both easier for computers to read and extensible. It has more types, so you can have as many types as you need for real web applications.

Scott: Talk a little bit about scaling. You mentioned that because you don’t have joins and those sorts of things, and the data is all stored in one place, it scales horizontally very well. What does that actually look like, and how does it put the data back together, find which nodes different pieces of data are stored on, and resolve those kind of issues?

Eliot: The simple case is user profiles. Let’s say you have a billion user profiles and you needed 10 machines to store them all. Basically what you would do in this case is to say, “OK, I want to start my users collection,” and you could shard it on anything you want. You could shard it on email address or first name, or you could sort it geographically. You could shard it by country and then by state, depending on what kinds of queries you do.

Let’s say you sharded by using email addresses. Looking for a particular user is very fast. Now let’s say I want to find all people who live in Connecticut. It basically does a merge sort across all of the different shards.

It will go to each one and say, “OK, give me your users from Connecticut and get all of them,” and then it will aggregate them for you on the fly. From a driver/client perspective, it’s just the same if you’re sharded or if you’re not sharded. This is what people have been doing manually for years.

I’ve built many systems in the past where I needed 11 different databases to store something, and I did the sharding myself, which is a lot of manual work. What we’re doing is basically just generalizing what developers have been doing themselves in one-off solutions for years. It’s much simpler, because you don’t have to do joins.

A distributed join is very, very hard, but when your user profile is one object, it makes the problem much simpler.

Scott: It’s always interesting for me to ask where the edges are. What are some scenarios where a relational database may still be a better choice? Are there other tables where, maybe, a big table store is a better choice?

Eliot: I think big table solves the same general problems, although there we have some advantages over them. I think the document model is a little bit better than the tabular model, and there are a lot of subtleties there.

The big reason you use a relational database is if you need multi-object interactions. We don’t support multi-object transactions because of the performance overhead and complexity of doing distributed joins.

If you’re writing a banking system and you need to debit my account, credit your account, and do it completely atomically, we don’t support that. However, you can act atomically on any single object.

For example, you could say, “Increment this field on this object by five.” You can have it so lots of threads can do it, lots of different machines can do it, and it’s all atomic and safe.

Scott: Another issue that’s interesting is dealing with versioning. For example, in the course of versioning an application, you may change your objects, add properties to them, or various other things. With a relational database, it’s a relatively painful process up front, where when you change the schema, you’re changing the schema for the database.

That can mean that you have to populate a whole bunch of new columns with default values or do other things to migrate the database to the new version. With Google App Engine, it doesn’t work that way. You’re never really rolling the whole database forward to a new version, but you have to do more defensive programming

For example, let’s say you pull a user out of the database and that user hasn’t been touched for a version or two of your app. You’re pulling out the data the way it was stored the last time that user was saved, and so you may have to do some stuff in your code to map that older user data to your newer object. How do you guys handle that?

Eliot: You could migrate the entire database, but you don’t have to, which is the big thing. The simplest case is where you want to add a field. Here, the nice thing is that you can just start using the field. If an old object doesn’t have it, it will just return null, and it’s pretty easy and clean.

If you want to really change the format, you could actually have a migration script that migrates all your data, but that would require you to bring your site offline for an hour or so.

That’s another big change over the last ten years: people are no longer are willing to have maintenance windows. I remember working on systems where it was totally acceptable to have a two-hour maintenance window on a Saturday morning.

If Twitter went down for a two hour maintenance window once a month, I don’t think people would be too happy, and I think databases have to handle that. In any case, databases typically require you to do a little bit more work on the client side to make sure that you know what you’re getting out and you know what you’re putting back in.

Scott: On another topic, the cloud is a very important subject these days, and it seems like MongoDB is very well suited for something like Amazon EC2 and infrastructure on demand, where you’re able to install your own software and those kinds of things.

Obviously, the things that are more “platform as a service,” whether it’s ads or Google App Engine, have their own store, and to some degree you get what you get, which is what they provide. Talk a little bit about where you’re seeing usage of MongoDB in the data center versus in the cloud and some of the interesting uses you’re seeing for it.

Eliot: It does work well in the cloud, because it’s horizontally scalable, although it’s certainly not dependent on that usage. We see a lot of people running it in their own data centers, on their own hardware.

If you look at the startups that are using Mongo, a huge percentage of them are running on EC2, and I think that has less to do with MongoDB than with the fact that EC2 is a popular place to host sites.

We’re perfectly happy to run anywhere, but if you look at where people are building websites right now, it’s very heavily EC2 and other cloud-based solutions. You do not see very many startups buying and racking hardware to go on the Internet, which I did six years ago.

Horizontal scalability is a great thing for the cloud, because if you do grow quickly, you can add capacity very quickly. If you’re not horizontally scalable, the cloud doesn’t really help you that much; it’s actually worse, because you can’t buy or rent a $2 million Sun box on EC2.

Scott: If you start out with a small instance and you want to upgrade, and it’s your own hardware and your own data center, you can put more memory and CPU in it. With a cloud provider, you tend to get the type of instance you said you wanted, unless you want to shut it down and migrate the whole thing over, which causes that downtime.

Is this basically a Linux technology, or do you have significant Windows users?

Eliot: We have more Windows users than I would ever have guessed. If you asked me nine months ago if we were going to have anyone running in production with Windows, I would have said, “Maybe one or two.”

It’s a low percentage, but there are definitely many people running on Windows in production. We fully support Windows, OS X, Linux, and Solaris.

Scott: That’s interesting for the Windows developers. Is it sort of like PHP on Windows, or are people actually using it as a back end for .NET apps?

Eliot: PHP and .NET; not so much anything else. There’s a C# driver, and someone is working on a pure .NET driver. My knowledge of that right now is a little weak, but there are a bunch of different .NET things going on. I think there is a link adaptor, also.

Scott: We’ve talked a lot about the database and the project, so can you talk a little bit about 10gen and the company wrapped around it? What do you do around MongoDB, for instance?

Eliot: All the core MongoDB developers and the main developers of all the official drivers are 10gen employees. We basically sponsor the projects, and we are doing most of the development work. We also offer commercial support and training.

There are other interesting revenue opportunities for us, but those are the main things right now.

Scott: Talk a little bit about where it’s going. What are some of the top feature requests, or what do you see as some of the opportunities and interesting directions?

Eliot: If you look at our road map for this year, there’s no one big feature. I think the only big thing we’re doing right now is getting the auto-sharding to be fully production bulletproof. Traditional master-slaves and all that kind of stuff are pretty much bulletproof at this point.

The rest of it is just a lot of features. We are sort of similar to MySQL or PostgreSQL in terms of how you could use us, and people want all the features that they’re used to in MySQL and PostgreSQL. These include things like full-text search, SNMP, and all the assorted add-ons providing special indexing.

Scott: That must be an interesting thing to balance, because for example, MySQL’s original claim to fame was that it was simple, it was fast, and it wasn’t trying to be your enterprise relational database.

Then it just sort of kept growing and growing, because people kept asking for more things, until people looked at it and said, “What happened to my simple, lightweight, fast database?” And then you saw people going off and starting to work on Drizzle and those sorts of things.

How do you balance that request for one more thing versus wanting not to lose sight of what you set out to build and the scenarios that you really wanted to nail?

Eliot: I think it comes down to a few key things. One is making sure that we don’t regress on performance, which is one thing we’re very careful about. We constantly monitor our performance and make sure we’re not doing things that hurt it.

If there was a feature that would hurt our performance, we would think long and hard about implementing it, and we are definitely more interested in making the basics work than we are in adding more features.

We’re also careful about which features we let in, emphasizing common web infrastructure needs. It’s not useful for us to be simple and fast if you also need another database because you can’t do x, y, and z.

It’s very much the 80/20 rule. We want to cover the most general, wide-reaching things that people need, but it really comes down to good software engineering and good product management to make sure that we don’t screw up the basics. You don’t break the simple stuff by adding extra features. Another important aspect is good documentation.

Scott: I don’t want to throw too many software companies under the bus, but there is often a willingness to regress on performance, because they figure that hardware has gotten faster, so the user won’t be affected, even if, in terms of CPU cycles, it’s taking longer to do stuff.

How hard and fast is your policy of comparing the old version and the new one on the same box to make sure it didn’t get any slower?

Eliot: The next version coming out in a couple of months will be significantly faster than the previous version on the same hardware. We’re constantly working on that, so I’d like to think that that’s a pretty hard rule.

Scott: As time goes on, ultimately you want people to be using it, and you want it to meet people’s needs. Can you talk a bit about cases where, even if a change might hurt performance but really helps you nail a usage scenario, implementing it might be the right choice?

Eliot: I think a lot of that really comes back down to software engineering. For example, you can have such a feature but make sure that people can turn it off if they’re not using it, and that it won’t hurt their performance if they do turn it off.

We’re really trying to be very good on that level.

Scott: You mentioned that the people who are writing MongoDB mostly work for 10gen. Talk a little bit about the community story that lives around that.

Eliot: We’re seeing very few contributions to the core server code, mostly because it’s big, it’s flawed, and it’s complicated. There is a huge community, and it is unbelievably helpful in terms of the drivers. We’re getting lots and lots of great patches and contributors to the official drivers, so if you look at the PHP, Perl, Ruby, and File/Print drivers, there are tons of contributions.

The community has also been building a lot of great tools that sit on top of the drivers. One of the things we’re trying to do is keep the drivers pretty low level, very simple, and as fast as possible. If you want to hook into a login service, or if you want to hook up Apache to MongoDB for logging, we’ve seen a lot of great projects like that.

Scott: That’s interesting, in the sense that it’s a relatively common thread across a lot of successful open source projects that the number of people who contribute to the core might be really small, but the number of people who write add-ons, for example, might be very large.

It sounds like the ancillary components, for lack of a better term, related to MongoDB largely consist of those drivers, as well as administration and reporting tools that layer on top and add a lot of value.

Eliot: Right, and for example, there’s also a pretty standard adapter for Apache to log to MySQL. People are going to naturally want that to go to MongoDB, and there are going to be a lot of other different things that people will want.

That’s where the community has been very helpful, because we can’t be experts on a thousand different technologies, so it really makes sense for the experts in those fields to build them. It really works better for everyone.

Scott: Well, we’ve gotten to the end of our time, and I think we’ve covered everything I wanted to. Thanks for taking the time to talk to me today.

Eliot: Thank you. I’ve enjoyed it.

Bookmark this:

• The beginnings of DotNetNuke and engaging with Microsoft on open source
• Open source development paradigms interacting with proprietary software
• Seeking venture capital and monetizing DotNetNuke
• Response by the community to creating a commercial version
• Overcoming the pressures of a down economy
• Future directions: cloud platforms and improved distribution channels

Scott Swigart: Shaun, why don’t you take a second to introduce yourself and DotNetNuke?

Shaun Walker: I’m the original creator of DotNetNuke, which I started back in 2001, based on a reference implementation that Microsoft had released for the .NET Framework 1.0 Beta, an application called the IBuySpy Portal.

It was a sample application, released under a fairly liberal license, that allowed developers to come up to speed on the new .NET Framework. The application had some interesting characteristics, in that there was this notion of extensibility, although it wasn’t a fully baked solution.

I spent about a year on my own toying with that application and enhancing it. Really, what I had in mind at that point was building out a solution for amateur sports teams, so they could manage a website presence for themselves.

After about a year of doing that, I realized I really didn’t have the resources to convert what I had into a commercial venture. I had been active in the Microsoft developer community, specifically the forums on the www.asp.net site, where there was a small group of developers sharing their enhancements around that application.

It was actually Christmas Eve, 2002 that I decided to release my source code as an open source application. That initial release I called the IBuySpy Workshop, because I wanted it to have some connection to the original IBuySpy Portal.

I just announced it in that discussion forum on the ASP.NET site, and immediately, there were a lot of downloads and excitement around it, so I worked feverishly for about three months as people downloaded it and sent feedback. I think I was releasing every couple weeks at that point.

In March, Microsoft reached out to me because they had recognized that there was a bit of a community building around the application I had created, and they didn’t have any plans for enhancing the original IBuySpy Portal or building a community around it on their own.

Scott Guthrie reached out to me and invited me to come down to Redmond. We chatted about the vision for ASP.NET 2.0, because that’s what his team was working on at the time. We also discussed how DotNetNuke could be aligned with that vision and the importance of community, which I found to be quite refreshing, because it was not apparent to me that folks within Microsoft were thinking seriously about the benefits of open source.

We worked out an arrangement where Microsoft actually sponsored me for one year, which allowed me to leave my full-time job and focus my full efforts on building the DotNetNuke product and managing the ecosystem that built up around the project.

After that one year sponsorship, I had to find my own source of revenue to sustain the ongoing development. I relied mostly on advertising and sponsorship in the early years, then took on additional full-time resources to help manage the project.

It was in late 2006 that we actually took the bigger leap and formed DotNetNuke Corporation, which was a more suitable vehicle for governing the ongoing needs of the project.

At that point, I brought aboard as co-founders three of the contributors to the open source project who had been committed and loyal to the project. We worked under arrangement for about a year before we got serious about looking for venture capital. We were successful in getting a Series A round of venture capital in late 2008 from August Capital and Sierra Ventures.

Scott: We’ve talked to people who work on a lot of projects, from smaller ones to the largest ones like the Linux kernel and the Apache Foundation. Even though every open source project obviously evolves a little differently, what you’ve laid out there is almost a textbook story.

You started out scratching your own itch, like you said, with the IBuySpy sample that Microsoft put together, and it turned out that what you did was interesting to a lot of other people. A community formed around it, and so you decided to really invest in it and made it your day job.

The other people who came on board were basically people from the community who had made a name for themselves working on it. If you look at other open source communities, they’re also very reputation-based and reward people who actually get things done. Having an idea is great, but the ideas that tend to win out are those where someone has the idea and then rolls up their sleeves and implements it.

It’s really interesting to hear that basically that same road map was followed here as for the LAMP stack. This is built on top of Windows and the .NET Framework, completely proprietary technologies, but you’ve built an open source project and an open source business following the same trajectory you’d see elsewhere.

Shaun: I think that the open source paradigm transcends platform. It’s not necessarily a Windows versus Linux debate. The whole notion of open source is just built on openness, transparency, the willingness to share, and open protocols. All of that can live in either environment very comfortably.

It turned out that the solution that I’d come up with in those early stages actually related to a very large problem. That’s one of the other fundamentals of starting an open source project: you have to be scratching an itch that a lot of people are feeling.

The problem has to be large enough that it’s interesting for the initial person to solve, and it also has to be such that other people are willing to get involved in it and get excited about it. Without the passion, that organic growth really can’t happen.

DotNetNuke, right from Day One, enjoyed a very passionate developer community, because they were all working toward solving a fairly large problem: both individuals and businesses need website presences.

Some of the early tools that were available for entities to get online were really quite limited in their functionality, and so with the emergence of content management systems and web application frameworks, the sophistication of those online properties has gone up significantly. It’s really been software applications like DotNetNuke that made that happen.

Sean Campbell: I imagine that a lot of people, especially given the last year or so in the economy, are curious about that journey, going through and getting VC funding. What challenges would you reflect upon and say, “Well, if I’d known this before, I would have done something a bit differently during that process.”

Shaun: We spent a considerable amount of time doing fundraising in 2008, and I was surprised to learn the extent to which it’s not easy to raise money.

The fact that we had a very large and vibrant open source community wasn’t nearly enough for a lot of investors to show commitment. Really what they were looking for was some commercial aspect of the project as evidence that people were willing to spend money on this solution and not just download and use it for free.

Prior to this year, we didn’t really have much of a commercial component to the project. We could point to huge download numbers and a huge number of extensions that were available for the platform but we really didn’t have any evidence to point to business traction. That’s really what the venture capitalists are looking for.

They felt like they were taking a significant risk because it was an unproven model. In the end, we did find a couple of investors who were willing to take that risk and it’s worked out very well. We had a great year in 2009.

We offer support and other proprietary features around the platform and we’ve grown from 0 to nearly 400 paying customers in year one. We’ve hit all of our revenue goals and that wouldn’t have been possible without the injection of capital at the end of 2008. That allowed us to add more full-time resources to the project which really started to accelerate its growth.

Sean: How did you make the decisions around which aspects of the project you thought people would be willing to pay for and how did that guide the development of the project as a whole?

Shaun: In the very early stages of the project when we didn’t have a commercial version, we would implement pretty much any feature that came to the surface from the community as being desirable. Over time, we found that a lot of those features that we were adding really only catered to a small subset of the community, largely the more serious business users.

When it came time for us to introduce the commercial versions of the product, it was actually fairly simple to dissect the needs of the different constituents. We could look at a typical community user as really only running a single website off of a single install, with fairly limited content management needs.

On the other hand, a more serious business user might be using our platform to run multiple websites off of a single install, and they might have way more advanced content management and scalability needs, as well as demands around performance and security. So, if you looked at those different vectors, especially performance, security, and advanced content management, it became fairly obvious where different features should reside across the different product lines.

The other thing that’s important to note is that, right from Day One, the DotNetNuke application was architected in a way that’s very modular. So introducing proprietary functionality around the open core was fairly straightforward for us to do because the application had always been designed that way.

We have a very vibrant extensions ecosystem for both what we call skins, which are the look and feel, as well as mini-applications that can be embedded in a DotNetNuke framework, which we call modules. There are more than 6,000 different extensions available today on our marketplace at www.Snowcovered.com.

With that vibrant ecosystem already there and people already used to paying money for extensions, it was a fairly easy transition for people to adjust to the fact that there was going to be a commercial version of the core application as well.

Sean: Did you have any resistance from users who had been getting the application for free for years, when all of a sudden you started telling them that they would have to pay for certain features? Sometimes it’s just a vocal minority, but there’s generally a group like that.

Shaun: That aspect of open source in general speaks to the transparency aspect. Basically, anyone who’s involved in an open source project can get up and stand on their soapbox and vocalize what their concerns are. Unfortunately, their concerns can be quite self-serving, rather than considering the broader community or the broader context.

As we made the transition from being a pure, free, open source product by adding a commercial version, there was certainly a vocal minority in the community that expressed their concerns through our public discussion forums.

As a project owner, you have to be very attentive to staying on top of those and not just ignoring them, but speaking to people’s concerns so that they understand that broader context because often they hadn’t considered some different perspective.

It was surprising, but a lot of people in the community who were vocally opposed to us having a commercial version didn’t consider the fact that we were actually getting requests from many businesses that wished there was commercial support available for the product.

Some people have the perspective that if you use it for free, then support should be free, and they hadn’t considered the broader business implications of that.

Once you start explaining those things, people can start to get a bit more comfortable with it. You’re never going to please everyone all the time, though. There’s always going to be a certain subset of people that are not going to buy into your arguments, and they’re going to continue to be vocal.

That becomes a challenge to deal with on an ongoing basis but I guess we’ve gotten pretty savvy at dealing with those things over the last seven years. It’s no different than any other open source project.

To be honest, that friction is a good thing because it keeps everybody a bit more accountable. Even if certain people have a very extreme point of view, by being forced to listen to that point of view, it keeps you more from straying from what your true goals should be.

Scott: It’s become almost cliché, I guess, to talk about the economy, but you guys were going for VC funding during a pretty tough economic time. It’s always interesting to hear stories about people who have thrived in those circumstances.

Talk about some of the implications of seeking funding during this climate.

Shaun: We signed the term sheet in September of 2008 with August and Sierra, and a couple weeks later, the bottom fell out of the economy. At that point, we had the term sheet in hand and we were going through due diligence. Due diligence took much longer than expected, because we had some clean up to do in terms of making sure everything was in order from a legal perspective to satisfy the investors’ needs.

But the investors were committed at that point to doing the deal, so we worked through due diligence and were successfully funded and then they gave us a runway, which a lot of companies didn’t have, to focus on introducing some new products in 2009, adding a support component, and staffing up the business.

Where a lot of businesses were laying people off, we had the ability to actually hire people. We got to hire some really great people, because a lot of very talented people were out of work.

The other thing that worked in our favor, obviously, is that open source companies traditionally have lower costs than proprietary offerings. So when we look at some of the competitors that exist in the proprietary space, DotNetNuke is a more economical, more competitive solution in that regard.

I think that we picked up a lot of business this year based on the fact that we are a more economical offering for putting up a website.

The combination of those factors really helped us in this down economy, and as we’ve come out of this, hopefully 2010 is going to be a much better year for the global economy. We’re hoping that it helps us gain some additional momentum.

Scott: What do you see as the future of DotNetNuke?

Shaun: In 2010, we have some high-level objectives in terms of things we want to deliver. One area, in particular, is the marketplace where commercial extensions are sold for the platform. Like I said, there are about 6,000 extensions in that marketplace today, but it’s not really a seamless experience.

A customer has to install DotNetNuke and then they have to browse through our marketplace, make a purchase, and download a product. Then they have to upload it to their site.

As a software marketplace, the iPhone’s App Store provides a much more seamless experience, where you can view the different extensions that are available for the iPhone and choose to purchase them immediately. They download seamlessly to the device and they’re installed.

We want to build that type of tight integration between the DotNetNuke platform and the DotNetNuke marketplace in 2010. We believe that will really help consumers and businesses better manage their websites in a more seamless manner with less obstacles.

Another area that we see a tremendous amount of interest is around the cloud. It’s still challenging today if a business wants to put up a website. Although there are over 50 hosting partners today that offer a one-click install of DotNetNuke, if somebody comes directly to our site to download the software, they have to figure out what the next logical step would be to get it up and running. Basically, there are a number of obstacles in the way of a person accomplishing their goal.

This whole notion of cloud takes that complexity out of the equation as it provides a turnkey, instant gratification installation process that enables a customer to get up and running immediately in a suitable environment with all the necessary resources in place.

Scott: What’s your story around the Windows Azure platform? That seems like a logical cloud for you to land on because it is .NET based, and it’s not the most painful thing to migrate an ASP.NET application over to that.

Shaun: We are definitely talking to the Windows Azure team and it’s one of the cloud platforms that we would like DotNetNuke to run on. We are also talking to the Amazon EC2 team and, in fact, we have DotNetNuke running on EC2 already.

Like you said, Windows Azure is a natural choice because it’s native Microsoft technology but, at least in the early releases of Azure, it’s not a no-brainer in terms of getting a .NET application running on Azure. You have to do some re-architecture and, at that point, I don’t know if you’re tied too closely to the Windows Azure platform.

Microsoft has recently come out with some announcements that they are going to be offering some Windows virtual-machine-type deployment scenarios, much like those Amazon provides. That might be an interesting cloud option for DotNetNuke, as well.

You’ve also got to consider that those are the two predominant cloud players but there are a lot of smaller players, as well, like Rackspace and even smaller hosting providers that are offering cloud type offerings that give you a lot more control.

One thing that I’ve noticed with some of the larger players is that, in order to commoditize the service and allow it to be offered at a low price point, they actually take away a lot of features and a lot of flexibility that you would traditionally get in a typical web-hosting environment.

That’s where a lot of the smaller players are able to compete, because they can provide cloud-type services but with greater flexibility. So, we’ve got to look at all of these things when we look at running in a cloud.

Scott: It sounds like you are definitely rapidly pursuing the cloud, in a provider-agnostic way, and you’d like to be an option for as many clouds as reasonably makes sense.

Shaun: Right; we are avoiding lock-in, which aligns with our open source ideals.

Scott: Thanks for taking the time to talk today.

Shaun: My pleasure.

Bookmark this:

• The origins of Freenet and anonymous peer-to-peer architecture
• On the mathematics behind “Six Degrees of Kevin Bacon”
• Observations about optimal strategies for managing an open source community
• The Swarm project’s approach to distributed data
• Implementing fork/join and related concepts in distributed systems
• Relative beneits of open source and proprietary approaches to development
• Incorporating open source into an overall business strategy

Scott Swigart: To start off our conversation, could you please take a moment to introduce yourself and talk a bit about the different efforts you divide your time between?

Ian Clarke: Sure. I have a degree in computer science and artificial intelligence from the University of Edinburgh in Scotland. I started my career as a subcontractor to the European Space Agency.

Prior to graduation, I did some work on emergent systems and anonymity on the Internet. That grew into an open source project for which I act as founder and coordinator called Freenet, which started back in 1999 and has been ongoing ever since.

When I made Freenet publicly available after I graduated in 1999, it ended up getting quite a bit of publicity, and we released our first version in March 2000. In fact, a paper we released in 2000 describing Freenet ended up being the most cited computer science paper of that year.

Freenet was kind of novel, in that it was the first defense that allowed anonymous peer-to-peer architecture. Freenet also was precursor to the concept of the distributed hash table, which is a technique to efficiently store and retrieve data from distributed computers.

I moved to California and started a company, raising $4 million in venture capital, primarily from Intel. We developed a peer-to-peer content distribution technology, which we sold in 2003. All this time, Freenet was my hobby going on in the background.

I then did some consulting, and as part of that, I designed a peer-to-peer video distribution system for Janus Friis and Niklas Zennstrom, who I’m sure you know are the creators of the software you’re currently using.

I handed that off to their team in 2006, and it became a project known as Joost, which you can look at as the next big thing after Skype. I then co-founded a company called Revver, which was one of the early entrants into the short-form video-content sharing space. Revver is unique in that we were advertising against video back into 2005 and sharing the revenue with content creators.

Scott: How did that set the stage for the projects you’re working on now?

Ian: One of the problems I was working on as Chief Scientist at Revver was how to target individual videos with specific ads to maximize revenue. I became very interested in the whole field of collaborative filtering, predictive analytics, and recommender systems.

My current day job, which is a project called SenseArray, grew out of that. One area I learned a lot about with Revver concerns the challenges of scaling up a website, scaling up databases, and worrying about what data goes where and what data is cashware.

I believe that a lot of the architectural problems inherent in building a website and having a bunch of people actually start using it should not be the concern of the software engineers. Those issues should be handled by the framework in a mostly transparent way.

Those thought processes gave rise to a project called Swarm, which I spoke publicly about only a couple of months ago for the first time. Right now, it’s at a very early conceptual prototype stage, because I’m trying to get people interested in helping me work on it. Swarm is extremely ambitious and encompasses some very difficult problems.

Scott: Freenet sounds like the kind of thing that people might have been more interested in 2002 or later, in the sense that it predates the Patriot Act and widespread eavesdropping over global communications. Freenet provides an anonymized way to get out content and communicate with other people.

What do you think made Freenet interesting at the time you launched it, and how did it become more interesting because of changes culturally and geopolitically between 2000 and today?

Ian: When we first launched Freenet, it really generated two kinds of interest which were almost orthogonal to each other. One could have easily occurred without the other, but they both happened at the same time. The first kind was academic interest in what Freenet was doing and how it was doing it.

Freenet did a couple of things that had never really been done before. One of the core problems that Freenet tried to solve was to find information quickly from data that is distributed across a large network of computers. Many of those computers are unreliable, and some could be operated by your enemies.

You cannot rely on any form of centralization in order to do that. You probably remember Napster, which was a peer-to-peer network that maintained a central directory recording where everything was. Of course, that central directory can be shut down, and that really was not an option for Freenet.

At the time, Freenet was designed to be used in countries like China, Saudi Arabia, and Iran, where governments are not constrained by constitutions and laws in the same way that they are in this country.

Freenet was intended to operate in a very hostile environment, which creates some difficult problems. How do you find a piece of information, without consulting some centralized entity that keeps track of where it is? I developed a technique for doing this that relates somewhat to small world theory.

Scott: That’s the same theoretical basis as the Six Degrees of Kevin Bacon game, right? [laughs]

Ian: Exactly. The whole notion that you can connect between any two people through about six other people actually has a lot of mathematics behind it.

It turns out that if you just randomly create a network of things, you cannot so easily get from any one point to any other without omissions. For that to work, it must be a small world network, and it just so happens that human relationships do form a small world network, whereas that’s not true of any arbitrary network of things.

One of the novel things in Freenet was exploiting small world theory to solve the problem of searching and retrieving data in a decentralized, efficient, scalable way. I think that’s part of what generated the academic interest, and some of the stuff we were doing with cryptography was also quite new.

We certainly did not invent any new cryptographic algorithms, but we were using existing algorithms in some novel ways. That generated interest in the academic communities, computer security communities, cryptography communities, and what later became the peer-to-peer communities, although Freenet slightly predates the term “peer-to-peer.”

The other type of interest we were getting was mainstream interest from media like the New York Times and 60 Minutes, which related primarily to the copyright implications of something like Freenet.

Around about that time, of course, Napster was getting a huge amount of publicity, but a lot of people realized that Napster would probably get shut down, and here was Freenet, which could not be shut down. That generated a lot of mainstream media interest that really didn’t have much to do with the academic interest.

In some ways, they conflicted with each other, especially since some academics dismissed Freenet because it was getting so much mainstream publicity, which is kind of a weird effect.

Scott: What did you learn from Freenet in terms of managing an open source community? A lot of people have talked about why communities succeed or fail, but what are some of your lessons learned from that experience?

Ian: I have made a lot of mistakes, and I think I will continue to, so I certainly do not really see myself as being an authority. I think what has worked quite well for us has been being very open in terms of allowing people to join the development effort. I know that in some open source projects, it is incredibly difficult to gain access to source controls, and it is a big deal when you do.

We take the opposite tack. If somebody emails us and says they have an idea about how to do something, we give them access. If they do something wrong, we can revert it, which is the beauty of source control. We do everything we can to lower the barriers to entry for people to contribute to the project.

Another lesson has been that it’s very important to explain to people that they shouldn’t wait for someone else to tell them what to do. I find that a lot of people send messages to our mailing list, saying that they want to be part of the project and asking for something to do. Invariably, nobody will respond to them.

People need to know that nobody is going to laugh at them or shout at them or get mad at them if they just try to improve something. They can just find something that needs doing and then go do it.

Initially, one of our most prolific contributors over the years was Oskar Sandberg, who actually ended up doing a Ph.D. on stuff related to Freenet. He initially joined the project by writing a very simple Perl utility. To be honest with you, I can’t even remember exactly what it did, but it was something extremely simple, and from there he got more and more involved.

It’s really about lowering the barrier to entry, but also making it clear to people that they have to take stuff. They can’t wait for it to be given to them.

Scott: In other words, it’s better if someone comes to the project with their own itch that they want to scratch rather than “Hey, I just want to be part of this put me to work.” We hear from a lot of people that the best developers are often the people who are really using it all the time and so they know, “Hey, this is cool but it would be even cooler if it could do X.”

To return to Swarm for a moment, one of the things you mentioned that got my attention is that this is an incredibly ambitious undertaking. What is it that makes Swarm so ambitious, and how is that likely to impact the management of the community around it?

Ian: To put that more in context, I should first talk a little bit about what Swarm is. The basic premise is to build an environment where computer programs can be written so that they can either run on one computer just like a normal program or be distributed across many computers in a way that is mostly transparent to the programmer. The fundamental concept behind Swarm is to move the computation, not the data.

All computer programs are essentially a collection of code that operates on data, and the data can potentially be distributed across multiple computers. The concept with Swarm is that, if code that is executing on one computer needs to do something to data that is on a remote computer, you should move the computation to where the data is.

The conventional approach would be to move the data to the computer where the code is executing, and the Swarm approach is actually a little bit like the MapReduce concept that you may be familiar with.

Scott: In fact, I was just thinking that I should ask you how this approach is similar or different from Hadoop’s approach.

Ian: They both share the maxim “Move the computation, not the data,” but Swarm is really a lot more general. In order for a problem to be solvable through MapReduce, it needs to be possible to break it down into a map operation and a reduce operation.

Some problems are reducible in that way, especially stuff like indexing web pages, but many problems are not. In particular, anything that requires the equivalent of a database Join between two tables simply can’t be broken down into a MapReduce in any kind of efficient way.

In short, then, MapReduce is the same general concept but only applicable to a specific class of problems, whereas Swarm is a lot more general. Really, you can write any computer program, and with Swarm in particular you can write your code in Scala

I’m not sure how familiar you are with Scala, but I consider it to be the successor to Java. It has a lot of benefits of Java but without a lot of its annoyances and with some neat features that Java may never get, like closures, type inference, and that sort of thing.

You write a computer program in Scala, and it can really do anything. There is no requirement that you can break it down into MapReduce operations. It can just be an arbitrary computer program.

Scott: Can you share a few particulars about the Swarm approach to handling distributed data?

Ian: Let’s say an object on the computer that is executing happens to have a pointer to an object on a remote computer. At that point, Swarm essentially freezes your computer program, serializes it through a mechanism called continuations, and migrates the computation to the remote computer, where it starts up again.

From the perspective of the programmer, you don’t even need to know that this is happening or where your code is executing. Swarm can actually move the execution state of your code around between multiple computers in a way that the programmer doesn’t really have to think about it.

Now obviously, it’s not inherently very efficient to serialize a computer program’s state and migrate it across the network, so the second component of Swarm figures out how to arrange the data to minimize the number of times that the program state has to migrate from one computer to another.

Those are essentially the two components of Swarm. Firstly, allow a computer program’s state to jump around to follow data, so that you can just distribute data across multiple computers and then the computer program just follows it where it needs to go. The second component helps us be smart about where we put the data to maximize efficiency.

In terms of what makes this hard, the continuation mechanism I mentioned is something called portable continuations, which are supported by very few programming languages. Even Haskell cannot support portable continuations, and you know, if you can think of an obscure capability for a programming language, chances are Haskell can do it.

[laughter]

Ian: It just so happens that in Scala 2.8, which is the upcoming version, somebody implemented portable continuations. In many ways, this is the kind of thing that makes Swarm even possible.

In other areas of difficulty, there are all kinds of concurrency issues when code is acting on data that can be distributed across multiple computers. You have to deal with persistence and consistency, and all of that.

Problems of consistency and concurrent access to data are part of a very active research area at the moment, and there are a lot of hard problems there. How do you know when it’s safe to delete data so you can efficiently implement garbage collection across many computers? How do you implement stuff like atomic transactions when your data is distributed across multiple computers?

All the time, you’ve got a separate overseer process that’s watching how your computer program is jumping between computers and trying to move data in order to optimize that. I’ve got a lot on my plate, and I cannot follow all of these problems myself. At the moment, I’m trying to attract people with expertise in areas like handing concurrency and software transactional memory to the Swarm project.

Scott: I am thinking about the approach you describe and comparing it to a more traditional environment, where 100 web servers are talking to a handful of middle-tier servers, which are talking to one huge honking database. You scale that thing up as large as you possibly can, and if you can’t scale it up any larger, you fail.

I’m also thinking about the big table approach, where data is shared across lots of nodes. With something like Google App Engine, you get scalability, but you lose some relational database operations like Joins. You may be limited in terms of not being able to do the equivalent of a select that returns a million rows. You’re limited to a thousand results or something like that.

You talked about Joins and a lot of these concepts, and it sounds like what you are doing with Swarm is envisioning a novel data store. The data is distributed across a lot of nodes, and there’s an optimizer that’s always positioning the data so it will be the most efficient for the code to come to it.

At the same time, though, you’re still supporting what people tend to think of as relational operations, like Joins and those types of things. Am I envisioning it correctly?

Ian: I think you’re absolutely right that you could do things like Join using Swarm, although that would really be at a higher level of instruction. So, for example, somebody could implement a relational database on top of Swarm, but Swarm is not itself a relational database. Really, Swarm’s representation of data is as an object graph, much as data is represented in memory.

Scott: It seems like the concept of moving the execution to the data becomes more and more important as everything now just generates terabytes of data, from super colliders to oil fields.

It’s no longer practical to move the data to the computation, and you talked about your code running in one place for a little while and then having its state packaged up and running somewhere else.

Still, a lot of times what you want is for your code to sort of fan out. How does Swarm deal with the notion of making the logic fan out and go parallel when it can across nodes? And does it take that responsibility off of the developer, or is that something they would have to be conscious of?

Ian: This is something I’m thinking about at the moment. For example, one thing you could do is implement something like a fork/join approach to parallelization. I know there is a fork/join framework that will probably be part of the next major version of Java. You could absolutely implement something like that on top of Swarm, where you explicitly say to it, you can parallelize this if you want.

I wouldn’t say the parallelization is completely transparent to the programmer. It’s not like the system is examining the code and being very clever about figuring out what can we do in parallel and what we need to do serially.

What you can do is implement something like fork/join so the programmer can explicitly give the system permission to parallelize something. Then Swarm will decide where it makes sense to run on one thread or whether to fork it and send different parts of the execution to different other machines and then recombine it all at the end.

In short, the answer to your question is that you can do parallel processing with Swarm, but it probably won’t be completely transparent to the programmer.

Scott: Still, it sounds like it would be implemented from the programming standpoint in more of a declarative way, rather than the with imperative fork/join syntax that you would have in other languages.

Ian: I think that is almost certainly true, although I should say that a lot of this really is very much still on the table. You could implement fork/join as a library on top of Swarm. Swarm gives you primitives such as fork a process or communicate between processes, and you have a lot of flexibility about what you build on top of that.

Scott: Taking the conversation in a slightly different direction, with Freenet, you manage the community, and with Swarm, you’re trying to attract people to tackle some nearly intractable problems. Then you also do proprietary software development.

Talk a little bit about the difference in the way you see things working in the proprietary world versus the open source world and where you see advantages or disadvantages to either approach for certain kinds of problems.

Ian: There is the obvious benefit in the proprietary world that when you employ people, they more or less they have to do what you say, which is certainly not the case in the open source world, where you have to persuade people.

Open source can make it a lot more difficult to put processes in place, because programmers by nature often resent processes unless they strongly believe in them, although even in the commercial world, if you impose something and nobody believes it’s any good, then it’s probably not going work very well either. You can implement structure more easily in the proprietary world though, I think.

The proprietary world also tends to present fairly clear goals. If you’re going in the right direction, people will buy it and keep paying for it, and if you’re going in the wrong direction, they won’t buy it, and you don’t have a business.

With something like Freenet, it is hard to know what exactly you’re trying to be. For example, are we trying to service people living in countries like the U.S., people living in China, or both? The goal of proprietary software is to make money, and hopefully you can translate that back to how you need to prioritize different feature-functionality today.

Scott: I’ve never heard anyone call that out as an advantage, although it makes perfect sense. To put it another way, in proprietary software, there’s very low latency between the moves you make as a software company and the signals the marketplace sends back to you.

On the other hand, a lot of open source projects don’t even know how much of it is out there. It’s very nebulous how much “share” it has, so there’s a little more latency between the time a project starts to head in the wrong direction and the point where it’s really receiving clear information, that it is negatively affecting the use of that project.

Flip it around and talk about some of the advantages of open source that aren’t really present in proprietary software. For example, if I am a Fortune 500 company, I’m not in the business of building a web server, but I need one. It’s a lot more beneficial for me to devote a handful of programmers to Apache so we get the things we want, rather than building one from scratch.

Ian: That is certainly very true. From the perspective of the industry in general, open source permits people to create and collaborate on aspects of infrastructure that don’t need competitive advantages, but just need to work.

Because of that factor, you can have Microsoft and other companies, whether they like it or not, wind up cooperating with each other to the benefit of all. If you look at it from a game theory point of view, proprietary software can be more of a zero-sum game that tends to discourage collaboration, whereas open source tends to encourage or even enforce collaboration if it’s a viable open source license.

I certainly believe that the software industry is much better off with open source software. But on the other side, in many situations in my career, I’ve been building proprietary software, and people have said, “Oh, you should open source it. Wouldn’t that be great?”

It would be great for people in general, but it would remove the incentive to create the software in the first place. Because generally speaking, open source business models tend to be services-based business models, which do not scale as well as product-based business models.

To return to your original question, the advantages of open source versus proprietary are different, depending on whose perspective you take. You can build a business creating open source software, but I think it’s more difficult.

Scott: We talked to a VC, and his comment was “Sometimes we give open source credit for being more different than it is.” There are a lot of companies that go with kind of a “freemium” model, where they have a free version of the product, and then they try to up-sell you to the enterprise version. It’s the equivalent of a loss-leader, which isn’t new in business, or the modern equivalent of the 30-day evaluation.

Do you think it’s true that we sometimes highlight more differences between open source development and proprietary development than there really are?

Ian: It all depends on your strategy. I think there are many different reasons for open source. For example IDEA, the Java IDE, recently released a community version. I certainly hear a lot about companies that build open source community versions and then try to up-sell people to a commercial version.

I have not seen a lot of solid data demonstrating the success or otherwise of that approach. I think there are some situations where a company has a wider strategic interest to create a platform; like for example the Google Android platform, or Google Wave.

But, in those cases, a much longer-term strategic interest informs those high-risk projects. I think a lot of Google’s open source projects do not make money, and will not make money, even in the longer term. But luckily enough, they’ve got a cash cow in the form of advertising that they can use to subsidize all of these things.

Scott: You see a lot of experimenting, but it seems that notable successes are not all that prevalent. It’s also hard to determine whether success is because they went with an open source/enterprise business model, or whether something else contributed to their success.

Just to be sensitive to the time, can you add any closing thoughts?

Ian: I’m a huge user of open source, and I have several hats on my head. I’ve got a software engineering hat, where I just want there to be cool stuff, and anything that helps there be cool stuff is all right by me. I think almost all open source falls into that category. I think open source’s existence benefits the software ecosystem and innovation in general.

Then I’ve got my entrepreneur’s hat on, where I’m thinking in terms of whether being a producer of open source can make me money. The software I write is predictive analytic software, and I typically sell it to CTO-level people.

It’s quite a technical sale, so one of the challenges is to find customers and get them to pay attention for long enough that they actually understand what the software does and whether it can help them.

I have a blog where I will release small, relatively self-contained snippets of code that do stuff that relate, in some way or another, to what my product does. I know that the type of people who are going to find these snippets of code may well be the type of people who work for a company that could use the software I create.

It’s not so much a loss-leader as it is a publicity tool, and that is definitely one way in which producing open source is in the business interests of even a small software company.

Scott: Thanks for taking the time to chat.

Ian: Thank you.

Bookmark this:

• Eucalyptus as a technology-neutral abstraction layer for cloud computing
• Enabling utility pricing models for compute resources
• Toward the ability to mix and match platform services with infrastructure services
• Open source significance and strategy in Eucalyptus
• The evolving role of community in the Eucalyptus project
• Real-world implementations and usage scenarios

Scott Swigart: Could you start off by introducing yourself and Eucalyptus?

Rich Wolski: Sure. I’m the Chief Technology Officer and the college professor whose research group did work on Eucalyptus originally at the University of California, Santa Barbara.

The project began as a computer science research investigation. We were looking at how to combine large-scale HPC programs in fields like chemistry, physics, weather forecasting, and biology with the emergence of utility computing offered by third parties, which is to say, cloud computing.

We started looking at that question in an empirical way, using a driver application of weather forecasting code. The goal was to combine Amazon’s AWS, a number of university data centers, and the NSF supercomputer center site in one large execution platform for this legacy weather code. To do that, we actually wound up having to build an interface layer for that code that could run in university data centers.

To make our programming job easier, we made that interface layer look just like Amazon AWS. We were faced with the problem, from the application perspective, of porting the code to Amazon. That interface layer–that emulation, if you will, for university data centers–is what we released as open source, which became Eucalyptus. That’s how we got ourselves into commercialization.

It’s an open-source platform that’s actually architected to support a bunch of different APIs, with flexibility in terms of virtualization, operating systems, and hardware. It implements cloud computing with a sufficiently high-fidelity imitation of Amazon that the weather forecasting code we were working with couldn’t really tell the difference between when it was running at Amazon and when it was running at the universities.

Scott: When I think of things like the Amazon API, there are two things that spring to mind. The first is the API that you access, generally remotely. From my machine here I could spin off a hundred instances and provision them with Amazon machine images.

I could shut them down, I could take snapshots of images or take snapshots of the block source. All of the admin stuff that you can do through the management console is what I think of as the Amazon API.

If I understand what you just said, you guys replicated that API so that if I’ve got code locally that’s spinning up instances and kicking off jobs on Amazon, it could also kick off jobs through, say, a third-party vCloud provider, as long as they’ve bolted on Eucalyptus.

It’s exposing that same API, so I can federate my clouds and do the same logical operations even though, for example, starting an instance on VMware’s infrastructure is different from doing so on Amazon’s. Do I understand correctly that you have built a sort of abstraction layer so that I don’t really see that as a user?

Rich: Yeah, although I think it’s important to understand that that abstraction layer is more than just the API. Really, what Eucalyptus does is implements the cloud abstractions in an essentially technology-neutral way.

There are a set of abstractions that all successful clouds today support, and we’ve figured out how to implement those abstractions independently of the API that accesses them and independent of the virtualization technology that is used to implement them.

Eucalyptus is in the middle, as a generic cloud infrastructure. It is built so that you can manipulate that cloud infrastructure using a number of different APIs, although the one that has been most successful for us is Amazon. That’s the one that we ship with open source, but we’ve done others.

On the bottom part of it, there are lots of different virtualization technologies that can be used to realize those cloud abstractions, both in open source and in proprietary forms. Eucalyptus is built so that it could use any of those virtualization technologies to implement the abstraction.

Scott: So, in other words, you are positioning Eucalyptus to be the lingua franca of the clouds, in terms of starting instances, stopping instances, and that sort of thing.

Rich: Right. There really are three sets of abstractions that most clouds support. There are VM abstractions for starting, stopping, and manipulating virtual machines. There are abstractions having to do with storage, such as what persists, what does not, how it is accessed, and what the range of access is. Finally, there are abstractions having to do with networking: how your VMs are named, how can they communicate with each other internally and externally, and those types of things.

We have an internal framework that is able to combine different virtualization technologies to implement those abstractions and then export them via different APIs.

Scott: When an instance is running on Amazon, the instance itself doesn’t really persist data. So if you shut that instance down, its “local drive” basically gets blown away, whereas with other providers, that may not necessarily be the case. Or you may set up a portion of your data center so the instances are persistent.

How do you deal with those underlying differences that surface up through whatever kind of cloud or virtualization architecture somebody is using internally?

Rich: That’s a great question, and the answer is that the Amazon API, and in fact all the other APIs we’ve looked at, have really well-developed notions of Quality of Service attribution. I’ll just speak about Amazon concretely, because that’s the one that seems to motivating a lot of others.

In Amazon, there is this notion of VM type, and there is also a notion of availability zone. Availability zones in Amazon are there specifically to give you some idea of independent failure probability. Two different VMs working in different availability zones have independent failure probabilities, so you can calculate the likelihood that you’ll lose both of them simply by calculating the joint probability.

We actually co-opt that a little bit and use the notion of availability zones as a collection of common QoS characteristics. One of those could be “is it failure-independent?” but that is not necessarily the only one.

For example, you could have two different availability zones in a Eucalyptus cloud, where the underlying infrastructure is implemented in two completely different ways. And what you want to export to your users is the notion that each specific availability zone has a particular set of QoS attributes.

Therefore, if VM persistence is an attribute that you want to export in one or more of your availability zones, you can do so. Often, there are capacity-allocation issues associated with that. If you’re going to do VM persistence, you have to have more of a persistent store available, and the Eucalyptus administrator is responsible for doing that capacity matching, so that the size of the availability zone, the things that can go in it, the users that are entitled to request image starts from it, and so on and so forth, are properly controlled.

Scott: Does costing show up in the API at all, in terms of VM, storage, and network resources? In other words, is there a way through the API to know what an instance is costing you, or is that something that you need to know in advance?

Rich: That’s actually something we think about a lot. There’s no problem in putting the cost into the API, but it’s difficult in many cases to compute exactly what that cost should be automatically. If you’re charging against a credit card, you can calculate the occupancy cost or the number of bytes or whatever. There’s some monitoring that’s necessary to do that, but you can do it.

On the other hand, if you’re making calculations based, for instance, on the financial costs of running a VM, it’s far more complex. There are factors based on the cost of the hardware, the operating costs, and a specific life span of that hardware in a specific company. It’s a very difficult problem, because of all the factors that go into the cost benefit associated with owning that machine.

Of course, you can use average costs. Rather than calculating the operating cost on a particular machine, you can assume a reasonable rate and feed that to the system as a parameter. It would be no problem to export that through the API, although I’m not sure you get much utility out of calculations based on a static number like that.

On the other hand, Amazon has just announced spot pricing, which changes the equation. Now the price is going to be fluctuating dynamically, and you don’t want to know so much what something cost you in the past, as much as you want to know what it’s going to cost you in the future, and that’s a whole different ball of wax.

I do think that almost all of the APIs at some point will include some notion of cost, but the utility of that is really going to be predicated on how predictive it is. I don’t think the providers have gotten to a point yet where they can reliably produce those kinds of predictions, so I suspect we’re a little ways off before we see it.

Scott: I was thinking more in terms of the customer-facing price showing up in the API, rather than the internal costs. That seems to be a simpler matter, since as you said, there are a million things that factor into internal costs, many of which aren’t really programmatically discernable, such as what your people are costing you and those sorts of things.

The notion of spot pricing leads me to another thought. Today, an IT manager may have a workload that they think may be well suited to the cloud, because their demand is fairly elastic. Maybe the organization needs 100 nodes for just a week, or two hours, or whatever.

Customers only want to pay for peak capacity when they need it, so they’re manually looking at various cloud providers and trying to find a fit for their needs. They may also be considering an internal cloud that bursts to a public cloud as needed.

Of necessity, they’re looking at all these relationships very individually, in terms of the characteristics of each provider and so on.

I think about when web services were just catching on, as the trendy technology of the moment. There was a lot of talk at the time about web services marketplaces and directory services, where if you wanted a weather service, you’d go to a directory service and find one. The stuff you’re talking about seems to lend itself to the evolution of a kind of cloud marketplace, especially if cloud capacity really is a commodity.

In fact, that’s the direction that people like Amazon seem to be pushing it, where there are spot prices for instances, for CPU time, and that sort of thing. It would allow you to monitor a marketplace of many providers and place workloads dynamically based on changing prices. Am I thinking too sci-fi here, or do you actually see this happening?

Rich: I actually agree with everything you said, and I would enhance your description in the following way.

There are business uses for which commodity markets with fluctuating prices make a lot of sense, assuming that the offerings are completely interchangeable. Typically, customers buy the lowest common denominator, because it’s a volume business.

At the same time, there are real advantages in certainty and long-term contracts, and being able to repurpose things you buy based on unforeseen events. Spot pricing can be either a hindrance or a boon, since if the spot price happens to be poor when you really need to move on something, you might have been better off with a longer term fixed price fee for service.

There was a study at UC Berkeley, I believe, of network bandwidth. They tried all sorts of pricing models for people in their homes, and they found that, in fact, spot pricing for network bandwidth at home is a really bad idea. People liked it the least of the various pricing options that were made available to them.

I think there’s going to be a computational commodities market, and I think the key there is going to be interchangeability. You have to know that if you buy it from one provider or another, or if you run it in-house, you really are getting the equivalent service. And in computing that’s tricky, because it’s a multidimensional quality of service bundle, and you have to match on all dimensions. Multidimensional matching is a hard problem, particularly if things are statistical.

It’s kind of hard to do that equivalence relationship, but you can do it, at least to some approximation. And for the things where you can deal with spot pricing and make those kinds of decisions, I think it’s going to work just fine.

I don’t see that as the only way that public cloud providers are going to offer their services, though, and I also think that independent of whether the market is commoditized or not, the on-premise cloud is going to play a role, acting as a hybrid.

Scott: That makes sense relative to the electricity market, too. Governments, agencies, and companies enter into long-term contracts with each other to get determinism around price, but there’s also a spot market for power.

To turn to a different subject area, Eucalyptus lets you get to the notion of arbitrary infrastructure as a service. While each service provider may have its own unique set of value adds, broad availability facilitates federating workloads across providers.

It seems like platform as a service started to evolve almost completely separately, but they seem to be coming together a little bit more now. Google App Engine was probably the first platform as a service provider that a lot people really noticed.

The first option was to go the infrastructure as a service route with Amazon, and then you were your own administrator. You had to figure out what OS to load, patch your own boxes, and so on, but you could basically do whatever you could with a virtual machine. On the other hand, you could go with Google, which required a bit of a rewrite, but there were all kinds of things you didn’t have to worry about.

Now there seems to be work happening toward being able to layer platform-as-a-service frameworks on top of infrastructure-as-a-service offerings. I can envision future where you can sort of mix and match those.

How does Eucalyptus fit in to that world, where you’ve kind of got arbitrary platforms as a service on top of arbitrary infrastructures as service?

Rich: Let me start off by saying that when Eucalyptus was a research project, we teamed with another research group at UCSB that was interested in doing for Google’s App Engine what we did for Eucalyptus, namely building an open source version of GAE. They layered it on top of Eucalyptus, and it’s called AppScale. It’s still a research project today. You can get AppScale and Eucalyptus, run them in your data center, and have GAE and Amazon all to yourself.

From the beginning, we really thought that people would want to mix and match platform services and infrastructure services. I think it will be interesting to see how applications work at different levels in the stack. One of the key questions with regard to this layering is how much of the inter-relationship between the application and its data can be explicitly represented.

With platform as a service, you wind up uploading into the cloud the data you wish to process on and the code you wish to process it with. In infrastructure as a service, you have to upload those two things and you have to upload the environment in which that code is going to run.

I think you can characterize software as a service in the terms that you upload your data, or you upload no code. With platform as a service, you upload your data and you upload your code, but you upload no environment. With infrastructure as a service, you upload the data, the code, and the environment.

Given that schema, what really differentiates platform as a service from infrastructure as a service in terms of the power-benefit trade off is how much you can tease apart your application code and data from the ecosystem of software in which it lives.

That’s a harder job than it might sound. We all think our code lives in isolation, but it’s been a long time since people have written anything of substance that lives completely by itself.

If you want to use platform as a service, you have to get back to that sort of software engineering purity, where the environment in which it runs, including the libraries, operating system version, security policies, and so on are really kind of digested from the code.

Particularly complex, scalable applications consist of multiple components, some of which are perfectly suited to platform as a service deployment, and some of which are not for whatever reason.

Because of that, we think the key is this layering, which allows platform as a service and using application components that make the most sense. If that’s the premise, we hope you use Eucalyptus, but if not, we still think that layering makes a lot of sense.

Scott: I talked to some guys who were doing Google App Engine development, and one of the things they said that was really sort of eye opening to me was that platform as a service is fantastic if you can live inside of that sandbox. So, with Google, for example, everything is a request and a response, and you have a certain amount of time to process the request; otherwise it just basically kills it. But, you get infinite scale, you don’t have to patch your OS, and those kinds of things.

That worked for 80-90 percent of the project they were working on, but for the other parts, they would have Google App Engine call out to other stuff running over on Amazon Web Services, for example, that would do some of the longer-running kinds of things and generate results.

The picture you just painted allows that whole thing to exist inside of that data center. You can have the infrastructure-as-a-service piece and the platform-as-a-service piece talking to each other, relatively seamlessly. Maybe you even run them on the same servers, and to accommodate service peaks, things move around relatively transparently, and from an internal IT perspective, that’s going to be the norm.

Platform as a service is going to be great, infrastructure as a service is going to be great, and they’re going to have to communicate with each other, because sometimes one of them is going to be a better fit than the other, and large complex apps are frankly going to require both to some degree.

Rich: That’s right, and I think you see in Azure that kind of acknowledgment. I know this sounds strange, but when I look at the Azure architecture, I see those platform-as-a-service types of restrictions built in. Still, you could imagine exporting in Azure more infrastructure-as-a-service type functionality, and then that would have to be coordinated in a reasonable way.

I think one advantage of Eucalyptus is that as you were saying, if you want a Ruby platform, a Java platform, a Python platform, or some other platform of your choosing, all of those things can happily coexist over the same infrastructure as a service framework.

Scott: Let’s flip over and talk a little bit about the open source nature of Eucalyptus and its significance.

Rich: There are a couple of ways to view the presence of Eucalyptus as an open source project. The easy one is the motivation for us to do it as open source, which is that we as researchers have really have spent a lot of time working with and benefiting from other open source efforts.

If you are working in the university environment today, doing empirical system science, you are almost assuredly using somebody’s open source tool kit or platform or software infrastructure to do your work.

And so we felt like it was paramount that if we built something that might have utility for other researchers’ investigative activities, we should make it available as an open source. And frankly speaking, before we even thought of commercializing Eucalyptus, we knew we were going to make it open source. Having now gone down this path and having become a commercial entity, we still feel very strongly about this.

Even from a personal experience perspective, we feel that open source has a lot of value that can be passed on to the community, as long as you can support that notion commercially. From a business perspective, I think there are some fairly good reasons to do it as well.

The first is that we get access to a lot of information very quickly about what Eucalyptus does and does not do. That information is freely given to us as people take the software and try things with it. We can see, almost instantly, which things we have done that have been successful, where we need to work on the quality of the system, where we need to work on its stability, what features might be missing, and those kinds of things. The open source community is very good about providing that information.

A second aspect that was a little less clear to us when we started this, but which is actually turning out to be very valuable, is that Eucalyptus is built to run in pretty much whatever environment you have.

We consider Linux environments primarily, but we really spent a lot of time when we were building it initially thinking about university data centers where we don’t know what version of Linux is there, we don’t know what the vintage of the hardware is, we don’t know well how it’s administered, and those kinds of things. As a result, we’ve architected the system to be flexible in this way.

What we’ve learned from the open source community, though, is just how widely varied those environments can be. We’re constantly hearing from people who say, “This is what my data center looks like, and Eucalyptus doesn’t do X, Y, or Z. Please help me.”

That type of interaction is not really feedback about the code or what Eucalyptus does, but rather it’s feedback about how Eucalyptus can be configured. Through that information, we’ve been able to make Eucalyptus extremely configurable, so that it really does allow itself to be molded into whatever environment you want to put it in. And that, I think, is part of its value proposition in a commercial sense that we didn’t anticipate.

Scott: Just to drill down a little bit, remind me what license Eucalyptus is under. Is it an Apache license, BSD, GPL, or something else?

Rich: It’s actually under two licenses, which has to do with the university and commercialization phases we’ve been through. The university license was a BSD-style license, modified in a way that the University of California, Santa Barbara absolutely had to have in order to allow us to release it from the university.

We had to modify that license when we commercialized, because it’s a UCSB-specific license. At that point, we took a look at it, thought about what the rest of the community was doing, and switched to GPL. It’s actually GPLv3 going forward, and as time goes on, less and less of the university code is present in the actual open source code base. At some point, it will be all GPLv3, but for now it’s both.

Scott: That makes sense. In terms of community involvement, you mentioned that you get a lot of feedback from the community about how it’s working in a certain environment, specific issues, and things like that.

With some open source companies, the majority of the people working on the project work for the company. That was sort of the way MySQL was originally structured.

Rich: Right.

Scott: Others really want the community contributing code. Where do you see yourselves on that spectrum, in terms of where the code comes from?

Rich: We’re actually moving away from one extreme. When we started out and looked at taking community contributions, we found that we were rolling the code forward so fast that people were contributing code that was absolutely against our internal code base. And so they would get upset because they put a lot of energy into building code that we couldn’t take.

What we decided was that, in the first phase of the project, we would really limit code contribution to things that talk to the Eucalyptus API, meaning things on top of Eucalyptus or attached to it or communicating through the API, and then to take patches. We would take bug fixes, as long as they were not obsoleted by a bug fix we had internally or whatever.

What has happened is that we were trying as hard as possible to do a complete implementation of AWS while AWS was adding features. As a result, we decided to finish off the initial AWS API for open source, do a set of bug fix releases to re-factor some things internally for performance and stability reasons, and then adopt a much less aggressive release schedule.

Moving to a status where we’re releasing every six to twelve months instead of every six or eight weeks makes it more feasible for us to entertain contributions to the core. It’s important for us that the QA be done well.

We’ve put a lot of energy into making Eucalyptus extremely concurrent, which is a big differentiating factor between Eucalyptus and some other open source offerings. Eucalyptus is a very, very concurrent system, and what we find is that when people add things to it, in order to get stability, they often put locks in, because it’s easy.

That approach gets their thing to work, but at the cost of reducing the concurrency. We probably will look very long and hard at those contributions to make sure that we preserve the quality of what’s there, but I imagine that in the next year or so we’ll see community contributions to the core as well.

Scott: I remember seeing news articles that mentioned, for example, Ubuntu and Eucalyptus in the same paragraph. What are your relationships to distros or other open source projects?

Rich: We’ve always wanted to make Eucalyptus as widely valuable as possible in the open source community. The Ubuntu partnership has been fantastic in that way, as they have really put a great deal of energy into adding things that they believe their distribution community really wants.

They put a lot of energy, for example, into improving the installation experience in a way that is much more compatible with what the Ubuntu community is accustomed to seeing. And we think that’s great.

By the same token, we are open to working with other projects or companies that have different needs. We do work with the Debian community, we’re actively working to package it for Squeeze as part of their open source offering, and we’ve talked with some of the other distros about whether it makes sense for us to be part of the distro or just to remain compatible.

We have agreed to continue to package Eucalyptus for the Linux distros that people ask us to package it for, and we explore ways that we can have closer partnerships. That’s ongoing, but so far, in the distro space, our closest partnership is with Ubuntu.

There is a lot of synergy between what Eucalyptus does and what other projects do. So again, in this platform, there is this sort of layering capacity. We have ongoing compatibility and synergy discussions with a number of different partners.

There is also an ecosystem building up for tooling. RightScale is a good example of a company that provides powerful tools for using Amazon. You can program your Amazon allocation much more effectively through RightScale than you can by hand.

Because we’re Amazon-compatible, RightScale works with Eucalyptus, so that’s a nice partnership. If you want to use the same kind of dashboard to manage your Amazon and Eucalyptus clouds, you can, and there are a number of companies that provide that kind of tooling, backup services, and other things that complement what we do.

We think of those partnerships as being really important, particularly since we’re open source, because we think that the success of cloud computing today is going to come from an active ecosystem and not the success of any one single technology.

Scott: Talk a little bit about your business model. How do you guys keep the lights on around this open-source project?

Rich: We’re practicing what I think is commonly referred to as an open-core model, where we provide the core of the system as open source, and then there are value added components that can be purchased from us under license. The bright line differentiator for us is whether the feature or component is really tailored towards the notion of commerce or not.

If you want to experiment with cloud computing, if you want to understand it better, or if you want to see what it does and what it doesn’t do, then open source Eucalyptus is for you. If you want to make money with it and start a business using Eucalyptus, then there are things we think you need that we feel like you should, in some measure, pay for, because you’re going to go off and resell the value that comes from them.

So, for us, it’s actually pretty easy to see what goes in open source and what doesn’t. We have professional engagements that surround our proprietary products. We also offer support, under contract, for the open source code, but that’s not really our main business. We’re really a software development company, and we’re leveraging the open source community and the open source project for the proprietary products we ship.

Scott: Obviously, people are using Eucalyptus internally for things that are kind of confidential and not public, but what are some of the really cool stories you can talk about where you’ve seen Eucalyptus put into production?

Rich: The best example of where it’s in production is actually something we can talk about. NASA took Eucalyptus and added Luster, which is a high performance shared file system, and some other things, and they’re running it in production at NASA Ames.

They offer cloud services to other agencies within the government, and we know it’s in production, because they call us and they yell at us about it. We think that’s pretty exciting.

The other stories around Eucalyptus are really more usage scenarios. We hear a lot from the open source community about what people are doing with it. We can sort of talk about that, in the sense that they only tell us so much.

There are interesting use cases that have emerged from Eucalyptus. One is the cloud bursting of the disaster recovery model. People think of that one almost immediately. A lot of people are also looking at the idea of augmenting their on-premises infrastructure with public clouds.

But then there’s also this funny phenomenon, where developers become frustrated with the pace at which they can get resources dedicated to their projects, and they pull out their own credit cards and start developing in a public cloud. Often that code is very valuable, and when that comes to life, the company for whom that developer is working often wishes to pull that stuff back into the company and run it on premises until it can be properly vetted, software engineering can be done on it, and so forth.

So, we see this going both ways. We see businesses that want to burst from their on-premises IT into the public clouds, and then we see development which taking place in one venue, such as on the public clouds, and for whatever reason, the people who own the development want to move it back into the private space, perhaps only to move it forward.

The result is a kind of homogenization of the on-premises IT with public utility providers. We think that represents a really exciting future usage scenario for Eucalyptus.

Scott: It’s a slow moving trend, because it’s difficult for companies to get there, but there’s pressure for internal IT organizations to restructure themselves essentially as service-delivery arms.

We’ve talked to people in several companies who have told us, “I have to compete with everything out there. Any business unit that comes to me is free to buy from Amazon, they’re free to buy from Google, they’re free to buy from whoever they want, and I’ve basically got to make them a deal. I’ve got to deliver a better service at lower cost.”

Rich: We hear this idea that internal IT finds itself competing with commodity providers, but I predict that will prove to be a short-term phenomenon. Whenever a commodity emerges, it becomes inexpensive as well as very common. In the technology business, there’s often an advantage in using something that’s more rare, before your competition and before it becomes a commodity.

Where your business has to innovate, that’s really not necessarily a commodity business. I think what we’re going to see in the future is that on-premises IT will really be the engine of innovation, experimenting and coming up with the best ideas before the competition can do them.

And then as those advances commoditize, becoming the sorts of things where you’re going to compete on price and really try to drive the margins down, there will be pressure to move them from your internal infrastructure, out to the utilities, to make space for the next innovation.

I think long-term, that’s really what IT is going to look like. I don’t think it’s all going to be public, and it’s certainly not all going to be private. I think the business model that rationalizes how to partition resources between public and private is going to be driven by that innovation cycle.

Scott: That’s great. Looking at the time, I think that’s a good place to wrap up the interview. Thanks very much

Rich: Thank you.

Bookmark this:

• Background on Hadoop and its origins
• Implementing support for MapReduce to handle distributed data
• Suitability of Hadoop for cloud computing
• The relationship between Cloudera’s Hadoop version and the Apache version
• Cloudera’s business and monetization model around Hadoop
• Licensing models as they apply to Cloudera and Hadoop

Scott Swigart: Could you introduce yourself and Cloudera?

Amr Awadallah: I’m CTO, VP of engineering, and cofounder of Cloudera. I was previously VP of engineering for data systems at Yahoo!. That’s where I first got exposure to Hadoop and recognized the need to form a company to commercialize it. The company is just 13 months old now.

Scott: Can you fill in some of the blanks for our readers who may not be entirely familiar with Hadoop?

Amr: Sure. With traditional data systems, you have a bunch of filers where you store your data. A compute grid is connected to these filers, which essentially is a bunch of computers that fetch over the data, do computations on it, then store the data back.

That model worked fine, but it started to break when data sizes got too big and moving the data between the storage network and the compute resources became a big stress on the network infrastructure.

There was also a big stress on the filer heads. A filer consists of a bunch of disks, and all these disks are attached to a head, which is really what does all the serving. The filer head itself starts to become a bottleneck when you’re running a job that accesses a large proportion of your data. If you’re just searching a file or two within your data, it’s still OK.

Those factors are what drove the need for Hadoop. We had massive amounts of data to process, and existing systems were not capable of scaling sufficiently. Hadoop itself came out of the open source community, though Yahoo! played a very big role, as well.

Scott: I’m always interested in the actual mechanics of how something as complex as Hadoop comes to be. Can you speak to that specifically?

Amr: Two developers named Doug Cutting and Mike Cafarella were trying to build a web-scale, distributed search engine. They built a system called Nutch about seven years ago.

Obviously, the system needed to be massively scalable to index the whole web, so they were spending a lot of energy in that area. Around that time, Google published two key papers, on Google MapReduce and the Google File System, GFS. It’s the marriage of those two technologies that enables the large scale data processing that we can do today.

GFS allows linking a lot of commodity hardware together to present one large, unified file system. MapReduce has two parts: the programming model and the job execution framework.

The MapReduce job execution framework lets you spread your job over multiple nodes, so that the first level of mapping, where you’re reading your data in and doing the first level of aggregation, is all local. All the communication is purely local between the computation resources and local disks. Therefore, you don’t take a performance hit or stress the network from passing data over it.

In the second phase, which is called the reduce phase, some inter-node communication takes place, which does necessarily stress the network infrastructure to some degree. Usually by that stage, though, the data has already been reduced in size to the subsets that you care about, so the stress on the network infrastructure is dramatically reduced.

When Doug and Mike saw these papers, they had a “Eureka” moment and started to integrate these technologies into their web-scale search technology with Nutch.

Scott: How did the work at Yahoo! coincide with what Doug and Mike were doing with Nutch? I assume that Yahoo! must have adopted an open source approach in place of some of their proprietary algorithms. Is that true, and what went on behind the scenes with those decisions?

Amr: At the time, Yahoo! was also doing a lot of work internally on their infrastructure for serving search. As the web grew exponentially, the cost of serving a search was getting prohibitively high, and Yahoo! needed a more scalable system to cost-effectively build the web index.

They saw the MapReduce and GFS papers from Google as well, and in light of the work being done by then on the open source Hadoop project, they recognized that they had a choice. They could either build proprietary MapReduce and GFS technology from scratch, or they could adopt Hadoop internally and support the open source project.

A very big debate went on inside Yahoo! about the right direction to go. The proponents of the open source approach made the very good argument that supporting Hadoop would take away one of the key advantages that Google had. Namely, only Google had the systems to really take advantage of MapReduce and GFS, which locked in a tremendous advantage for them.

They argued successfully that it was in Yahoo!’s best interest to unlock that technology, so other startups could start adopting it and building ideas around it. The idea was that this approach would shift the power from Google.

In response to those arguments and others like them, Yahoo! hired Doug Cutting on staff and invested very heavily on building a large Hadoop team. In fact, the vast majority of the code base in Hadoop today came from Yahoo!, so they certainly deserve a lot of credit for that.

Scott: How did Hadoop move beyond its foundations in search?

Amr: After Yahoo! started to integrate the technology within its web infrastructure initially, Hadoop started penetrating a lot of other areas of Yahoo! as well. Now it’s being used in Yahoo! mail for spam filtering, as well as on the Yahoo! front page to model which stories to display to which users based on their interests. It’s being used for ad modeling and ad targeting, as well. Facebook was another early adopter and supporter of Hadoop.

Around that time, which was maybe two years ago, it was starting to be clear that Hadoop had implications for more than just the big web properties. The technology is useful to many other domains as well. In fact, this notion of moving computation to run as close as possible to the data is something that everybody should be doing, given how big data is growing these days.

This trend is being reinforced by the fact that commodity servers are becoming very powerful in terms of the number of cores and the number of discs you can have in them. Now you can have eight cores very economically, and within the next year you’re going to have 32 or even 64. You can have anywhere between four terabytes to even 48 terabytes of disk space on one of these standard server nodes.

It has become very clear that the trend is toward large hardware topologies, where systems will need to be very resilient to failure. They will need to know how to shift workloads and data around when things fail, so the system just heals itself.

Scott: One of the things that this model depends on is moving away from a traditional relational database that doesn’t really fragment the data across lots of nodes. Instead, those traditional systems are really based around the idea of a single, monolithic body of data.

That seems to be one of the big paradigm shifts for developers. When they start using something like BigTable where the data is distributed across a lot of nodes, they have to think a little differently about the way they build applications.

Working our way from the bottom up, talk a little bit about thinking differently in how you store your data so that you can use a MapReduce process on top of that.

Amr: First, it’s worth making the important clarifying point that Hadoop is not a database. Hadoop is a data processing system, and in fact, I would even go as far as saying Hadoop is an operating system. The core of an operating system boils down to a file system, the storage of files, and a process scheduling system that runs applications on top of these files.

There are many other components that help with devices, credentials and user access, and so on, but that is the core. Hadoop is exactly the same thing. The core of Hadoop is the Hadoop Distributed File System, which is a file system that’s runs across many nodes. It links together the file systems on many local nodes to make them into one big file system. Hadoop MapReduce is really the job scheduling system that takes care of scheduling jobs on top of all those nodes.

That is the key distinction between Hadoop’s approach and that of database systems. Hadoop, at its heart, does not require any structure to your data. You can just upload files directly from anywhere, like a web server, RFID device, or cell phone mobile device, directly into Hadoop.

They could be images, videos, or just a bunch of bits. They don’t have to have a schema with column types and so on, which gives you tremendous agility and flexibility.

Hadoop has a very nice model that I sometimes refer to as schema on read. Whereas defining your schema as you’re writing the data in limits what you can put in by requiring it to be conformant to the schema that you created, Hadoop allows you to define the schema as you’re reading stuff out.

That gives you a lot of flexibility and agility, since you can add files that have dynamic parts like JSON or new standards coming up like Avro, which is a very good project coming out of the Hadoop project that’s similar to protocol buffers from Google and Thrift from Facebook. Avro makes files have a schema around them as well, but these schemas are semi-structured, rather than conforming to a strict relational model.

That said, it’s also important to point out that structured stuff is a subset of unstructured stuff. The fact that Hadoop at its heart is a file system doesn’t mean that it can’t do database relational stuff. It does actually, in the same way that Windows at its heart is a file system, but you can run SQL Server on top of it to get the relational services, schemas, column types, and so on.

One of the key projects on top of Hadoop is Hive, which actually came out of Facebook. Hive essentially provides a relational database on top of Hadoop that utilizes the underlying file system but has a metastore that keeps the schema of the files.

It knows that a given file is tab delimited or whatever, it knows the column type for these files, and Hive allows you to write SQL against these files. It will look up the schema and then it will write for you the MapReduce jobs so that you don’t have to go and learn MapReduce from scratch.

Now you have the flexibility of going either way. One approach is to get at the core of the MapReduce framework using Java MapReduce, which we sometimes refer to as being like assembly language for Hadoop. It gives you the most flexibility and performance, but it is fairly complex and difficult to learn.

Alternately, you can go in with a high level language like Hive. In this case, you can just use SQL, if that’s what you’re used to, to write your job. Hive itself has lots of optimizations. It understands the underlying MapReduce framework, so it can properly map your problem on top of your data.

Scott: Obviously, search is a logical use of MapReduce, but I’m sure there are many others. For example, in the oil and gas industry, a single oil derrick might produce terabytes of data a day. In other industries, there might be huge amounts of financial data to plow through.

It seems that every industry is faced with figuring out ways to deal with these large bodies of data. Given that, what are some of the more interesting uses or problems you’ve seen thrown at Hadoop?

Amr: The group at Yahoo! that I came from was using Hadoop for data analytics and data warehousing. We had something like 100,000 web servers across the world, and once we collected data from across all these servers, we dumped it into Hadoop, which became the place where we stored all of the data, instead of traditional network storage.

Our reasoning for doing that was a matter of economics, given the quantity of hardware. Hadoop lets us scalably process that data, clean it up, and normalize it so we could pass it along to the systems that need it.

Hadoop is getting very wide adoption in the data warehousing and business intelligence domains. One of the biggest uses within Yahoo! right now is dealing with all of the log information from servers. Analyzing that information allows for better spam filtering, ad targeting, content targeting, A/B testing for new features, et cetera.

It’s not web-specific. For example, everybody does data warehousing, and we see very strong adoption there.

Separate from that, your example of oil companies is a very good one, as is the financial sector. Right now, we do have a couple of very large financial institutions working with us on these exact problems, taking huge amounts of data from domains like credit card processing and building predictive models for fraud that enable better decisions, for example, about whether to block or allow a given transaction.

In the stock market, Hadoop is being used to do simulations that help predict option pricing and related problems. That’s another very healthy market that we’ve seen growth in.

Another interesting market is genomics, specifically for doing gene sequence mapping and protein sequence alignment. In many ways, that is a fuzzy string matching problem, where you have a string of DNA that you need to match to a template string that has certain diseases identified in it.

Any individual task is not that big, but there are so many of them that you need to split the work over many nodes and do the matching in parallel. Government agencies have a number of implementations that involve massive bodies of data as well.

Scott: Let me turn our line of conversation in a slightly different direction. Cloudera obviously has the word “cloud” in it, and implementations like Amazon Web Services have Hadoop implementations.

Hadoop seems to intersect with the cloud fairly well. When I think of the cloud, I think about having an API to start up and shut down nodes, utility computing, and being able to provision and shut down relatively large numbers of nodes remotely, programmatically.

What makes the cloud and Hadoop a good fit for each other?

Amr: I completely agree with you that the cloud is essentially a set of resources that you can scale up or down with a nice API, based on dynamic demand. There are many services that fit that description, of course, and I should add that a cloud is not always external to your organization.

Even though Cloudera has the word “cloud” prominently in our name, we are not building a cloud or a cloud infrastructure like Amazon AWS or Microsoft Azure. Rather, what we’re building is software that enables cloud computing. Hadoop is at the heart of that, although we’re also building other components around Hadoop that strengthen that kind of operating proposition.

Hadoop fits very well with the cloud model, because in many ways, Hadoop is like a virtual machine, albeit one that actually takes the inverse approach to what one generally associates with virtual machines.

That is, a virtual machine hypervisor allows you to partition a single server to look like many small servers. On the other hand, Hadoop is about linking together a bunch of physical servers to look like one big server, which starts looking like a cloud service.

You can have many users using this resource, and the resource can be scaled up or down, depending on the demand it’s being hit with. That’s exactly the sweet spot for Hadoop.

We’re working very closely with Amazon, and their Amazon Elastic MapReduce service gives you the option to use the Cloudera specific distribution of Hadoop, although Amazon has the generic Apache distribution of Hadoop as well. If you want to get a maintenance support contract, you obviously need to use our distribution.

Amazon’s business is about providing the cloud service itself, and not about building Hadoop. Our business is about building Hadoop itself, so that’s the synergy there.

Scott: As you mentioned, there is an Apache version of Hadoop and then there’s the Cloudera version. As different companies wrap themselves around different open source projects, they’re structured in different ways. Talk a little bit about Cloudera and what you add to the public open source version of Hadoop, in terms of additional software, support, or services.

Amr: I should start by saying that Cloudera is an enterprise software company. Open source is an enabler for us, and it’s part of what we do, but our mission is about building enterprise software for large-scale data processing in internal or external clouds.

In terms of how we go about working with the community, Hadoop is an Apache licensed product, and the Apache license is one of the most consumer-friendly licenses out there.

The Apache license allows anybody to take the code and do whatever they want to do with it, whether that means using it internally, redistributing it, even packaging and selling it. That flexibility makes it a very nice license, but of course, it means that you have to add value in order to make money from the code.

The way we make money is a bit similar to Red Hat, although Red Hat does charge for the distribution, whereas we don’t. You can get our distribution directly from our site without having to pay us anything.

There are many differences between our distribution and the Apache distribution. Primarily, our distribution is well-packaged as RPMs, or RPM packages for Red Hat, or Deb packages for Ubuntu or Debian. That makes it much easier to install using standard tools, and in fact, we are talking right now with Microsoft about how we can package it properly to run on top of the Microsoft Windows platform.

The other distinction is that the Apache organization can be a bit slow in terms of launching new Hadoop releases. We release new distributions almost every two weeks, with the latest and greatest features.

Of course, we release them as test distributions, and we tell people to be careful how they use them. As with other fast-moving distributions, the bleeding-edge stuff can have problems as well. Yahoo! also has a distribution, which is essentially the Apache Hadoop code that has been tested on the Yahoo! internal infrastructure. Everything that Yahoo! has in their Hadoop distribution makes it to the Apache Hadoop release, and similarly for any contributions that Cloudera makes to our distribution. We aren’t forking off the code base, it’s just a matter of the Apache Hadoop official release lagging behind a bit.

Scott: What is the ongoing relationship between Yahoo! and Cloudera?

Amr: As I said before, Hadoop owes a lot to Yahoo!, in terms of how they made it into what it is today, and they continue to contribute a lot. Yahoo! has a very big Hadoop team–I’d say almost 100 people right now. They have lots of open positions as well, for new people to join that team.

A lot of the Hadoop patches that we pick up were developed and tested by Yahoo!, and that’s what we include in our distribution. We do our own development and testing as well, in addition to what Yahoo! does. We listen to our customers to decide on which features/improvements to start working on next. Yahoo’s development roadmap is obviously a function of their internal needs.

Scott: How does Cloudera monetize their work with Hadoop?

Amr: Right now, the only reason you would need to pay us is if you need our maintenance service. Think about software where there’s no upfront licensing fee but there is a maintenance fee. The maintenance fee buys you access to our team, and you can open tickets and ask questions.

Companies using Hadoop in a production environment need a partner they can fall back on when they have problems, and we are that partner. We can also provide advice about what network hardware to use, how to properly balance the number of hard disks and cores on your machines, and how to configure Hadoop itself.

Hadoop has something like 200 different system parameters that you need to properly configure depending on the infrastructure you have and the type of job that you’re running. Writing code using Java MapReduce is a bit of an art right now, and we can help with that to make sure your code is properly written with respect to the MapReduce model.

These are the kinds of services that we offer. We also offer quick patches. If you encounter a bug specific to your problem, and you want to make sure you get the patch for it right away so you can continue operating your cluster, we’ll fix the patch for you very quickly. We have a number of Apache Hadoop committers on staff.

I forgot to mention that Doug Cutting, the inventor of Hadoop, left Yahoo! in September and joined our team. Tom White is another key committer we have at Cloudera. He is also the author of the O’Reilly “Hadoop: The Definitive Guide” book, which is the “bible” for Hadoop right now.

The other thing we do is training, in terms of learning this new technology, how to work with it, and how to develop problems within this new scalable way of thinking. We provide training from the very basic, fundamental kind of stuff all the way to writing algorithms specific to the problem that you are trying to solve.

That’s how a lot of open source companies start, where their main revenue stream is maintenance fees; this allows them to make healthy revenues in their early years.

Most open source companies evolve to an open core strategy, where the core platform is free, but functionality around the core is not necessarily free. That is the direction that we will be moving in.

We’ll keep the Hadoop core free, because that’s Apache, and we’ll continue contributing to that. Indeed, more than half our engineering team is focused on making the Hadoop platform better, but at the same time, we’re now investing in building infrastructure around Hadoop to strengthen the platform.

Scott: It’s interesting to consider that dynamic among companies that are wrapped around open source projects like Joomla, WordPress, or MySQL. Every one is different, even with their similarities.

It’s interesting to me to understand how different companies are structured, because it seems like early on, there were lots and lots of different models people were trying around open source. It seems like the market seems to be settling out a little bit.

One common approach is a sort of aquarium model, like Google or MySQL, where it’s being written by people who all work for a company, and they put the source code out there for all to see. There may be a patch from outside the company here or there, but most of the development is being done by people who all work for the same company.

On the other hand, there’s the example of the Red Hat model, where there’s a pretty vibrant community and the product draws code from a lot of different places. They do a lot of value-added stuff around it, such as providing an enterprise-grade version, fast patching, training, and consulting services.

These two seem to be the predominant models that at the moment. Do you see it in a similar way, or do you see other ways companies are structuring around open source?

Amr: I completely agree. The Apache Software Foundation follows the latter model. In fact, they only allow projects to be “official projects” within Apache if the committers in the project are diversified over a number of companies.

If just one company controls the full code base, Apache will not let it become a core project, which I think is a very healthy approach. It does lead to certain logistics and coordination problems, but it allows the software to evolve much more quickly.

It also has sort of a Darwinian advantage, in the sense that when a number of companies are all working on a problem and there is more than one solution, the community can just take the best one. That said, you do lose control a bit, which might slow things down, even though it helps foster innovation, so it is certainly a balancing act.

Scott: Maybe a year ago, we interviewed Justin Erenkrantz of the Apache Foundation, and that was one of the things we discussed–that requirement for a project to have maintainers from at least three different companies to make it out of the incubator and become an official Apache project.

In another area, the Apache license is very different than the GPL license, for example. How does that fit into the strategy of Yahoo! or Cloudera? Do you think the Apache license versus the GPL license affects the evolution of Hadoop?

Amr: It does play a very big role. As I said earlier, the GPL license has a viral aspect in terms of distribution. Whenever you redistribute the code, you are giving up some rights back to the owners of the code. On the other hand, the Apache license doesn’t have that viral nature to it, which makes it a much friendlier license, in terms of adoption.

We have to work harder to make our distribution, training, and maintenance services really stand out, and to provide additional value on top of the vanilla Apache Hadoop codebase. After all, customers can just go and get the code directly from Apache, without involving us at all.

Scott: There are obviously differing viewpoints on licensing, with some people who really like the GPL for what it does, and some people who really like the Apache license for what it doesn’t do, or what it doesn’t make you do. People argue about which is really the most free.

Amr: That’s a good point, and of course, I have my own opinions about it. Our CEO, Mike Olson, led the open source revolution from the beginning. Mike was the CEO for Sleepycat, which commercialized BerkeleyDB, one of the first open source companies.

Scott: We are running out of time, so I don’t want to miss the chance to ask you whether there’s anything you’d like to bring up that we haven’t discussed yet.

Amr: Hadoop has been around for a few years, but it’s still very young, relatively speaking. There is still a lot of maturing for it to do, and that’s what we are spending most of our energy on right now. We, with the Apache Hadoop community, are doing some great stuff in areas like performance, usability, availability, security, auditing, interoperability, and so on.

We are also building new data system products that augment the Hadoop platform. For example, a big problem that many companies have today is how to reliably collect data from across hundreds or thousands of servers, and bring it into a central data repository. That’s a problem that we are very excited about.

Scott: Cool. We’ll keep paying attention to what comes next. Thanks for taking the time to talk today.

Amr: Thank you.

Bookmark this:

• The role of the Software Freedom Law Center in the world of free software
• Fostering voluntary compliance with the GPL
• Handling violations without putting people on the defensive
• How communities protect themselves with self-regulation
• Negotiating the urge for proprietary “secret sauce” in the marketplace
• Niche markets where open source has not had much impact

Scott Swigart: Could you take a minute to introduce yourself and tell us a little bit about what you do, as it relates to open source and free software?

James Vasile: Sure. Primarily, I work for the Software Freedom Law Center, doing legal work for people that make and support free software.

My client list includes the GPLed content management systems, like Joomla and Drupal. I go to work for WordPress as well, but I also end up doing work with organizations like GNOME, Samba, and KDE.

Just about any large free software project is going to come through the Law Center at some point. We try to give them whatever help we can, and that help varies quite a bit from client to client.

The legal challenges that free software projects face are sometimes unique, and sometimes mundane. The unique challenges are often about making sure that these projects can continue to operate as free software projects, in a world that is occasionally hostile to freedom and open source software.

More mundane legal issues also arise, because it turns out that running a medium or large free software project is an awful lot like running a small business. The main difference is that nobody gets paid. But you have a lot of the same legal issues trying to get out there and distribute your software all over the world and manage the wealth of resources that it takes to make world class software.

The other side of my work is that I help free software projects with managerial stuff, trying to help them make decisions that put them on the best footing for managing their project, creating new strategies, and getting the word out.

It turns out that we at the Law Center talk to more people in the free software world than almost anyone else. That gives us a pretty good point of view to tell them how a given path has or has not worked well in the past.

We offer that experience to people not as a legal service, but just as an ear to listen, and we offer advice where we can. In that capacity, I sit on the board of Open Source Matters and on the advisory board of GNOME. We give a lot of off-the-cuff advice to software projects.

Scott: We’ve talked to a lot of people about the different ways of building open source software, whether it’s corporate built or community built. We have also had a lot of conversations about choosing between different licenses, such as deciding between a GPL license versus an Apache license, or something like that.

One aspect of how I view your organization is as an entity involved with enforcing the GPL. Is that an accurate characterization? And what are some of the issues that people bring to you, and how do you help resolve those issues?

James: We do a lot of the GPL enforcement, at least in America, but that’s definitely not the majority of what we do. Even though we do a certain amount of enforcement, most notably for BusyBox, and Samba, I wouldn’t say that we think of ourselves as primarily GPL enforcers.

More often, we’re trying to help people to find reasons to comply. I think the GPLed CMS world is a really good example of that, because like most of the rest of the GPLed world and the free software world, enforcement is a non-starter. I mean, the vast majority of licenses are respected, because people voluntarily respect them.

Most people know that if they violate a license, the chances of them actually getting rapped on the knuckles for it, or even called out on it, are pretty low. People comply to avoid the risk of the embarrassment at having to retract some code, but more importantly, there are social norms at play that help them decide this is the right thing to do. There are structures that incentivize compliance that have nothing to do with lawsuits or lawyers.

For example, we just had a really good discussion about this in the WordPress community. It was recently decided that WordPress themes have to comply with the GPL.

Themes are extensions to WordPress that include everything from images, to CSS files, to JavaScript, to PHP code. Some of these components are reached by the GPL, which is why themes, to some degree, have to comply with the GPL.

I won’t say that this was a new development in the WordPress community, but there were a lot of theme distributors and theme publishers in the WordPress community that were not compliant with the GPL. This is a shift in strategy for WordPress, in the sense that they’ve decided they want to encourage people to comply.

Obviously, this can be controversial when people are making a living selling themes, but most of the people who decided to come under compliance were fully aware that WordPress was never going to launch a bunch of lawsuits trying to get people to comply. Rather, the reason to comply was to get along well with the rest of the community, rather than thumbing their nose at the project that is producing their livelihood.

More to the point, there are reputational benefits to complying with the GPL in some of these communities. If you work with the project, as opposed to against it, the project is happy to link to you, for example, include you in their extensions directory, their theme directory, and that kind of thing.

There are some practical benefits to complying, but for the most part, it has nothing to do with a legal analysis. Our work in GPL enforcement is not really about trying to whack people upside the head. Sometimes we do that, but for the most part, that’s the option of last resort.

Scott: A lot of times, projects grow up kind of isolated, and then they might reach critical mass, like WordPress or Drupal. On the way to becoming practically household words in the developer and IT community, I imagine that they go a learning process as they decide what kind of a cultural identity they will have.

I could see it being a fairly seismic shift in something like the WordPress community for them to decide that enforcement is important to them. That seems to fit with the notion that the goal is not to file a bunch of lawsuits, but rather to create the conditions where complying with the GPL is the logical thing to do.

As you’ve worked with different projects, have you observed a set of conditions that need to be present for compliance with the GPL to become self-enforced, in a matter of speaking?

James: Its actually extremely varied. Every project has its own culture associated with it and its own values, even. The things that different projects think are important is hugely variable, and so there is no mental checklist.

It really just comes down to looking at a project and the structures and values they have in place, as well as their history. Based on that, one can make a case to people that the license they’re using should be respected for the good of the community.

We point out the ways in which the community benefits as a whole and the way the individuals benefit within that community. Eventually people start talking to each other about how much they care about licensing.

A really good example of that is in the Joomla world, because Joomla started out as a project called Mambo, and Mambo was released by software company in Australia under the GPL. A bunch of people flocked to the product and started making changes to it. The company said, “Thank you very much for the changes, but now we’re going to close the code and release it under proprietary licenses, and basically take this code private.”

It was the GPL that protected them from having this happen, protected their investment in the code, and kept the code free. Even then, though, many people within the Mambo community did not think the GPL was that important, and they did not have a community norm of respecting the GPL.

They sort of treated the code base as quasi-open for most people and then occasionally closed when they needed it to be closed. Certainly, they weren’t requiring third-party extension developers to comply with the GPL.

The shift from that world to the current world where the license is a big part of every conversation about what to do with the project has been one of pointing out to them how much the protections of the GPL mattered in the past. That becomes a reason for maintaining those protections in the future.

But obviously with a project like WordPress, which has never had a crisis of who can access the code base and who can make derivative works of it, that sort of argument isn’t going to be available. So every project, having its own history and culture, is going to have its own reasons for caring about and wanting to respect the GPL.

Scott: I would imagine that it can be an interesting challenge if someone starts out small with a great theme, starts welding more functionality into it, and eventually starts to sell a significant amount. In a case like that, how do you go about making that initial communication?

I picture it being somewhat like music downloads. In that case, of course, the industry decided they needed to bring it under control, so they picked a bunch of people and said, “We’re going to go after you,” and then they created a culture. You are fortunate that you don’t need to take an aggressive approach like that, but how do you educate people who may not even really suspect that they are violating a license?

James: The initial contact can be very difficult, because we need to convey to people that they’re probably violating a license. That rings alarm bells for anybody, especially someone running a small business.

We want to communicate that to them, but in a non-threatening manner. The challenge is to reach them without making them sort of shut down, call their lawyers, and never talk to you again. We want to engage them to talk about ways they can come in from out of the cold and come into compliance.

Having that conversation with people and letting them know, “I’m not here to knock you about, I’m not here to extract money from you, and I’m not here to change anything you’re doing except for this one tiny thing of you agreeing to freely license your product, ” is very difficult, because people have a great lot of fear about what is going to happen if they comply.

Scott: Sure. The small business owner probably hasn’t ever talked to a lawyer except when they originally set up their articles of incorporation, if they even have done that. How do you typically play it out from there?

James: Sometimes we do it well, and I’m the first to admit that sometimes that interaction doesn’t necessarily go well. Usually by the time a case gets to me, the project has already contacted this person, so the discussion has happened outside of my view and certainly outside of my control.

Even though we never know what that is going to look like, obviously there are best practices here. It typically works best to send an initial polite email saying, “It looks like you might be violating. We should have a discussion about that. We’re not here to ask you for money, we’re not here to give you bad publicity, and we’re not here to do anything but discuss ways that we can convince you to come into compliance.”

I think that phrasing of “We would like to have the opportunity to convince you,” is probably the most important aspect, because it can help people realize that force isn’t really what’s on the table. After all, Word Press isn’t going to waste its precious resources suing a bunch of theme designers.

Scott: What’s it like for you educate the projects themselves on how to engage? I imagine that a project owner who has invested their heart and soul in building an Open Source project might be less than diplomatic in approaching someone they feel has violated the license. How do you help them be constructive?

Conversely, how do you educate something like a three-man design team that it’s in their best interest to rewire how they’re going to market? How do you educate them about the fact that doing so will ultimately benefit them?

James: When projects make the first contact, the reason it can be a little bit dicey is that sometimes the people involved truly feel wronged. When they contact the people who seem to be violating the license, some of that unhappiness often shows through as impatience and negativity.

That’s probably not the most auspicious beginning, especially since the theme designer probably doesn’t agree with that position. From the very first sentence of the email, they tend to have diverging viewpoints, and the conversation can go awry.

Projects get better at it just by doing it more often, either in tandem with my office or just by thinking about it a lot more. They can generally benefit from consulting with us before they make the effort and running the emails past us for the first couple times they do it.

Honestly, though, once they’ve done it a couple of times, anyone who regularly does compliance work gets into a rhythm about what the techniques are for getting people to actually talk to you as opposed to shutting you out.

It really is just a question of experience and reminding yourself that the person on the other end of that email has a very different point of you than you do. That’s how we address the initial contact, but what we actually say to people to convince them to comply in the theme context has a lot to with introducing them to theme designers that are complying and making money.

It turns out that the way people find themes is not very GPL aware. In the same way that designers don’t care much about the GPL, customers don’t care much about the GPL. When people grab a theme off the Internet, whether it’s GPLed or not, it’s probably available from both its authoritative official source and some FTP server or bit torrent somewhere.

And the GPL is not going to change that fact. In the same way that Joomla is not going to run around wasting money suing theme designers, theme designers aren’t going to run around wasting money suing FTP sites.

That means that the legal rights they are giving up by GPLing their themes are rights that they were probably never going to exercise in the first place. It turns out that the protections that theme designers hold onto are not worth a whole lot, and giving them up does not cost them anything. GPL compliance is a zero cost proposition in 99.9% of cases.

In the theme and extension world, that makes the decision a lot easier, and it helps people realize that they can comply with the community standards without it costing them a dime. Most people truly want to comply, and they don’t want to be in violation of the standards of the community in which they operate.

Sean Campbell: In view of the fact that most people want to follow the rules, what do you think the projects themselves can do to facilitate that? What should they be putting on their own sites to help educate people before they download source and get started building with it?

Do you think they should be doing more educating?

James: I think that, in the same way that GPL compliance is less about the license and more about the community, open source is less about the openness of the source and more about the community.

The openness of the source code creates great communities, and I think when you’re talking to potential clients, customers, downloaders, and users, the thing to lead with is not the fact that they can look at the source code, or that there is a license that they probably don’t care much about anyway.

Projects should explain to them all the benefits of the community, and why they would want to download software that has a community behind it as opposed to just a company behind it.

They should also talk about what it means to join that community. They should explain that extending the software or building on top of it happens in this community context, where there are certain norms.

I think that’s probably the best approach, because that gives people what they really want to know, which focuses on the social context, rather than legal obligations. It also helps them understand what is so great about the world of free software, which is that we are a collection of people who provide support in this mode for the software that we build.

Scott: You said something very interesting about the idea that following the GPL is essentially a zero-cost proposition. Most people getting paid to write code are working for companies that may not have a strong participatory connection to open source, and that has been even more true looking back some number of years.

In companies large and small, I think there is the perception that releasing their code under the GPL could compromise their business position. They are often concerned that they would be ceding control of their product.

That position, of course, isn’t supported by something like MySQL. No one is going to take over MySQL, because the company building it has all of this expertise around that code base, as well as enhancing it, supporting it, maintaining it, and providing services around it. And when you get down to consumer-level products, that’s even more true. My mother isn’t going to want to go search for the source code. She’ll pay $15 for a theme because it will just work.

In light of that, it was fascinating to me the way you phrased it, that 99% of the time there is no cost to using the GPL. The fear people have that if they go the GPL route, they’re out of business because they’re forced to basically give products away instead of selling them isn’t really well-founded in most cases.

That’s a message that I don’t see out there a lot. What success stories can you point to, where people have gone the GPL route and it didn’t cost them anything, and in fact may have given them benefits they would otherwise not have had?

James: I think you give some good examples. Nobody is going to buy MySQL from anybody except MySQL, at least not currently. The people within MySQL over at Sun have concentrated all the expertise in deploying MySQL and using it. It’s possible that competitors could spring up, but on top of having the expertise in deploying it, the people at MySQL and Sun have the apparatus to support it long term.

It turns out that the people who actually make the software have a huge competitive advantage over everybody else in monetizing that software. No reputable company is going to take the cut-rate deal and compromise on the quality when what they really want is the best support they can get for a reasonable price.

As the people who make the software, you are the best positioned to do other things related to it, and so you become the best source when it comes to servicing that software.

That’s the big reason that it’s a zero-cost proposition. At the end of the day, no matter who might want to go into competition with you using the GPL rights you give them, you are going to out-compete them every day of the week.

The only business they would take from you is business that you wouldn’t get anyway, because you would be pricing too high for that customer. Anyone who could afford to pay your rates would pay your rates, because they want the best that they can get, and you are the best.

Therefore, by definition, anyone who is making GPL-ed software and selling it is in the best possible market position to monetize that software, no matter what the business model ends up being. That’s definitely true both at the high end and at the consumer end for things like games, which are a highly proprietary market where very few companies allow free redistribution of their games.

It turns out that even though you don’t allow free redistribution of your games, there is free redistribution of your games. There is rampant pirating of gaming software for both console games and PC games, and despite that the PC game manufacturers, the game manufacturers continue to sell lots and lots of copies of their games.

The reason for that is that at the low end, at the retail end of things, just like you were saying, your mother doesn’t necessarily want to troll through FTP and BitTorrent sites to try to find a theme or to download the source code for the theme and try to figure out how to install it.

People want one-click convenience and easy, supported deployment of software, and companies that GPL their software are able to do that just as well as companies that don’t. In fact, sometimes they can do it better.

Whatever the mechanism by which you are making money and being “the source” for this software, none of that changes when you go GPL. We used to say that business models might have to be adjusted for people that sell GPL software or distribute GPL software, but it turns out that that’s not really the case. In most cases, you make money off your proprietary software in the exact same ways you make money off your GPL software, and we’ve seen that time and time again.

I think you’re right that we need to start putting up more examples of that and start showing people the examples of companies that have been successfully GPLing software and people that are successfully servicing GPL software that they make.

That’s probably the biggest thing we could do to alleviate the fear that people have, which is that once they GPL it, the door is open and they can never go back, and all their profits will disappear.

Scott: What is the corresponding message in the most extreme cases, where somebody feels like they have an algorithm that really is their competitive differentiator. I’m thinking of something like a gaming engine, a compression algorithm, or a market-analysis algorithm where they know they’re doing things that nobody else is doing.

I guess, what’s your message to those companies that are convinced–perhaps correctly–that they’ve got a relatively small amount of code that gives them a huge competitive differentiation?

James: First, in most cases they’re probably wrong, because most of the time, people who cook their own algorithms are not going to beat the combined wisdom of everyone else trying to solve the same problem cooperatively.

That said, it might be true that in some cases, a private algorithm could beat whatever the current state of the art is. Still, they’re probably not going to beat it for long, and moreover if what you’ve done is truly novel, there are other mechanisms for protecting the patents and whatnot that come into play.

Most of those people are probably mistaken about their unique competitive advantage, and a lot of what’s left would probably benefit from having everybody work to produce a better model or algorithm, to combine the advantages of many different services.

If you have five different companies, each with their own private competitive advantage, imagine what they could do if they got together and gave everybody the benefits of all their advantages in one package.

I think that there are still a lot of reasons to go the free software route, even at that point. At the same time, your expertise matters, and your awesome algorithm is best used by you. Even if you free it and let other people improve upon it, you are in the best position to use those improvements. You still have the competitive advantage there and it’s quite a large one.

Imagine if Google were to release the source code behind Gmail. Nobody else on the planet would be able to deploy Gmail, because none of them have access to the technology that Google has, meaning the entire stack that is so uniquely developed for them and their unique purposes.

I’m sure Gmail connects to tons and tons of applications and pieces of infrastructure that don’t exist anywhere but within Google, so even if Google were to hand me the source code to Gmail tomorrow, it would be almost no good to me at all.

Scott: We see that sort of effect with the company I believe is called Aquea that’s wrapped around Drupal. Likewise, Cloud Air is wrapped around Hudu. They would have to do really bad by the project for a long time before somebody could fork it and get the community of users to move over to them.

It sort of happened with Netscape and Firefox. You can see examples, and you mentioned Mambo becoming Joomla.

James: Even in the case of Mambo and Joomla, when the company tried to close source Mambo, that wasn’t even enough to cause a fork. It was a couple years later that the fork happened, after the community tried very hard for a long time to work with this company. And the company tried hard to work with the community as well, and finally it wasn’t working to anyone’s satisfaction, so there was a fork.

Experience shows us that every time people are in a situation where forking might be good, they choose not to do it, because it’s tremendously costly to try to split a community in half and figure out who’s going to do what.

Scott: You mentioned games as being an area that’s very closed and proprietary. Search is another of those areas, as are financial analysis, oil and gas, seismic analysis, and others.

Are you seeing any areas where there is starting to become a real credible GPL-licensed option that there is a lot of community effort around that may be a game changer a year or two or three from now in some of these areas that have traditionally been very dominated by closed source?

James: I wish I could say that I did see a lot of these imminent game changes attracting large communities, but unfortunately I haven’t. The thing about free software currently is that it usually starts with somebody scratching an itch or somebody trying to solve a fairly widespread problem.

In these niche markets where there isn’t a lot of free software adherence already, there aren’t a lot of coders sitting around directly going to benefit from a problem being solved in that area, and it breaks down pretty quickly. One place we see that quite clearly is in the small business application market.

Think about medical billing, for example. Medical offices spend tens of thousands of dollars on software that is by any measure extremely rudimentary, and all it does is help them manage appointments and then submit invoices to insurance companies.

That software is ripe for being knocked off by a free software project, and yet I’m not aware of anybody that has cobbled together more than one or two programmers committed to working on it, giving it any significant amount of attention or time.

It turns out that the free software community has been relatively bad at supporting these niche areas. There are business models that would allow easy entry into these areas and make it possible for free software developers to make the software and make a good living supporting it, and yet still nobody is taking that opportunity.

That failure is one of the things that I’m curious about, and one that I’m seeking solutions to over time.

Scott: This has been a great conversation. One of the things we usually do at the end is to ask whether there’s anything interesting that we didn’t ask about.

James: I think the most interesting thing about the work that I do is really the community-building aspect of it. I touched on that earlier when I said that it’s not about the legal structures or the legal norms, but rather about the social and community norms.

That’s true across the board, and one of the things that I see on project after project is people confusing the value of what they do as license-based as opposed to community-based.

You see people with religious allegiances to certain licenses or even people denigrating other licenses or worrying about who’s using what license in other projects and things like that. What they miss is that the value of the code they’re writing isn’t just based on what license it’s under.

Scott: Thank you very much for taking the time to chat with us.

James: Thank you.

Bookmark this:

• Joomla and its relationship to website owners
• The difficulty of creating success metrics for free software
• The relationship between the core Joomla environment and extensions
• Cloud computing and platform independence
• Transitioning the web to new platforms and the semantic web
• The future of rich applications in a web browser

Scott Swigart: To start, could you introduce yourself and tell us a bit about your relationship to Joomla?

Louis Landry: Sure. I first used the predecessor to Joomla, Mambo, in 2001 or 2002. I was a computer science student at the time, and I was building a website for my friend. I sort of fell in love with the interface, and honestly, I was shocked to see some of the things that they had pulled off on the web.

Over time, I continued to build more and more sites and get more and more involved. At the time that Joomla became Joomla, I was in contact with several of the key members who started a fork to do some development patches and things into the core.

After I was asked to join the core team, I eventually served as project manager, communications team leader, foundation working group coordinator, development coordinator, and pretty much every aspect of the project. I am now development coordinator, working on future research and development, infrastructure, and things like that.

Scott: Talk a little bit about Joomla itself, the strengths of its architecture, and so on.

Louis: Joomla’s an open source content management system, designed to help you build and maintain your web presence. It doesn’t take the same approach as some of the older, larger-scaled content managers like SharePoint do.

It’s not designed to be a full-on enterprise thing. That doesn’t mean it can’t be used in enterprises, but it’s more of a presentation and content organization system than some of the more document-centric systems. Our main target market has been websites for small businesses and other small organizations.

The last reasonable numbers we saw showed maybe one or two million websites out there running Joomla. Most of those are individuals or small organizations putting up sites to share a voice related to some things they care about.

Joomla itself is really a platform with something like 4,000 extensions. It’s much more of a product based system than other applications like Drupal, where all of the extensions tend to be much more tightly integrated with the core system. With Joomla, you find a lot more third-party developers building product-based things. It’s much more packaged up, and you just sort of upload a Zip file and everything installs itself.

Scott: How is the Joomla organization structured? How do you make money, or do you?

Louis: The non-profit organization that holds assets for the Joomla project is called Open Source Matters. It’s incorporated in New York, and it holds the trademarks, copyrights, and those sorts of things. Most of their income comes from Google Ads, I believe.

It’s not really a money-making corporation. I am an owner in a company called JXtended, which sells third-party components, extensions to Joomla, and that sort of pays my bills, along with consulting work that I do around Joomla. That tends to be the trend with people around the Joomla project, if they’re making money here.

Scott: That makes sense. I would imagine that a lot of people are looking for certain functionality, and they may not even know that Joomla’s the solution. Then a consultant can come in and relatively quickly have a site up that meets their needs and just so happens to be based on Joomla.

Louis: That’s actually the case more often than not. People will hire a firm to put together a website for them, and they don’t really realize that Joomla is the platform on which it’s being built. A very large proportion of the people on our support forums and engaging in the Joomla project are small web design shops that are pumping out websites for different clients.

We’ve had more than one case of a client seeing the Joomla offline screen and thinking that we somehow hacked their site. Something went wrong with their Mind Skill server, and when the little screen comes up and says, “This site’s offline,” and it’s got the Joomla logo, they’ll email us and say, “Why did you hack my site?”

[laughter]

Then we have to explain to them, “No, really that’s not what happened.” So it’s funny, but it’s happened quite a few times. That’s indicative of the fact that a lot of people using Joomla don’t really think of it in those terms, since the site was put together for them, and Joomla is transparent to them.

Scott: That’s an interesting aspect of that space. There are obviously a lot of people who need a web presence but are not technical.

You’ve also mentioned the really rich ecosystem, for lack of a better term, around Joomla: 4,000 plug ins, consultants, developers, web design firms, and so on. One to two million sites must rank as a highly successful open source project.

Louis: It’s a pretty sizeable chunk. I think since July, we’ve had three million downloads of Joomla off of our sites. I’d have to double-check the numbers, but I think in the last two years, it’s in the range of 14 or 15 million. It’s been pretty astonishing to see the downloads crank up.

That’s also just the English version. There are plenty of versions that are distributed by our language partners or translation partners all over the world in something like 30 or 35 languages.

Scott: That’s an interesting difference between success metrics with proprietary software as opposed to open source. With proprietary solutions, you can point to an exact number of licenses sold, whereas with open source projects, you often have to rely on the number of downloads. That makes it somewhat more difficult to get an accurate picture of adoption.

Louis: That’s true. It’s difficult to gauge the ratio of deployed sites versus the number of download packages, so it’s really hard for us to know what the actual market share is.

There were some numbers at one point from a company that I believe was called Netcraft, and some people in our community have built crawlers to try to figure out market penetration. It’s been interesting to see these various numbers, and they vary widely.

Scott: It kind of goes against the ethos of open source, but open source projects could easily embed something in them that by default makes the thing phone home, when it’s up and running.

Louis: There’s been a lot of talk about that over the years. A few open source projects have tried it, but those efforts tend to backfire, because a lot of users don’t like it.

We’ve talked about the concept of anonymous usage statistics and those sorts of things several times, but it’s just never happened. I am actually not sure it would be a good idea, simply on the basis of the fact that I have no idea what sort of traffic that would generate for the server collecting the data.

Scott: Right; a little Google App Engine app could just shove the statistics into a big table data store.

Louis: Yeah, let Google suffer. [laughs]

It’s something that we’ve talked about, like having some sort of optional switch upon install, but even there you run into trouble, because one of the biggest ways Joomla is installed is via cPanel and Fantastico, which are server scripts to auto install web applications.

They don’t even run though the Joomla installer, so we wouldn’t see it. That’s another example of how our download statistics would be completely irrelevant, because that doesn’t get downloaded through us–it gets deployed through a completely separate system.

Scott: On the topic of the cPanel and Fantastico installers, talk a little bit about how Joomla updates. There’s always a tension between a really vibrant ecosystem of plug ins and the necessity of keeping the main project moving forward.

Louis: That’s been a real area of balance for us as well. Anyone who follows Joomla knows that we don’t release very often, as far as feature incremental releases, and that’s not always by design. Version 1.5 came out, I think, two years after 1.0, and 1.6 is set to come out probably toward the end of this year, which is about two years or so, as well.

It moves rather slowly, but the wealth of functionality created through the third-party extensions allows us to do that. And because it’s almost entirely volunteer driven, and we’ve been going through growing pains since this is still a relatively new project, we’re still trying to find our footing in a lot of different areas.

We don’t really have an update mechanism in the core quite yet, although that is something that is being actively worked on for Version 1.6. As far as extensions are concerned, there isn’t a real updating mechanism for that either.

Historically, it’s been either a case of uninstall and install the new version, or else install a new version on top of it. With PHP, it’s generally as simple as FTP-ing files up, if you want to do it manually. It has not been painless, but it has also not been a huge problem for us.

Scott: I imagine that the 4,000 or so plug ins are in various different states at any given time, from having been abandoned to having multiple people working on them full time, and everything in between. It’s kind of like when you upgrade WordPress, you just kind of cross you fingers and push the button, and hope all your plug ins work afterwards.

Louis: [laughs] That’s the thing with the slower release cycle. We put maintenance releases out to fix bugs and things like that pretty regularly, but those updates don’t change things in such a way that any of the third-party stuff would break, really.

There have been a couple of cases where we’ve had to change things, but we got out in front of it pretty well, and it didn’t really cause a whole lot of issues. Still, it’s true that when you’ve got a large third-party developer community building external extensions, there’s an awful lot of inertia.

If we go in and change a system in the core that’s going to have a cascade effect on how everybody uses the API, or the systems, or something like that, that’s a lot of ramp up. What we saw with our huge changes in 1.0 to 1.5 was that it took a while for the third-party development community to catch up.

In a lot of cases, they did abandon things, and other people picked it up. That’s part of the beauty of the open source community, right? Somebody says “Oh, well I’m not going to bother with this,” and somebody else comes behind them and says “Oh, I’ll do it.”

Scott: Or, in some cases, there are just a lot of options. When something gets abandoned, there may be four other projects that do something pretty similar.

Louis: That’s certainly true. For example, in just the category on our extensions directory for adding comments to articles, there are pages of different options. Some are commercial, some are not, they have different features, and so on. The richness of choice is really great.

Scott: That brings up an interesting point. I’ve talked to people in corporate environments who are considering using open source software about their methodology for evaluating the safety of a certain project.

They will look at all those stats to see if it’s being actively maintained. They will see if the contributions are just coming from a few people, or from a lot of people. They often have a specific set of criteria they use to judge whether they want to make a bet on it.

Do you have any advice for people as they look at those several pages of options for similar functionality? Obviously, the one that shows up at the top of the list isn’t always the safest choice.

Louis: That’s a great note. I find that this is one of the areas where the commercial side of open source really flourishes.

If I were going out to build somebody’s website, and I was presented with a commercial choice and a non-commercial one where they both met my needs, I would probably choose the commercial one and pay for it. That would give me higher confidence that somebody would be backing it.

That’s not to say that I don’t appreciate non-commercial open source software, because obviously I do. Still, in a mission-critical situation, that’s likely the path I would take. That said, I also tend to look at how active the people are who are running the project.

If I’m looking at those same two options, and the commercial people are nowhere to be found but the people running the non-commercial project are very active, then that tells me a lot. I feel better about their commitment to the platform, as well as their knowledge of how everything works.

I know that they’re more able to integrate certain things. They’d probably be better suited to finding answers to problems, if I ran across them, and that sort of thing.

I think there’s a balance to be made there. I really like to be able to know that whoever I’m getting my software from is active in the open-source community.

Scott: What’s the Windows story for Joomla? Is it common to run it on Windows? Is it one of those situations where some people do development on Windows but they pretty much always deploy on Linux?

Louis: I think that since the majority of Internet servers are Linux, it’s probably safe to say that the majority of Joomla installs are on Linux. And since the majority of end users use Windows, it’s probably safe to say that most people that test it and build up sites and all those sort of things are doing it on Windows.

There are certainly people using Joomla on Windows servers. We officially support Apache, and we don’t support IIS. We know it generally works on IIS, but there have been some issues. We don’t have access to those servers and the software on a reliable basis to catch all of the problems that come up.

We’re actually in talks with Microsoft now to figure out ways in which we can advance our Windows support and make sure that we have the tools that we need so that everybody can more officially assert that Joomla works in a Windows environment.

Scott: What’s in that for you? Do you feel like that will open up a significant new population to becoming Joomla users?

Louis: I’m sure it does, even though on a personal level, I don’t really care that much because I don’t use Windows. I’m not one of those hate Microsoft guys, but I just use Mac OS and Ubuntu.

Still, our mission is to provide a flexible platform for digital publishing online. And what matters there is that I would like it to be available for as many platforms, OSs, and databases as possible.

Whatever we can do to make it easier for people to deploy Joomla, within reason, is something that we will want to pursue.

Scott: Cloud computing is obviously very fashionable at the moment. What interest have you seen around running Joomla on a cloud provider? Is there any work or any thinking about enhancing Joomla to take advantage of some of the attributes of the cloud, like being able to dynamically scale up and scale down and that kind of stuff?

Louis: There are a lot of people in the Joomla sphere looking into playing with the cloud, and there are a few companies I know of in the Joomla community who are actively building things on the cloud and trying to provide integrations there.

The EC2 thing at Amazon is interesting to me, and I’m also very interested in the Eucalyptus project that Ubuntu is starting with, I think, this next version of Ubuntu server. I’ve been talking with some friends in the FreeBSD project about cloud solutions and what a path might be, as well as a bunch of other places.

I think one of the things that I will focus my research on for the next generation of Joomla will be ways we can restructure Joomla from an architectural standpoint to better live in that space and be able to really reap the benefits of the cloud.

Scott: You mentioned Eucalyptus. I’ve heard that name, but I don’t know much about it. Can you talk about it?

Louis: I don’t know an awful lot of the technical details behind it, but Eucalyptus is an open source project that Ubuntu is adopting as sort of their cloud platform. It’s got a bunch of functions, such as the ability to do node requisitioning and various things, and I believe that it’s the software that would tie the servers in the cloud together.

Scott: What I’ve heard, if it’s the same thing, is that it also exposes an API that’s compatible with Amazon’s API. So even though you’re not running on Amazon’s infrastructure, I could write code that could spin up and down instances over in Amazon, but I could point that same code at something Eucalyptus-based and it would do effectively the same thing, even though the whole underlying virtual infrastructure is completely different.

Louis: That’s certainly an aspect of it, and that’s one of the reasons why it was really interesting to me. I also am really excited about the stuff I’m hearing coming out of the Ubuntu server crowd.

Scott: Talk a little bit about that, because it’s not something I’m aware of. In the hosting space, I think of CentOS as being really popular.

Louis: It certainly is. CentOS and Red Hat are probably the standard bearers. Ubuntu’s appeal really stems from being what I chose for my desktop OS. I like the way the file system and the configuration stuff are structured, I like the toolkits, and I like a lot of the things that Ubuntu is doing across that space.

The fact that they’re starting to focus on the server is interesting to me, and I really like that they’re thinking about cloud and moving into that direction, because it looks like it’s one of the focuses that that whole community is taking with the server distribution.

Scott: One aspect of that space that really stands out to me, and that I can’t really figure out, is that one of the earliest cloud providers was Google App Engine, and when it came out, there was a lot of excitement around it. And then people were kind of turned off because they didn’t want to be limited to Python, and then they added support for Java.

It seems like the most obvious thing for Google to support would be PHP, and the second they came out with Java support, some guy ran off and started a project that I can’t think of the name of at the moment, but it lets you run PHP on top of Java.

So this guy said, ” I’m going to port WordPress to run on top of PHP, on top of Java, on top of Google App Engine, using their BigTable, as kind of a science experiment.” He ran off and did it, and it works, but obviously, it’s probably not the stack you really want to build on with something real.

When I look at that, I think that if they supported PHP relatively quickly, a lot of PHP projects–WordPress and maybe Joomla and Drupal–would probably be supporting some version that could use a BigTable data store and run on Google App Engine.

Louis: I’m actually not certain that that’s Google’s vision of what App Engine is about. They’re certainly not a PHP shop in any way. I’ve got several friends that work at Google, and I’ve never heard of them using PHP in anything, unlike Yahoo!, which is a “PHP shop.” They do a lot of PHP stuff.

Google is a Python shop. Yahoo! hired Rasmus, Google hired the Python guys, and that’s the direction they took. I think when they launched it, as is typical for Google, it was closed. You had to be invited to be able to get into the App Engine experience.

I don’t think they were certain about their ability to scale it out if adoption got too high, and the ubiquity of PHP and Drupal and Joomla and WordPress might actually have been a negative for them. I’m not certain that signing on to host all those apps would really be to their benefit.

I would almost think that part of their rationale there was wanting to raise awareness around Python and maybe get some clever people working on Python to see who those people are. Then they could collaborate with them, or hire them, or at least get a better feel for what’s going on in the Python community and expand upon it.

Scott: To turn a bit toward the future, you mentioned that cloud is an area that you guys are thinking about. How does Joomla fit in there? How does it play on this new computing paradigm? What else do you feel are the big opportunities on the roadmap for some deep thinking and important work?

Louis: The obvious thought is the semantic web, which everybody is thinking about. I feel that we in the open source CMS community can play an important role in terms of providing new ways by which information can be organized.

I think the semantic web is a place where a relatively small amount of work could have a very large impact. Still, that focus is difficult, in the sense that I’m really more focused within the Joomla community on the next generation, not necessary iterative versions that are coming out.

I’m trying to rethink the architecture. Joomla as it exists now is based on the same principal concepts as Mambo in 2000, namely being centered around software designed to run on a $6 a month hosting account on some offshore server.

The concept around one site, one instance is still a very serious bread and butter piece of the market for us, but I also want to be thinking about ways in which you can scale out multiple instances running the same software. That ability will position us to fall much more inline with cloud and enterprise deployment schemes.

I want to think about how we can make data more portable, so that you can transport things across instances or just have protocols to push in and pull out data, as well as incorporating the ideas of the semantic web.

I also think a lot about this concept of the rich Internet, like Adobe Air and various other technologies like the high-powered JavaScript engines that are starting to evolve thanks to Google.

It would be really incredible to bring out that rich Internet environment and rich interfaces to make it easier to build a more interactive experience. Those are the things that I have been thinking about.

Scott: This has been a great interview. Do you have any closing thoughts from your end?

Louis: I think we’re really just growing accustomed to this whole new phase of the web, and I’m looking forward to seeing where it evolves. I don’t think that in five years it’s going to look at all how it looks today, and I don’t think the experiences are going to be the same.

I’m really excited to be a part of how this whole thing evolves, including what Microsoft and Google are working on with their Web OS initiatives. It could really change the ways people think about their computers and connectivity. I find all of these things converging and I can’t wait to see what happens.

Scott: I remember one guy saying basically that, until you can run Quake in a web browser, Google is not done. [laughs]

Louis: Actually, you can. Google released an experimental thing called Native Client. It’s not for public consumption yet, but you can compile Windows apps or whatever into this native client code, and then it will run in a browser with a plug in.

So it’s kind of like a virtual machine. It’s interesting what they are doing with that, and I’m very curious to see things like the plug in that they have built enable OpenGL in the browser via JavaScript. I’m curious to see how they integrate that into future stuff.

Their example is quick. You can click in the browser…

Scott: It sounds almost like an ActiveX plug in.

Louis: I think it probably is very similar to that concept, and hopefully they can find a way to make it more secure than that.

Scott: That’s one approach: running a whole different something in the browser, which was kind of the Flash and the SilverLight approach.

Louis: We would much more like to see it work in JavaScript, though.

Scott: It seems like there is also an awful lot of work going that route. These are just things that are mainstream use cases for HTML and JavaScript, and they’re very fast. It’s just a whole new set of scenarios.

Louis: The ability to access the machine-level function-called things like that within the browser would be just intense, but also intensively dangerous. I’m very curious about how we are going to resolve that, because at some point it is going to have to get resolved.

You’re going to have to be able to access the file system and things like that, and I just don’t know how we are going to do it.

Scott: That seems like a good place to wrap up. Thanks for taking the time to talk today.

Louis: Thank you.

Bookmark this:

• How development on the Linux kernel has changed in the past several years
• Drawing in the larger community with the Linux Driver Project
• The breadth of uses for and contributors to the kernel
• The future trajectory of Linux, including functional balance with hardware
• Overcoming the challenges of maintaining universal hardware device support
• The range of change and growth in the kernel

Scott Swigart: Can you please introduce yourself and your association with Linux?

Greg Kroah-Hartman: Sure. I’m a Novell fellow, and I’m working on the Linux kernel with the SUSE Labs division at Novell.

I’ve been doing Linux kernel work for more than 10 years now. I’m a maintainer of several different subsystems, including USB, the driver core, and some other minor things. I’m also responsible for releasing the stable Linux kernels. I used to be a maintainer of a lot of other driver subsystems and such over the years, but I’ve thankfully handed them off to other people.

Scott: Let me start with a really big-picture question, since you’ve been involved in Linux for a long time. Obviously, there’s still lots of stuff happening on the mailing list, but fundamentally, the way code makes it into the kernel hasn’t really changed, and yet the scale of development is much larger than it was, say, five years ago.

How have you seen Linux kernel development change to handle the scale of the effort that’s going on today?

Greg: I’ve been tracking this for over five years now, and we’ve gotten a lot better. There are a lot more people doing the work.

To give an example, for the 2.5 to 2.6 kernel development series, which took about two years, the top 30 people did 80 percent of the work. Now, the top 30 people do 30 percent of the work. The sheer number of developers has also increased. We were running a couple hundred developers, and now we’re running a couple thousand.

Scott: How does that get managed? Are there more layers in the hierarchy?

Greg: We’ve invented some tools to help us out a lot. It used to be that Linus was the only one who could commit anything, or who would accept anything. It was all done through email.

And then BitKeeper was developed, and a number of us started using that. It was very nice, because we had a much faster feedback loop with Linus. I could have him pull 50 patches, see that he pulled them, and then I could hit him with another 50 patches, if I wanted to, within a day. Before, we had a lag of a couple of weeks.

Then, when we couldn’t use BitKeeper anymore, Linus wrote Git, and then that just increased it even more, because everybody can use Git. Our development model increased its capacity, and part of that was that we started trusting more people. Now we have subsystem maintainers and such.

As I said, I maintain the subsystems such as USB, and I have people who I trust enough that if they send me a patch, I’ll take it, no questions asked. Because the most important thing is I know that they will still be around in case there’s a problem with it. [laughs]

And then I send stuff off to Linus. So, Linus trusts 10 to 15 people, and I trust 10 to 15 people. And I’m one of the subsystem maintainers. So, it’s a big, giant web of trust helping this go on.

We’ve developed some procedures that help guide how and when we do merges and releases. And we’ve a very regular release schedule: every three months, we do a release. We have a two-week merge window, and all those patches in that merge window have to have been tested. We’ve gotten the development process down really well over the past four years.

We’re also increasing the rate of change in our development. The same amount of work one of the top 10 developers did last year wouldn’t have even made it into the top 20 this year. Our individual developers have got the work flow down, so we can actually contribute more, to an extent that’s amazing.

Scott: I understand that one of the things you’re involved with is the Linux Driver Project. It seems like there was a certain tension between Linux and certain hardware vendors, in the sense that, for a long time, hardware vendors had been really used to not shipping driver code.

They were used to shipping for Windows, and they didn’t have to be involved with drivers. They just provided an installer. And then, as Linux started to become a lot more popular in the server space and make some inroads in the desktop space, they wanted access to that market.

There was kind of a culture shift in a lot of companies to realize how to provide things like drivers in a way that matched with what the Linux community expected. Take the example of some hypothetical Acme hardware vendor that wants to submit a driver but has never done it before. What does the process look like?

Greg: First, people need to realize that our driver model is different than other operating systems, because all our drivers ship with the kernel. The license requires our drivers to be open, so everything is in the main kernel tree.

Because of our huge rate of change, they pretty much have to be in the kernel tree. Otherwise, keeping a driver outside the kernel is technically a very difficult thing to do, because our internal kernel APIs change very, very rapidly.

In other operating systems, APIs change more slowly, since their development process is comparatively very slow. So, it makes sense, from a technical standpoint and just to save money, to get your code into the kernel. It is maintained that way, and when changes happen within the kernel, your driver code will be fixed for you. If I have to change an API, I change it everywhere.

That approach also lets us see commonality. If see that a driver’s doing the same thing as another driver, I might merge them together in the shared core. That way, everybody benefits. The new driver gets smaller and easier to maintain.

That’s happened a lot. We’ve merged a lot of drivers from a lot of different companies, and everybody’s happy about that. The companies are happy, the users are happy with the smaller code base, and it’s easier to maintain.

The Linux Driver Project started out because of the perception that Linux doesn’t support many devices. It turns out that Linux supports all devices out there. There’s really nothing manufactured today that Linux doesn’t support, in a major consumer market. There are some one-offs, like some small video-capture devices I know of that we don’t have support for, but people are actually working on those.

The initial goal of the Linux Driver Project was to remove all barriers that could possibly be there, which were mostly managerial. To attain that goal, we said that we will write and maintain any driver for free for any company.

It turned out that not very many companies really needed that. A few niche markets needed it, and we’re writing some drivers for them, but most companies had existing drivers floating around inside their company. Their challenge was typically that they needed to get it into the kernel tree, and they didn’t know how.

So, the Linux Driver Project, over the past couple years, has just been a big educational thing. I work with the companies and show them how to get the code in. I maintain the code, massage it and get it cleaned up, and then merge it into the kernel tree. That’s been the majority of the work we’ve been doing over the past two years.

We’ve also now codified a few ways that we can play in the kernel. We can accept code into the kernel that doesn’t meet our normal standards into a staging area. Everybody can clean it up there, and then the code graduates into the real stuff. That’s worked out really well.

Scott: That makes good sense, because someone coming in from the outside isn’t necessarily going to know everything from coding standards to the best practices for doing things the right way.

And so what you’re saying is that you’ve got a place to get that code in, where people who know the right way to do things can look for things that can be factored out of drivers, drivers that can be merged, and that kind of stuff.

It doesn’t force every hardware vendor or everybody who needs to get a driver into the kernel to be a Linux driver development expert. Do I have that right?

Greg: Yeah, although we do have lots of documentation now. We have free books on how to write Linux drivers, and we have documentation of things like our coding style and how to submit code. If people don’t know where it is, we’ll point them in the right direction. In case anyone’s curious, it all starts in a file called HOWTO in the kernel, so they can start there.

Even though we do have those education tools, though, we will also work with companies that want it. We have lots of people who have done this work before and who will do it for you.

It’s a great place for people wanting to get involved in the kernel to start out, because they can just run a script to find some errors and go fix them. We have people that have started off just wanting to get involved who have ended up maintaining whole drivers and subsystems over time.

Still, in the end, my goal is actually to work with engineers from those companies and have them maintain and own it, so they become full-fledged members of the kernel community. That’s the only way we’re going to grow, and it’s working. The number of companies involved in the kernel has grown year over year.

Companies want to get the most value out of Linux, so I counsel them that they should drive the development of their driver and of Linux as a whole in the direction that they think makes the most sense. If they rely on Linux and feel that Linux is going to be part of their business, I think they should become involved so they can help change it for the better.

Scott: That makes sense. In a different area, I saw somewhere that around half of the Linux code that’s being contributed nowadays is driver code. Do I have that right?

Greg: That’s the percentage. Like I said, everything’s in the kernel itself. We’re at something like six or seven million lines of code, and over 50 percent of those lines of code are drivers.

I think 30 percent is architecture-specific stuff, for things like processors and networking. The core kernel is like five percent of the overall code. Those numbers have stayed pretty much the same over the past four or five years.

We change something like 5,000 lines a day, which is just scary. Fifty percent of that change will be in the drivers, and five percent will be in the core kernel. In other words, the kernel is being modified everywhere at that rate of change.

Scott: Do I understand correctly that the core kernel is memory management, process scheduling, and those kinds of fundamental things that an OS has to do?

Greg: Exactly. Basic system calls, memory management, scheduling, and inter-process communications.

And the only reason we’re changing is because people want that change. It’s not like we’re sitting there and going, “Hey! Let’s rewrite the scheduler again!”

[laughter]

We’re doing this stuff because we have to in order to survive, because people want it and because people need it. We do it for fun, but we don’t gratuitously change things. A big part of what drives that change is that what Linux is being used for is evolving. We’re the only operating system in something like 85 percent of the world’s top 500 supercomputers today, and we’re also in the number-one-selling phone for the past year, by quantity.

It’s the same exact kernel, and the same exact code base, which is pretty amazing. Nobody’s ever created something like this before. And we’re doing it in a way that doesn’t follow the traditional software design methodologies, which is fun.

I go and talk at a lot of colleges, and they’re changing their education system based on how the kernel has changed development models for large-scale systems.

Scott: Like you mentioned, the kernel runs on some of the smallest devices, as well as some of the largest. What makes that work is a lot of stuff that is or isn’t included, whether you’ve got fairly bare-metal, embedded systems, or whether you’ve got desktops and music players, or whether you’ve got systems that are optimized for number crunching.

The kernel is an interesting thing. When you talk to people who aren’t really in the Linux space, they often don’t make a big distinction between the Linux kernel and a Linux distro. There’s a huge distinction though, right? The kernel is a fairly small piece of a distro.

Greg: Sure, but consider Android, which threw away everything from Linux except the kernel, and they built something totally new on top of it. That’s a great proof point that the Linux kernel itself has to be really, really flexible to let people do something like that. It still meets the needs of a very big market, which is pretty funny to watch. From an engineering standpoint it was a pretty neat hack.

Scott: Give us some sense of the kinds of people who are contributing to the Linux kernel. Obviously, there are hardware vendors. The distros out there are contributing–Red Hat and Novell, for instance, are big contributors. Give me a sense of the categories of contributors working on the kernel.

Greg: You’ve got a pretty good short list there. Some distros contribute; some do not, and the same is true of hardware companies. Intel has come on like crazy this past couple of years, and they’re now one of the major contributors to Linux. Consulting groups and educational institutions play a big role as well.

There are also a lot of companies that just modify code for one-off little things, making support better in their hardware. Linux runs on yahts for a lot of automatic pilot steering things, and we’ve had some contributions from there.

More than five or ten percent of the US’s power is generated on turbines that are controlled by systems based on Linux, and those guys contribute some changes they need. It’s also interesting that 20-25 percent of all of our contributions by quantity are done by people that have no affiliation with anything.

Scott: The proverbial guy in his basement or dorm room or whatever.

Greg: Yeah. They needed something fixed, so they did it and moved on. The quantity there is still large, but on the other hand, 75 to 80 percent is done by people who are getting paid to do it.

Scott: What do you feel like is driving change? You mentioned Intel, and Linux seems kind of tailor made for a chip company, because it provides a great opportunity to put new things in a processor or a chip set. Contributing to Linux lets them build support for those kinds of things and to see how they are behaving in a real production operating system.

I can kind of see that new hardware architectures and capabilities drive a certain amount of change in the Linux kernel. I can just see the creation of new peripherals driving a certain amount of change. What other types of change drive the Linux kernel forward?

Greg: Well, those two are major. New hardware accounts for the majority of our changes. We get more processors with more and more cores. We had to do a lot of work to make, 8-, 16-, and 32-way machines run really well, and now we’re running 8000 processor machines really well. The scalability is there, and I actually know of people who have booted larger ones, but we’re not allowed to talk about it.

USB 3.0 was sponsored by Intel, and it was shown on Linux first–the first implementation of any operating system. Intel gets their hardware working well fast, and working with the Linux community lets them get it out to developers and other people who are making devices fast. The fact that they can do it for Linux much faster than they can for other operating systems is a huge driving factor for what we need to do for things like 10 Gigabit Ethernet or 40 Gigabit Ethernet.

Some of those speeds have required major changes to the way we do networking and the way we handle processes or data flowing through the kernel. We also have to accommodate workload issues, like real-time workloads, which required big changes to the scheduler and the way we do locking and the way we do other things merged in.

Note that those are all requirements driven by people who are using Linux. The real-time guys want to use operating systems like Linux for their stuff. It also drives Wall Street and the Chicago Mercantile Exchange and the Tokyo Exchange and the London Stock Market.

Scott: What do you see on the horizon, in terms of hardware use cases that aren’t quite mainstream or available yet but that are going to drive some additional kind of significant change all the way down to the level of the kernel?

Greg: 40 Gigabit Ethernet is going to need some changes, and the networking guys are actually working on that already. There are also going to be more and more cores, better real time operation, and USB 3.0. The USB 3.0 devices aren’t out yet, so once we start seeing more of those, we’ll start seeing optimizing things there. SSDs are great; they’re growing fast.

People seem to forget that changes they want to the kernel that it seems like no one else will want very often end up being valuable to a lot of people. For example, maybe the embedded group makes some power management changes, and it turns out that the server guys really want that, too. Then the server guys want better scalability, and it turns out that the embedded guys love those scalability changes, because they’re running dual-core ARM systems now.

Part of what’s helped us make things work is that it turns out that a lot of changes benefit everybody. We’ve become such a flexible operating system by requiring changes to work for everything, and it’s made everything work even better.

There are lots of power management changes in the pipeline to be reworked that help address a lot of the issues the embedded guys have come up with that servers traditionally don’t have. In servers, you can’t shut off that many discrete components, whereas in embedded, you can pretty much shut off every single discrete component in your system, depending how much you want to do with it.

Scott: I remember when we talked to the OLPC guys, one of their requirements was to be able to go to sleep, but not lose the network connection.

Greg: Mesh networking, right?

Scott: Yes. And they also needed super-aggressive power management, because they wanted to get the most battery life possible, because power is such a big issue in developing countries. If I remember correctly, they were actually putting the processor in a sleep state between keystrokes.

Greg: That’s right; they were, but it turns out that the hardware guys can do it better. If you look at more modern processors, the hardware is putting itself to sleep faster. In modern PCs and these new netbooks, the OS really doesn’t have control of the power management stuff anymore, because the hardware can do it better.

It turns out it’s better to run as fast as possible to get your work done, then go to sleep, than to run your CPU at medium speed. The hardware guys realize this, and they know they can handle their workloads better than the operating system can.

Scott: I don’t want to mischaracterize it, and I am intentionally overstating this to some extent, but when you talk to people at Intel, they always say, “Well, that should be in silicon.” It’s kind of like there’s this drive to put more and more and more in silicon, but at the same time, obviously software is growing at a geometric rate.

Talk a little bit about that balance between what ought to be in hardware and what ought to be in software, as well as the notion that something that ought to be in one place today should maybe be in the other a year or two from now.

Greg: The hardware guys are always like, “Oh, I can solve that in hardware. Let me do that. You software guys aren’t ever going to get around to doing this.” And part of that is true.

If you look at the software adoption rates of some operating systems, it’s very, very slow. So, the fact that they make a change to the hardware, and they can save power today in the new processor, means that overall, the amount of power consumption in the world goes down.

People run old versions of Linux and other operating systems very frequently, so there’s sometimes an advantage to solving a problem in both places. They can add it to the hardware, and then they also add it to the software.

In the kernel, we know we typically can’t change user space applications, so we have to make changes in order to accommodate how they want to do things, even if we feel that they’ve done it the wrong way. So, there’s also a tension there.

Some companies actually talk to the engineers really, really well. Intel and IBM are very good at this. They sit down, and they talk to the kernel people once a year. “Hey, this is what we’re thinking of doing. How should we do this? How are we going to implement this?” And they get feedback from us directly. And that’s turned out to be a very, very valuable feedback loop.

It often works out very well in terms of how chips are designed, and how the operating system will take advantage of that. Hopefully, we’re all talking to all the right people, and everybody’s interacting well.

Scott: Let me spin back to something you said very early on, which is that basically that all hardware is supported in the Linux kernel, even though there seems to be the perception that not all the hardware works.

Certainly, one of the things that gets brought up is that if I buy a new device, it’ll come with a driver disk that probably won’t have Linux drivers on it. Talk a little bit about where do you feel like this disconnect comes from between the people working on the drivers in the Linux kernel and those incorrect user perceptions, even from people who are fairly experienced with Linux.

Greg: That perception has existed for many, many years. Back in 2006, I gave a big keynote showing how it was all wrong. Everything is supported, but, yes, that complaint comes up a lot. I joke with Jim Zemlin of the Linux Foundation about this issue all the time. If he publicly says, “We’re still working on drivers,” I always say, “No, we’re done.”

[laughter]

So, there’s a disconnect there. Part of the issue is that we support more devices than any other operating system ever has. That’s a fact that’s been verified by other companies. The problem is that people only care about the devices that they have. Therefore, if your device doesn’t work for some reason, you don’t care how many thousands of other devices out there work.

When I started the Linux driver project, I’d been hearing this from all the major companies that were shipping Linux and cared about Linux. So, I went around to them individually and said, “OK, what do you need me to do? What needs to be worked on?”

Every single major hardware company, said, “Hey, you’re right. Everything works on Linux. We’re fine.” That’s shown by these companies that ship Linux on their machines. Dell, HP, and all the big hardware companies and laptop manufacturers now ship Linux, and we’re working with them.

Some of the disconnect is if you buy a machine that doesn’t have Linux on it, getting it to work sometimes requires weird BIOS tricks to get all the function keys working on a laptop, and things like that. New hardware or hardware when you’re updating a system should work. If not, then we messed up.

The graphics guys have issues that they’ve been addressing to help make that work much better. Intel hardware works wonderfully now on Linux. Their developers are very good. ATI and NVIDIA both ship closed source drivers that work on all their devices. Those all work and have very good support systems. The drivers aren’t in the kernel, but they’re out there.

Anything storage related or networking related has been working with Linux forever, and all new devices support Linux too. They have to, because that’s what those markets are going for.

Then you get to the tiny consumer devices, which is where I like having fun, because I do USB. I work really hard to get everything supported, and we don’t know of anything these days that isn’t. I was in Tokyo the other day for the Kernel Summit and walking around Akihabara and trying to find devices that we don’t support. We had all the kernel developers there and we couldn’t find anything.

I know there’s a disconnect, but if you look on a mailing list for the Windows guys, they have a lot of driver problems, where things aren’t working. Mac OS X does not support very many devices at all.

If issues arise with exotic configurations, I’ll be glad to work through them all. If something doesn’t work, tell me so I know to fix it.

Scott: Is my perception correct that when hardware companies get into Linux, they sometimes start out with a bare-bones driver? It seems that often, the piece of hardware will work, but it won’t necessarily handle the full spectrum of services like power management with the device. Then they’ll provide more capabilities, gradually contributing code to use the device to its full capabilities.

Am I at all accurate in that perception, or is it more often the case that the driver they contribute offers the same level of functionality as on any other OS?

Greg: You’re right. Some things don’t work, although we’ve made it easier for companies to write a driver that will do everything, because we put core functionality like power management into the core. Power management is now handled by the core of the kernel, so you don’t have to add that. You just have to add a few hooks in your driver and then you’re done.

The average driver for Linux is about one third the size of an equivalent driver for another operating system, so you have less code to write and maintain. Still, to your point, there are some new devices that come out whose value add is that they do X and Y differently.

Sometimes we don’t support that yet, because the manufacturer didn’t tell us how to do it or we have to figure out how to fit it into the Linux framework to support it properly. Those little webcams are a very good example. New devices come out that can support features like additional bit rates, frame sizes, or screen sizes, and we need to go and add support to the core of the kernel to support those.

So, your statement is very fair, and that contributes to the disgruntlement you’ll see on my face at times.

[laughter]

If a brand new device that says it can do X, Y, and Z is only doing X on Linux, we immediately start to try to figure out how to get it to do the other things. Sometimes, we just have to work with the manufacturers to do that.

A lot of companies are getting better at working with us to come out with drivers at the same time the hardware comes out. Sometimes we lag by six months just by virtue of that fact that Linux isn’t the first system they care about, so it takes us a while to get it going. Also, if you’re using an older distro, you may need to switch versions.

There are all sorts of configuration issues there, which is why Jim Zemlin of the Linux Foundation continues to say we need to work on driver development, and I agree with him there. There’s always going to be work, so I think I have a lifetime job here.

Scott: Correct me if I’m wrong, but drivers never really leave the kernel, right? Is it more or less true that that’s one of the reasons that Linux supports old hardware so well? That once it’s in the kernel, it’s probably going to be in the kernel for a very long time, so I can grab a six-year-old laptop or six-year-old server, throw Linux at it, and have pretty good confidence it’s going to work?

Greg: Yes, and actually that’s turned out to be a problem. Because our kernel size is growing so much, we want to make changes, and we’re running into places where we have drivers for hardware that we know hasn’t been made in the past ten years. It can be complicated to excise those pieces.

That came up at the kernel summit last week, and actually, I am going to be working on that. We’re going to move drivers that we think are broken if we are pretty sure that nobody out there uses them anymore. We’ll move those into the staging tree, keep them there for about a year, and if nobody complains, then we’ll remove them.

On the other hand, if anybody shows up and says they need one of those drivers, we can use our source code tool to restore and test it. If there is at least one user of an old device in the world, I will gladly maintain the support.

Scott: We’ve covered a lot of great stuff, and now I want to turn to a lazy interviewer trick and ask you what really interesting things I haven’t asked about. In other words, I’d like for you to supply both the insightful question and the brilliant answer.

[laughter]

Greg: Well, just to touch back on that rate of change that I mentioned before, I just looked it up, and we add 11,000 lines, remove 5500 lines, and modify 2200 lines every single day.

People ask whether we can you keep that up, and I have to tell you that every single year, I say there’s no way we can go any faster than this. And then we do. We keep growing, and I don’t see that slowing down at all anywhere.

I mean, the giant server guys love us, the embedded guys love us, and there are entire processor families that only run Linux, so they rely on us. The fact that we’re out there everywhere in the world these days is actually pretty scary from an engineering standpoint. And even at that rate of change, we maintain a stable kernel.

It’s something that no one company can keep up with. It would actually be impossible at this point to create an operating system to compete against us. You can’t sustain that rate of change on your own, which makes it interesting to consider what might come after Linux.

For my part, I think the only thing that’s going to come after Linux is Linux itself, because we keep changing so much. I don’t see how companies can really compete with that.

Scott: Like you said, a lot of the rate of change in Linux as a whole is due to changes in the drivers. It’s fairly unique to Linux that it carries the drivers in the kernel, so as the ecosystem grows, you’ve got more and more people contributing code to Linux.

That’s not the case with other operating systems.

Greg: Right. And while that rate of change is consistent across the whole thing, it’s also easier to write drivers. As I said before, your driver for Linux is one third the size of your driver for Windows, so even at this rate of change, writing a driver for Linux is less work than it is for other operating systems.

In Linux, we’ve re-written our USB stack three or four times. Windows has done the same thing, but they had to keep their old USB stack and a lot of their old codes in order to work for those old drivers. So, their maintenance burden goes up over time while ours doesn’t.

Also, as people change jobs, they generally stay with the Linux kernel community. I’ve been doing USB work for over 10 years now for Linux, and that’s a big body of knowledge. Within other companies, engineers usually move around, and that body of knowledge doesn’t necessarily stick around.

For driver maintainers and driver authors within Linux, it does stick around. The networking guys have been doing that work for 15 years with Linux, and they know this stuff in and out. They can tell you what you need to change in your driver to make it work really well.

They’re unique, and they’ll help out any company.

Scott: Thanks; we’re out of time, and that’s a good place to close.

Greg: All right. Thanks a lot; this was fun.

Bookmark this:

• Defining platform as a service and the creation of Engine Yard
• Advancing cloud computing beyond buzzwords
• Opening cloud computing as a simple, provider-agnostic framework
• Operating system and programming language flexibility
• Providing flexible scalability of data stores
• Paradigm shifts created by cloud computing, including admin roles and rapid start-ups

Scott Swigart: Please take a moment to introduce yourself and your company.

Ezra Zygmuntowicz: I’m one of the founders of Engine Yard. We started life as a hosting company, and we’ve evolved into an open source company with a platform as a service cloud offering that we run on other people’s infrastructure.

I’ve been heavily involved in Ruby and in the Ruby on Rails open source community ever since Ruby on Rails first came out in 2004. Back then, I was webmaster for a newspaper–the Yakima Herald Republic in Yakima, Washington. I was looking for some kind of new language to redo their whole website, because it was kind of an old mess.

I was reading up on Python and Ruby, and when Ruby on Rails came out, I took it and ran with it, building one of the first commercial sites in Ruby on Rails. That site is still live and running there, and that’s how I got into the deployment and hosting sides of things.

Back in the early days, nobody knew anything about deploying Ruby on Rails applications. There just wasn’t much information available on it, and there weren’t many big sites doing it. I was the only tech person at the newspaper, and they bought me a couple servers and said, “You need to put this new Ruby on Rails site online and take the 250,000 hits a day we get at the old site right off the bat.”

That pushed me to start researching how to deploy and scale Ruby on Rails applications on an open source stack. In the community, I became one of the de facto deployment experts, and I wrote a book, Deploying Rails Applications for the Pragmatic Programmers.

The business partners that I started Engine Yard with had a consulting company called “Quality Humans” that was a Rails consulting shop. They had a couple of big clients, and they were trying to figure out how to deploy them. They took them to Rackspace, but Rackspace didn’t know anything about hosting Ruby back then, and Tom and Lance saw a business opportunity for building a high quality hosting platform for Rails applications.

They went looking for somebody to help them with that. They found me in the community, and we got together in early 2006 and started planning this platform for running Ruby applications.

We scrounged up our first $100,000 to get a few servers and our first data center in the summer of 2006. We started taking customers in September 2006, and it’s been a wild ride since then. We’ve grown from a tiny startup to 85 employees in 12 locations around the world, running quite a few very high-traffic Ruby on Rails applications. That’s where we are today.

Scott: How does open source fit into your business?

Along the way we’ve had to build all kinds of tools, and our whole stack is built on open source. We were active in a bunch of open source projects, and we have a number of projects ourselves. We have a project called “Rubinius,” which is a new virtual machine for running Ruby code that we’ve been developing for awhile.

We just hired the guys who run the JRuby project, which is a Ruby virtual machine built on the Java virtual machine so it runs on the JVM. They liked the idea of coming to work for a company focused on Ruby on Rails, so they ended up coming to work for us.

On top of that, we had an open source web framework called “Merb” that we wrote to create a sort of high performance version of Ruby on Rails, and that had quite a bit of success. It became more and more of a compelling product, and we realized that we were stealing some thunder from Ruby on Rails.

We had started our company to support Ruby on Rails, so we got together with the Rails core guys and we decided to merge the two projects. So, Merb and Rails have merged to become Rails 3, so it’s kind of a cool open source story.

The long view is that we kind of forked off to make some performance improvements and provide some different architecting approaches. That was successful, and now we’ve merged that back into the main line of Ruby on Rails to make it better. That’s the little story of how we got where we are today.

Scott: That’s an interesting history, and it I think it’s pretty unique to open source. It proceeds from the idea that a group of people can take a code base off into the wilderness, take it in a different direction to support a different use case or optimize it, and finally bring it back and fold it in.

We see a lot of open source projects go through iterations like that. For instance, between Apache 1 and Apache 2, there were lots of changes, and PHP has gone through similar evolutions. As you mentioned with Ruby, sometimes things go their separate ways, and they may stay separate, or they may converge back together.

Ezra: That’s a pretty cool open source story there, because it shows how open source is a new way of working on software. I think it’s a better way than closed source, because you can get so many more eyes on the problem.

Anybody can look at the code, and if they’ve got an itch, they can scratch it themselves and improve it to do what they want it to do. We were interested in being able to run these Ruby applications in the most efficient and resource-conscious way we could.

When we started building Merb out, it kind of took on a life of its own, eventually starting to look almost like a rival to the Rails project. Both projects basically have the same goals: to build a high-level, full stack that’s easy to get started with and that provides a very rapid application development platform.

It almost felt like we were fractioning the community a little bit by having Merb as an alternative to Ruby on Rails, and like we might be confusing people a little bit. There was starting to be a little bit of rivalry between the two camps, and I just didn’t think that was useful for the Ruby community at large to have this kind of chasm.

We got together with the Rail core guys and started recognizing that our views on most things weren’t so different. We started looking for ways that we could patch the two projects together and come out stronger on the other end.

Since then, we have been focusing all our work on Rails, and bringing the good parts and the performance focus we had in Merb into the new release of Rails, which is going to come out in a beta release sometime fairly soon. Because they’re fairly similar projects, there wasn’t any reason for them to be separate.

It was cool that everybody was able to get along and see that we’re all basically working toward the same goal, and we’ll be stronger if we combine efforts.

Scott: I guess Rails is sort of a platform as a service on top of cloud infrastructure. And since cloud’s the hot topic, let’s go ahead and stick the shovel in there. To start, how do you define that?

[laughter]

Ezra: Nobody knows what a cloud is these days. It’s kind of become a term for anything that runs on the Internet.

Scott: There are some people who are very serious about having a very strict definition, while others are kind of like, “Well, if we just put cloud on our marketing material, we’ll sell more. So let’s call what we’ve been doing for 10 years ‘cloud.’”

Ezra: Absolutely. To me, there are a couple of different kinds of clouds. First are the infrastructure as a service clouds, with the Amazon Web Services platform as the canonical example. That’s what I mainly think of when I hear the term “cloud,” because it’s what I feel the true definition of “cloud” is–an on-demand compute service where resources are disposable and scalable.

I would call something a cloud if it’s API addressable first and foremost. It has to have an API so it can be manipulated programmatically. It has to be able to scale up and scale down on demand, so if you need more server capacity, you just bring a few more servers up. If your load has gone down, you drop a few and you go back down. The business model is kind of metered, like the electrical grid, where you just plug in and pay by the hour.

Then there’s all kinds of other stuff that people are calling cloud. There’s higher level stuff, like platform as a service, which runs on top of infrastructure as a service, but it’s higher level and deals more with application services, rather than the low-level infrastructure.

Scott: What’s your own history with cloud computing?

Ezra: When we started Engine Yard back in 2006, we needed a platform to be able to build this Ruby platform as a service thing. This was before EC2 was out, and before people were calling things cloud, and before there was Amazon Web Services.

We built something in our own data center that looks a lot like EC2, based on Xen virtualization and Gentoo Linux. We still have our own customized version of Gentoo that we optimize for Ruby.

We built a private cloud ourselves, although that’s kind of a loaded term as well. It was a virtualized cluster platform that ran in our own data centers. We built our platform as a service on top of that. That’s higher level and deals with the application.

As time has gone on, the Amazon platform has gotten very compelling, and other clouds have popped up, so we have realized that the value that Engine Yard adds is not necessarily at the infrastructure as a service level. We don’t really give ourselves a big differentiation by racking hardware and running data centers. We can’t compete with somebody like Amazon on that scale, as far as racking and stacking servers.

We realized that our strong point is what happens after you have a VM online with network and compute and storage. Then you need to put an operating system on there and bring that up all the way up to the application. Then you need to monitor, scale and recover your application from failures. That’s more where we fit in, and that’s where our value is–in platform as a service.

Starting last year, we started taking everything that we’ve built, all our glue code, and all our open source stuff that we call our stack, and we started porting it to run on top of Amazon Web Services, rather than our own cloud. We’re kind of abstracting that, so that we can farm that out to Amazon or any of these other providers that have vCloud or other stuff popping up.

It’s not a super interesting business, racking and stacking servers, but the code and the platform that runs on top of that is a lot more interesting to us. So as a company, we’ve transitioned from the model where everything happens in our own data centers to a platform that runs out on the cloud in other people’s servers.

Scott: We’ve talked to a lot of people who are Amazon Web Services customers, and they’ve talked about use cases. Generally, it seems to be fairly good at existing applications.

For example, we talked to some guys who do web resources for the Indy 500. They’ve got an enormously elastic need. When there’s an event, they’re going to have a tremendous amount of traffic, but when there’s not an event, there’s very little traffic. And so they’ve got this need to spin up 100 servers, have them be up for a week or so, tear them down to a handful, and then spin them back up again.

We talked to other people who do diagnostic kind of stuff, and a lot of these show up as Amazon case studies, so if somebody wanted to go look, they could go get details on these. At any rate, this company runs their own internal grid for these analytical models, and when they’d get a particularly large data set from a customer, again, they have a very elastic need.

It worked pretty well for them to take their existing apps and just run them as is. They don’t need to maintain enough hardware for their peak load, and their internal servers now are just whatever they need for ongoing day to day operations. All the elasticity happens out in the cloud.

So that’s one kind of cloud, and then there are other kinds of clouds coming on line. Google’s been out there for awhile with App Engine. There’s also Windows Azure, where you’re coding to a certain kind of API, within a certain development framework, and you don’t even see machines. You just put your code up, and you don’t know or care whether it’s running on one box or 10 boxes or 100 boxes. It’s very request-response based.

And there you’re making a bet. With Amazon, you’re not making a bet on Windows or Linux or any of that stuff; you’re not even really making a long-term bet on Amazon as a cloud provider, unless you program to something like SQS. With somebody like Google or Microsoft, you’re making a bet not only on whether you want them as a cloud provider, but also on whether you really want to stick with that framework. If you’re using Google App Engine, then the only place it runs is on their framework. And the only place that framework is available is on their infrastructure.

Looking at that world, you might think that maybe there’s an opportunity for something that’s sort of in between. On one hand, maybe you could choose an infrastructure as a service provider, as a somewhat independent choice, and on the other hand, maybe you could choose the platform as a service framework, also independently. You could sort of mix and match the two.

Is that the kind of scenario that you are enabling?

Ezra: I think you hit the nail on the head about what we’re trying to do. I look at it like this. You’ve got raw AWS as the canonical example of the infrastructure as a service right now. But they hand you very low level building blocks. You get compute, you get network, you get some memory, get some storage.

You can choose Linux or Windows or whatever, but they hand you these big building blocks, and you still have to assemble those into something that can run your application, and scale, and be reliable at the application level, and so on. You still require quite a bit of system administration, and it’s not something that Joe Schmoe can just use out of the box.

Then there are the higher level things like Google App Engine, where Google has made a lot of decisions for you. As long as you can fit within their box, you don’t have to worry about servers anymore. You just twist the dials and scale up as needed and pay for what you use.

That’s really compelling to people, but it requires them to buy into the lock-in of this Google App Engine thing. You can’t run elsewhere, and you’re going to have to be able to color within their lines. There’s a definite limitation on what you can and cannot do there. You are betting the farm on that being the right way for you to architect things.

We’ve built this platform as a service that takes the low-level building blocks of something like Amazon or another infrastructure as a service provider, and we have done the research on how to assemble the best practices. We have figured out how to load balance a scalable application farm, a database with replication, and all this kind of stuff for you, and we’ve figured out how to work with elastic block storage, and snap-shotting, and all that kind of stuff.

We’ve taken that and wrapped it up into a nice little push button platform as a service, where it’s pretty much just as easy to get going as with App Engine or something. You tell us about your application code and click a few buttons, and we’ll build a whole cluster for you.

But it still looks like your old school Web 2.0 deployment. You’ve got your load balanced applications and your MySQL or Postgres databases. You get SSH access with root login on your boxes, so you can still login and see what’s going on, and it feels familiar to people.

Our platform as a service is kind of in the middle of that. You can still not really know what you’re doing and come and just click a few buttons, and get this full cluster and everything up and running, and then add capacity and remove capacity as your traffic goes up and down.

But there are no limitations as far as what you can run there. It’s just Linux. You have root access, so anything you have been running on Linux servers will still run on this platform and none of this color within the lines stuff, where you can’t do a certain thing on a platform because they don’t allow it.

Scott: Will it actually spin up and spin down nodes as demand increases and decreases?

Ezra: We haven’t released that feature yet, but it’s something that we’re working on right now, and we hope to release it fairly shortly. Right now we can basically do that, but you would get alerts, and then you would have to make a manual decision to click a button to add or remove a node. We are working on adding the whole auto-scaling thing.

Auto scaling is really hard to do in a least common denominator type of way that works for everybody, because it can be so application specific. But we can kind of get away with it because we’re doing this platform that is all Ruby applications. They all have very similar needs and profiles in terms of how they work.

Scott: I don’t know your guys’ architecture, but one of the scenarios somebody mentioned to us was, “It’s not memory or CPU that I’m concerned with; I care about the queue length and the simple queuing service. And based on more queue items, I want to spin up more nodes.”

Ezra: Right. If you just build an auto scaling system that adds nodes when your load average goes above X, it’s never going to work that well. You’re either adding too many nodes all the time, or not keeping up, or whatever.

Scott: Or not adding them to the right tier.

Ezra: Exactly. That’s not super helpful, so what we’re trying to do is build something that takes load into account as one of the factors.

Like you said, maybe they’ve got a queue they’re listening to. If the disk goes above X, maybe they want to spin up two more nodes. Or maybe their application is processing X requests a second, and if they hit a certain threshold, they want to scale up or down in a specific way.

We’re trying to build in some general heuristics that are least common denominator, but still allow it to be a little more intelligent about what it does.

Scott: Amazon’s an infrastructure as a service, and VMware’s approach doesn’t look like they’re going to stand up five massive data centers in five places on the planet. It looks like they have more of a strategy of going after existing hosters and getting them to be vCloud partners. By being a vCloud partner, they’re going to have an API exposed to them that lets them programmatically control their instances, kind of like what they can do with AWS. They’re going to have a self-service provisioning portal, an admin console, and things like that.

Do you see the Rails stuff that you guys are working on down the line supporting this heterogeneous ecosystem of infrastructure as a service providers?

Ezra: That’s definitely something we’re targeting. We’ve been talking with VMware a lot lately about the vCloud stuff, and we had an early demo of our platform as a service working against vCloud that we demoed during the keynote at VMworld in Paris six months ago.

We’ve got the AWS thing down, and it’s working great. We’ve got lots of customers on there and customer rate is growing great. And right now we went GA with our platform at the end of September, so this first version is fairly feature complete.

Porting our system to work against vCloud would give us a bunch of bang for our buck, because that’s not going to tie us to just one infrastructure as a service provider. If we code to the vCloud API, there’s going to be 10 or 20 of these vCloud approved service providers that we can point our application at, and go to town. Then we know how to instantiate a cluster that would run your applications on AWS or on Terremark’s vCloud, or on Hosting.com’s vCloud, or any of these other places.

Scott: And that’s not the piece you are focused on, right? You’re focused on your framework, but like I said, you could envision a future where there’s a Rails cloud framework and a pile of hosts that people can choose from, and maybe some other Java cloud framework or PHP cloud framework or .NET cloud framework, or whatever, and they could mix and match.

In that scenario, if a given department prefers a certain language, there’s a cloud-based, auto scaling dynamic framework that supports that. That also provides a choice of development models and a choice of places to host it that can also be mixed and matched.

Today, I could run ASP.NET on Rackspace and run PHP on Hosting.com, but those are not really cloud frameworks, and those providers aren’t really in the path. The hosting providers haven’t really provided an elastic resource, and it’s been fairly static kinds of units that people have to buy in.

It seems like there’s a gravitational attraction toward the programming frameworks becoming better at cloud development and the traditional infrastructure providers becoming better at being cloud infrastructure providers.

Ezra: It makes a lot of sense. For example, with Google App Engine, you really have to code to their platform to run your app on there. Everything you write in your app has to take into account what they’ve done there, and how to use a different data store and everything.

But with our platform it’s just your typical Web 2.0 stack with your applications here, your memcache here, and your database here. It looks familiar to people, and they can run their existing applications on it but still take advantage of the scalability in the cloud.

If they get their application running against our platform on AWS, as soon as we add other back ends, they’re going to be able to click a button to deploy it out onto a different cloud without any changes in their code or anything.

We’ll take care of the differences in architecture underneath and still expose the same interface to the application developers. They can just worry about writing their code and let us take care of figuring out the best architecture for each different cloud.

Scott: One other facet is that if people want to run this framework internally, they can benefit from using the same framework internally and externally.

Is the stuff you guys are working on mainly a Linux thing, or could you see people running Rails 3 on Solaris? Does it work on Windows? What degree of choice do people have there?

Ezra: Rails does work on Windows and Solaris, but Ruby in general is kind of sub-par on Windows compared to Linux, and nobody really deploys production Ruby stuff on Windows unless they absolutely have to integrate with some .NET or other Windows-specific stuff.

Scott: As you look at the other choices that people have in that Linux ecosystem, a lot of times it’s Java, PHP, and a smattering of other things. And then there’s Rails, which has gained a lot of popularity in a relatively short period of time.

Popular conception seems to portray Java on one end as the strongly typed, object oriented, structured enterprise programming language and PHP on the other side that lends itself to learning in a weekend. Coming from a Microsoft background, I think of it like VB for the Web. There’s a very low barrier to entry, but the people with computer science degrees point at it as not real programming.

As a development framework in general, and then thinking about cloud development scenarios of the future, what do you think makes Rails a good fit?

Ezra: I think that Rails is really good at building a prototype and getting something out and usable to your customers very quickly. It makes some of these decisions for you in starting a new project.

If you’re starting a project in Java, you’ve got to sit down and figure out what application server you’re going to be using, what choice of frameworks, and which ORM, and which XML config files. You’ve got to piece that all together, and then spend a few days configuring it, so there’s friction to starting something new and just trying it out.

Rails has a really nice out of the box experience, where it makes all these decisions for you that are fairly sane. You can tweak them later if you need to, but it allows you to just jump the gun and get something started very quickly. And then there’s a whole slew of plug ins for doing social networking things, friend of a friend, or sending emails, or all this stuff just out of the box you can plug in with very little code.

One of the big advantages of Rails I see is that it allows you to take an idea and have something to show people in a week or two with a couple guys developing on it. That might take three or four months to get the same prototype in a Java system. It’s just very rapid development.

You can rapidly prototype a website, get it in front of your customer, and then iterate based on real customer feedback, rather than having these really long iterations where you get a bunch of specifications from a customer, and then you come back to them three months later.

Then the customer doesn’t like it, so they give you a bunch more specifications. Then you come back three months later, and maybe you’re kind of ships sailing past each other in the night, not really getting the feedback that you need.

With Rails, you could be showing stuff to your customer every day, or every couple of days, and then iterating based on their real time feedback. You can just be so much more productive with it, and I think that’s where its big advantage is.

As your app grows up and becomes more complex, Rails can support that too. And it has tunables and other options so you can get in there to make it really scale. There are good stories for the stack and the way to scale it now.

One of the other big advantages is that Rails enforces a directory structure, the way the app’s laid out, and the testing frameworks and so forth. That means you can hire a Rails developer, and they’re going to be able to find his way around pretty easily. There’s not going to be a month or two of learning how you customize J2EE or whatever. So there’s this shared knowledge between all Rails apps that allows you to get programmers up to speed much quicker.

Scott: When you compare it to PHP on the other end, is the difference the model of your controller paradigm, or what? Because people think of PHP as being a very rapid way to stand something up and prototype it.

Ezra: You can also rapidly prototype stuff in PHP, but when you do and then it gets successful, you’re probably going to have to rewrite it entirely because it’s probably a spaghetti mess. With Rails, you could rapidly prototype something, and then if that prototype is successful, you’re built on a very solid foundation where you can keep scaling that project and keep it maintainable as you add developers and stuff.

Scott: With things like Google and Google App Engine and Bigtable, it seems that the popular approach to scale a website is by using a traditional kind of relational database back end. It could be MySQL, PostgreSQL, Oracle, SQL Server, take your pick.

But if you really want to scale infinitely from the beginning, you’ve got to build on top of one of these distributed, shared databases, whether that’s SimpleDB or Bigtable or whatever. Talk to me about the options that you have at the data layer.

Ezra: There’s this whole no SQL movement going on, where people are exploring other databases, because they’ve outgrown MySQL or PostgreSQL. The reality is that I think there’s a little bit too much hype around that stuff right now.

Most people will never outgrow MySQL or PostgreSQL, even though we have a lot of people who come to us saying that they’re going to be the next Google or Twitter, so they have to be able to scale huge.

They might lose a lot of time by trying different options, whereas they’d be better off just starting with a SQL database and getting something out there fast. Then, as they hit the ceiling, they can move portions of that out into some kind of key value store or something else.

We have support for a couple different key value stores on our cloud. You can use Redis, or Toyko Cabinet and Tokyo Tyrant. We have out of the box support for those two. We also like MongoDB a lot. And also MySQL with as many replicas as you want.

The progression usually starts with the thing on MySQL server and a couple app servers, and as I scale those up, I might add a slave or two to the MySQL server and do read-only queries against the slaves and scale down my reads.

As I start to swamp that master database with my write capacity, I might move certain features out into a memcache or a Tokyo Tyrant tier where I don’t need as sophisticated data access patterns and ad hoc queries; I just need raw speed and scalability of storing a ton of data.

If you have some app that really just screams out, “I don’t need a database, I just need a key value store,” you might start with one of those from the beginning. Still, you’re going to want to keep your customer and your transactions–anything to do with money–in a SQL database with the proper constraints and stuff.

But there is quite a bit of stuff, especially in web pages. You can store rendered HTML template fragments in a memcache or a Tokyo Tyrant, because they might be pretty expensive to pull all the data and build a template, and then you might cache the result of that.

The idea is basically to scale MySQL up as far as you can go and then start taking features and pushing them off into a key value store to take a load off of the database, and to have a system where you’ve got a relational database augmented by key value stores, and you’re using them each for what they’re really good at.

Scott: That ties into another thought I had. As some sites have gotten really popular, they have crashed under their load, so they pushed all their images and static data into S3. The page would just have a bunch of image tags, and it would pull the images from Amazon S3. They would off-load a ton of bandwidth from their servers, and the user wouldn’t really see any difference.

There were a bunch of case studies around that, which told the story of how Amazon S3 saved some sites when they got popular, because they were able to off-load most their bandwidth, without really off-loading any of their processing.

Like you said, Google really forces you to color within their lines, with a limited choice of programming languages, no relational data store, and not really great support for a background process. It’s still sort of under the constraints of a request and a response.

People have taken the part that won’t fit in those lines and stuck that over on Amazon, and they just call out to it from the Google App Engine code, or those kinds of things. People I’ve talked to who have programmed on Google App Engine have said that data store is the paradigm shift.

That’s the hardest thing to wrap your head around if you’re used to programming against a relational database, which everybody is. Paradigm shifts are hard. It took 20 years to go from procedural programming to object oriented programming.

Ezra: I think those things are definitely in the future for a lot of apps and a lot of programmers, but like you said, they are paradigm shifts, and a lot of people just aren’t ready to make that shift yet. Cloud computing is pretty young still, and everybody is still trying to figure out how it’s going to work, what they’re going to use it for.

There’s another good five years here where people need to move their legacy applications to the cloud, or they want a cloud environment that looks similar to their on-premises stuff, but they might not be ready to make the jump to the full-on Google App Engine style.

I think something like what we’ve built will be more compelling for the next several years. It takes advantage of the elasticity of cloud, and the programming capability of the APIs and stuff, but it still kind of looks like your traditional infrastructure.

Scott: It might be compelling for a lot more than the next five years. At the end of the day, free is really nice, and it is going to attract a lot of customers. But, the corollary is that those providers are not necessarily going to make a lot of money off of those customers, because they were attracted by something that’s free.

You can kind of democratize and commoditize a certain kind of lower end of the hosting space, but I look at you guys, and I think, OK, there’s a framework that I could run. I could run it internally. I could in the future run it on a variety of external partners, whether it’s Google, or vCloud or whoever. It’s one of the frameworks I could choose.

Tell me if this is part of your pitch, but I could see companies that are looking at this saying, “Well yes, but you’re a small company, you’re a start up, we don’t really want to bet on you.” Is this a place where open source helps, since you’re able to tell them that they’re not really betting on you but rather on the longevity of an open source project?

Ezra: I think that one of the strengths of our software is that it is built entirely of open source components. We have a lot of proprietary stuff that ties it all together, but at the end of the day, the stuff running your app is Linux, nginx, Rails, MySQL, memcache, and a bunch of other stuff that’s all open source.

If we went out of business or you didn’t want to host with those anymore, you’re not locked in. You can pick up your app and go elsewhere, and be able to put something together that looks close enough where you’re not going to have to rewrite your code. That helps, because you don’t get locked in when you use our platform like you would with App Engine or something.

Scott: We’re getting near the end of our allotted time, so is there something interesting I didn’t ask about?

Ezra: I think we’ve covered the main points. The whole thing is a paradigm shift, since there are still a lot of people that I call “the server huggers,” who really like racking and stacking servers. They like seeing all those blinking lights, and they like having direct control over the servers and being able to choose the hardware down to every last card. Those guys are having a bit of a harder time with this transition.

I think the whole transition to cloud computing is really going to come to a head and force a merger between typical operations guys and developers. Since the infrastructure is programmable via API, developers are enabled with the power to control the infrastructure without having to call the systems guy every time they need a new server.

I think that your traditional operations guys are going to have to learn to become developers, and I think your traditional developers are going to have to learn more about operations, so that they can play within this new environment that can be totally programmable. I think that’s going to be one of the big paradigm shifts here.

Scott: It’s also interesting that you mentioned that in order to start out, you had to scrape together $100,000 and put together a small data center. Today, the distance between an idea and a product in the market is getting smaller.

That’s due in large part to the work that companies like you, Amazon, and Google are doing. The start up money that somebody needs to take an idea and run with it is far less, and if that notion doesn’t work, fail fast and try something else.

It seems that these technologies are just letting people try things out and spend a lot less time doing the market opportunity analysis and writing the business plan, and instead just trying stuff.

Ezra: The whole switch from Capex to Opex is a real game changer, and you’re right that it is largely cloud-enabled. If a couple of guys at a university have an idea that needs a couple hundred servers to run, in the old days, they would have to have a lot of money to buy a hundred servers and rack them up and get them available.

With the cloud, you can boot a hundred servers and play with them for a few days and shut it off by just spending 100 bucks or something. Being able to that easily try out an idea and see if it’s feasible or not is a game changer.

That flexibility is really compounded by things like Rails, which are enabling really rapid development of these sites. It gives a lot more power to individuals, and smaller teams these days can build pretty complicated products very fast and then scale them very easily.

That’s huge, I think. That’s one of the biggest things that this whole cloud trend is changing. All these open source technologies, along with the infrastructure on demand really does allow people to just try big ideas and then fail fast. Like you said, if it’s not going to work, they’re not going to be out a lot of money.

Scott: It also enables departments within a 150,000 person company to be really entrepreneurial and really experiment with things. The boss could practically try out an idea with his credit card, and the budget could go on an expense report.

Departments can go outside of traditional IT because they’ve got a budget and for things that are below a certain threshold, they don’t need approval. They can have a hosted CRM system or mail accounts or all kinds of things, including their own servers in the cloud that their IT department doesn’t even know about. They’re completely off the radar.

On one hand, IT cringes at that, but on the other hand, companies want risk taking, especially if there isn’t even really that much risk with this kind of entrepreneurial spirit, because it’s just hard to predict what’s going to work.

Ezra: There’s a big push for agile development. Test-driven development, Scrum, Agile, and all this kind of stuff has really changed the way people write software. They do rapid iterations and they test and they get stuff done very fast.

I see this whole space we’re in as enabling agility and deployment as well, because it used to be that maybe your development team is rapidly iterating and coming up with new features and all this stuff, but then there’s this big brick wall to get it deployed. They had to hand it to the ops guys and order servers and wait.

It was a big ordeal to get stuff deployed, but this new approach totally wipes that out and allows you not only to rapidly develop your product but also continuously deploy it and rapidly have it online and visible. It tears down that last wall of having your product go from idea to actually being out alive in the world and doing something useful.

Scott: I was just reading “Information Week” and there was a CIO saying, “One of the reasons I’m going with a hosted solution is my people never have to perform an upgrade. They never have to upgrade the CRM system. They never have to upgrade the mail system. Those used to be big projects that took lots of planning and lots of experimentation, and I will never have to devote staff to doing an upgrade. And at the same time, we feel safer making customizations because there’s a little bit more of a feeling that that fast provider isn’t going to break everybody’s customizations, because they’ve built the sandbox that we can customize within.”

And so they’ve got better visibility into what people are doing and not doing because it isn’t the case anymore that just anybody could be doing anything. They built the sandbox up fairly incrementally in terms of customers wanting to be able to do X, so we let them do X. Well, now they know they can do that, so if they’re upgrading their infrastructure, they’re able to take that into account.

I guess we are out of time. Thanks for talking with me today.

Ezra: Thank you.

Bookmark this:

In this interview, we explore the concept or layering an arbitrary PaaS
implementation on top of an arbitrary IaaS provider.  Hmmm…

• How Open Platform as a Service fits into the development process and the industry as a whole
• Building an open software architecture for loosely coupled components and environments
• Developer interaction with Open Platform as a Service
• Dealing with elastic resource demands

Scott Swigart: To start, please just take a minute and introduce yourself.

Brian Sullivan: OK, great. I’m the president of Sullivan Software Systems, which makes Open Platform as a Service. We officially launched on June 30th of this year, although we have had it percolating for some time.

Before this, I had various jobs in some large and small corporations and start ups. My background is in development, and I got into web development pretty much at the beginning of the web. My first browser was Mosaic, and I was fortunate enough to work at the first commercial ISP in Boston, Massachusetts, where I was born and raised.

I’ve been doing web development ever since then, so I’ve seen a lot of ideas, concepts, organizations, movements, and standards come and go. It was really a natural conclusion, rather than fight all the different forces that are in play on the Internet, to borrow the best ideas from different realms and try to apply them into a coherent framework that makes sense.

That’s really where Open Platform as a Service came from–the grassroots of developing hundreds of applications and websites over the years with different teams and folks. It’s also been self-challenging for me.

A lot of developers aren’t all that challenged in their nine to five jobs. There’s a lot of mundane work involved with just trying to get the project out the door, on schedule and on budget. Many don’t really get a chance to do things that are that interesting to them, although it helps if you’re in basic or applied research where there isn’t the profit pressure.

Part of the motivation for Open Platform as a Service is to do something that makes a lot of sense, rather than chasing after the latest catchy buzzword. It’s an opportunity to try to come up with a whole ecosystem that really makes a difference in software development where the rubber meets the road.

Scott: What does Open Platform as a Service offer to the development process?

Brian: At 10, 000 feet, it’s a framework for Platform as a Service. It’s the top layer that the developer interacts with, and it doesn’t have any slant toward any of the underlying tools or methodologies. The developer is free to choose whatever tools they want to get the job done.

To drill down a little bit with an example, let’s say you’re at a company that has a lot of Java code, and they want to add to that system, but for whatever reason, they don’t want to use Java. Maybe its legacy code and they’ve switched to something else.

With Open Platform as a Service, they can keep all that Java code and add different code, functions, applications, or systems from one or more other servers, in a disparate, heterogeneous environment.

Open Platform as a Service will glue that all together and make it all work, so it will handle all the traffic and calling in order to make different code on different servers work together seamlessly to the end user.

For the developer, this approach opens up a lot of opportunities, because you’re not limited to expanding the environment in any particular language. You can leverage your existing code base and the millions of lines of code that are out there on the web with an amazing amount of flexibility.

To take our view down to 5,000 feet, it’s a way to make all different code on all different servers interact in real time for the end user and for other developers. You can also expose the functionality to other users and developers.

Scott: Is all of that happening on the server side or the client side? Is it a server-side integration, or is it more like a browser mash-up kind of technology?

Brian: Open Platform as a Service would make standard HTTP calls over the web, so you would dictate where the calls are made, and which parameters are passed. You would get the return results from whichever function that you called over HTTP, and then you can do whatever you want with it.

For example, we could have a CRM system where functions are called from a combination of web services, home-brewed PERL scripts, Ruby on Rails, PHP, and .NET. With Open Platform as a Service, you can leverage all that different code on different servers but at the same time make it all work together seamlessly in a standardized format that makes sense in terms of what developers are used to seeing.

For anybody who comes from a background using any sort of IDE, Open Platform as a Service will be very easy for them to use. They could get up and running literally in a matter of minutes, and it doesn’t have any of the legacy baggage of trying to somehow capitalize on an existing investment in a given technology or infrastructure.

We’ve built on top of all of that in the most open way that we could think of.

Scott: Is it accurate, then, to describe it as a tool for building platform as a service that gives you a uniform way of getting data to and from other services, regardless of the programming language they’re using?

Brian: That’s a big part of it. It’s standard calls via HTTP to any web server you want, which is important in having an open system.

Scott: What does Open Platform as a Service run on? Is it a Linux technology? Is it a Windows technology? Is it sort of OS agnostic? What does it need underneath it to run?

Brian: It’s OS agnostic, and it can be implemented in any programming language. The actual development environment right now is in PHP, but there are also other languages available. The public cloud we’re running right now is PHP on top of Amazon Web Services for the underlying infrastructure.

Scott: That makes sense. Are there examples of things that are built on this technology that you can point to?

Brian: Yes, there are real applications using it now. I can’t go into names, but suffice to say some of the biggest software companies in the world are developing some applications and systems with us.

Scott: You mentioned running on PHP on top of Amazon web services. This is an interesting area–providers who are providing infrastructure as a service. Amazon provides the equivalent of virtual private servers, and while I could still go to a hosting company and get a virtual private server, I could also go to Amazon and through a self service portal and API, I could spin up and down instances myself. I’m only paying per hour, but basically what I’ve got is a full virtual machine.

On the other end of the spectrum, people like Google App Engine are providing platform as a service, but it’s very locked in, in the sense that they have a fairly constrained way of development. They’ve got a couple of languages I could pick.

Originally I could use Python, and now they’ve got Java. One day to store a program against, very constrained, but assuming I’m willing to play in that sandbox, I get the promise of free up to a certain amount of use and just infinite scalability.

And so what I’m seeing is opportunity in this niche that you seem to be fitting into: something in between where I can choose whatever infrastructure provider I want. It could be Amazon, or in the future maybe it could be other infrastructure providers like VMware or other infrastructure as a service providers.

So I can pick whichever one of those makes sense for me, and then I can pick a platform as a service to lay on top of it. I can sort of mix and match the way I want to build code and who I want my infrastructure provider to be, and I can even run it internally, if I want to. Can you speak to that environment I’ve just described and elaborate on it a bit?

Brian: Similar to how we are not forcing anybody to use a particular programming language, we are not actually forcing anybody to use a particular anything. We’re not coming into this as a cool way to slap some moniker on it, like platform as a service or cloud or whatever just to get in on that rush.

We’re coming at it from a pure play perspective, to build an open ecosystem that makes sense for developers so they can save time and make better applications in a standard format. Just as we invite people to use any language they want, the same holds true for every component up to development, creation, and delivery in the life cycle.

So pick any programming language you want, use any tool that you want. Use the same tool that you are using today, implement it on any server you want, implement it on any infrastructure, use any database you want, put it anywhere, put any of the parts anywhere you want.

Developers make functions so we can break down something that’s complex and make it reusable and manageable, as well as to distribute the development. We’re taking that same concept and mapping it over the web, so functions can be anywhere in any language.

The same reasons it makes sense to break down an application into functions, in our opinion, are why you would want to use Open Platform as a Service. From a developer’s perspective, it allows you to stand on the shoulders of the right people and try just to do things that make sense.

You could take any language you want, put the code on any server you want, use any cloud you want. You might want to use Amazon for some stuff, but not all of it. You could also use Google for some of the stuff and your corporate server for some other stuff, for example.

With Open Platform as a Service, you can do that; you’re not wedded to one or two. Mix and match whatever you want, use the best tools. Go into the shed, pick the best tools out, go out in the yard, and get the job done. That’s really the approach to the whole ecosystem of Open Platform as a Service. Other people in your company are counting on you to make the decisions, so we’re saying we agree with that.

This approach also helps standardize the way that you do things, rather than forcing you to be completely ad hoc, trying to glue stuff together as a one-off every time. After all, the most time and effort in software development happens after it’s been hacked together the first time. It’s the maintenance and upgrades over time that require most of the time and effort.

With Open Platform as a Service, you’re not stuck with a one-off solution that’s a beast to maintain, because the process has been standardized. The functionality is exposed in a way that makes sense. So to add, modify, and remove functionality is very simple.

As an example, let’s say you have an SMTP mail system. With Open Platform as a service, you can change that one component, not touching anything else and hit the ground running, test it, and cut it over almost instantly.

To go back to the same reasons you would make functions for a modular approach, now you can change out car parts. Keep your car going and put in a new air filter or carburetor, but you don’t have to change your whole engine.

I think the fact that most of the time and effort and money is spent after the code has been created the first time is often overlooked when people are coming up with development environments and looking at the total cost of software development.

Scott: When I’ve got these different components, like mail systems and Java code, in nuts and bolts terms, when I’m using your stuff, what am I seeing? It doesn’t sound like I’m seeing an API, it sounds like what I’m interacting with is more of a development environment or something that lets me put a user interface together that calls out to all of these disparate things. Is that about right?

Brian: That’s a very good general analysis of it. If we were to compare it to something else that’s out there that we know as developers, you could look at it as a very lightweight IDE. It doesn’t have the step through and the source code control and all of that stuff, because it doesn’t need it, but, it has an object selector and then a property sheet so you can select objects and then you can work on them.

It’s all 100 percent web based, of course, so there’s no software to install. You’re looking at standard web content and adding objects in an object selector. You’re selecting them just like you would in most any IDE, but it is lightweight, because it doesn’t need all of the advanced functionalities of a full blown IDE.

In addition to the object selector, there’s what would be analogous to property sheets, so you’re selecting objects and you’re setting properties. That’s what you’re interacting with as the developer or the business user.

Scott: Does that generate a user interface, or does it generate what’s one layer below the user interface, and then I put a PHP interface or something else like that on top of that?

Brian: This is where we focus in on the word “Open.” It can be anything you want. Let’s say that you develop an application, and you tell me what your end product wants to be. Is your end product going to be some business objects? Is your end product going to be some source code? Is your end product going to be an HTML report system or some business intelligence? What do you want to give your end user?

It is just that open, so it doesn’t limit you in terms of the output. You define that.

Scott: It makes me think back to the notion that using whatever traditional software methodologies you want, whether it’s agile development, waterfall, or whatever, you’re building different modules. Or you’re identifying modules that already exist out on the web and you’re using a tool to plug that stuff together and emit some kind of an application on top of it. The application could be an AJAX-based Web 2.0 application, or it could be sort of a web service that other stuff would talk to programmatically.

It sounds like a lot of what Open Platform as a Service is storing is metadata about how these things should all connect and come together. That’s an interesting way to tackle the issue.

One other thing I want to dig into, since you are running on top of Amazon, is the notion of elasticity. That’s one of the reasons that people turn to cloud providers. For example, people might have a website built around sporting events.

The problem space that those people live in is that, when there’s an event, there’s a huge amount of traffic, and when there’s not an event, there’s very little traffic. Because of that usage pattern, they want to potentially scale up and down their capacity very quickly. Today I have three servers, and tomorrow I need 40 servers. I need them for a week and then I can go back to three.

Talk a little bit about Open Platform as a Service from the elasticity angle.

Brian: Again, in trying to come up with the most open system, we’re saying use whatever infrastructure you feel comfortable with. If you want to go and rent out some time on the most robust system available because you’re doing a Superbowl ad, go ahead and do that, and Open Platform as a Service will enable you to pick any underlying infrastructure.

It sounds like your question is in part about whether there is inherent scalability, reliability, and availability in Open Platform as a Service. The answer is yes, because you can run that on whatever infrastructure you choose to run. The cool thing about that is that we don’t say it has to run on Amazon. We don’t say that it has to be public.

You can run Open Platform as a Service behind your firewall, in your private cloud, or wherever you want. On your public server, on Google, on IBM Blue Cloud, the next rev or what they’re calling that. When the next new kid on the block comes out with a cool new infrastructure, you’ll also be able to run on that.

You can run it wherever you want, on as many servers as you want, because we will provide that code to you if that’s the way that you feel most comfortable doing it. We’re not saying all of this is open and you can use whatever you want but you have to use this infrastructure. We’re saying we are open, pure play through and through. Use it however you as the architect choose the actual underlying infrastructure that’s best for you. And it can change.

To go back to the Superbowl example, you’re going to have an unbelievable number of hits for a very short period of time. So maybe you have it in a private cloud to test it, and everything looks good, and just for that day, you want to run it on the different platform. You can go ahead and do that. Just point it to that new underlying infrastructure, and then after the Superbowl or a week later, bring it back inside your private cloud.

Scott: Where’s the best place for people to kind of kick the tires and see it in action? Where do people go next?

Brian: OpenPlatformAsAService.com. Rolls off the tongue, doesn’t it?

laughter

Scott: Is there anything else you want to add?

Brian: One more thing that I would add is that we’re now launching the Open Store, which is a marketplace for developers to sell their applications.

Scott: So if they’re using Open Platform as a Service as a way to develop applications, then the store is where I could go to see what people have developed and consider whether it would meet my needs?

Brian: That’s right.

Scott: Cool. Thanks for taking the time to talk today.

Brian: Thank you.

Bookmark this:

• The early days of computing as the foundations for open source
• Open source growing up alongside proprietary technology
• Circumstances that led to the dominance of Linux among open OSs
• The economic motivation for open source as the result of technological progress
• On the rise of commercial entities as open source guardians
• The effects of OEM momentum behind particular open source technologies
• Low-cost, lightweight computing as an upcoming nexus of power
• Keeping customers for the long haul without being able to lock them in

Scott Swigart: To start, could you please tell us a bit about yourself and your background with open source?

Con Zymaris: I wear a number of different hats, probably the most prominent of which is my position as CEO of one of the world’s first open source companies, a company called Cybersource that we started in 1991. There are other companies called Cybersource, in the United States and elsewhere, but we were the first.

We focus predominantly on value-added services around software and technology. We started the company because we saw the Internet as a fantastic tool for business, and the software that made the Internet work in 1991 was open source software.

Among the other hats that I wear, I’m also the convener of a government-funded industry cluster, which is a group of open source companies here in Victoria, Australia, called Open Source Victoria. We’ve been running that for about seven years now as an advocacy and industry development forum for open source businesses.

My background is predominantly in software development. I started programming on Apple IIs back in the late ’70s. I moved from there to programming on CP/M systems and to UNIX and DOS in the mid 80s, and eventually on Windows and whatnot.

About 10 years ago, my focus shifted predominantly to business development and evangelism of open source software involved. I still dabble in software development, particularly to help my kids along when they’re doing software projects, rather than being directly involved in major coding efforts.

My involvement in open source projects is predominantly in terms of testing, documentation, business development, evangelism, and so forth.

Scott: So, did you skip over the Commodore 64 or did you just leave it out of the list?

Con: At the time in Australia, we had a number of different manufacturers of microcomputers. One of these was actually a very nice machine called “MicroBee.” It had a lot of features and was a very good value for the money. I spent quite a few years programming software on that.

There was a schism of sorts among the high school kids coding either on something like the Commodore 64, VIC 20, or these other eight-bit systems like the MicroBee. So I spent an amount of time in eight-bit assembler and whatnot coding up mostly game software.

Game software got a lot of us into the industry in the early ’80s. I spent the first three or four years coding up what would be classified as very poor games nowadays. But back then they were quite a bit of fun to make. Probably even more fun than to play, and a lot of us were more interested in coding than playing.

Scott: The VIC 20 and Commodore 64 were my entry point into software development. The various bulletin board systems that ran on them were in some ways the precursor to the forums and the social networks we have on the Internet today.

Con: Indeed, yes. There were a lot of people who joined the industry with the VIC 20 and even the TRS-80. It made computing not only affordable but also fun. Actually, that’s one of the things that caught my attention about open source software almost two decades ago. Open source software reinvigorated a sense of fun. The industry had become a bit too buttoned down, mundane, and mainstream.

There was an era just 10 or 15 years before then when it was just a ball of fun. Open source reintroduced that aspect to me, which is perhaps the main reason I joined the community.

Scott: I think the first time I really bumped into open source, I was a programmer for a Fortune 500 company. We were buying UNIX systems from Motorola, and we didn’t want to spend the very large amount of money for their development environment.

We started using this funny thing called GCC which seemed to work just fine, and it seemed to be supported by these strange people who gave us the compiler for free. But we were very happy to use it as part of our development.

Con: Having access to these excellent pieces of software that you otherwise wouldn’t use and couldn’t afford certainly does facilitate the extension of knowledge and the spreading of ideas. The world is somewhat different now, and people expect free. If you go back 20 years or so, to have quality tools at your disposal, which didn’t cost tens of thousands of dollars, was a godsend.

We had access to increasingly useful open source software even back in the late ’80s. That was the beginning of the realization in the industry that good software doesn’t necessarily have to come from a name-brand vendor. You’ve got a job to do, and at the end of the day, this is a functional tool for you to use for your job.

That’s one of the messages that we’ve been trying to get across to new players and new users of open source software. Putting aside all the religious stuff and the politics this is a tool. It either is viable for you or it’s not. But we see our job (as open source evangelists) as at least making such tools visible to the broader community.

Scott: How do you see the state of open source right now along that whole trajectory of twenty years or so? Back then, most technical people didn’t even know what open source was, and they were used to proprietary combined hardware/software platforms. Maybe they used the GCC compiler without really understanding what was behind it.

Where are we today, in terms of where open source has evolved to and maybe the next steps in terms of where it’s going?

Con: If we go back 30 years or so, pretty much every vendor built, not necessarily their own chips, but certainly their own computer hardware, which was different than other people’s computer hardware, and possibly their own operating system, although more often than not, they were using something like CP/M. There was a degree of “not invented here syndrome” that was quite dramatic.

That has changed a great deal, and Microsoft actually played a large and positive role there. I spent the better part of 10 years building applications for Microsoft platforms back when hardly anybody used Windows, back in the ’80s. I was building applications for Windows 2.0, whereas most people didn’t really start using Windows until Windows 3.0 or 3.1.

Microsoft in particular has helped to make the OEM experience much more mainstream than a company like Digital Research did with their CP/M product. Microsoft began its own operating system business by buying a CP/M clone back in 1981.

That was officially converted in name and eventually in feature-set to what became DOS 2 and then DOS 3 and DOS 4. Another thing that other people probably don’t know that Windows NT initially began life initially as OS/2 Version 3. And as part of the divorce proceedings with IBM, Microsoft got to keep OS/2 Version 3, which in 1989 and 1990, when it was being developed to be a multi CPU-platform version of OS/2.

So you have a great deal of effort by Microsoft to homogenize, if you like, an operating system that all these different PC manufacturers could share. And that was a great success that gave wings to the whole concept of mainstreaming the operating system platform.

So this is something that changed the industry quite radically. Microsoft has obviously benefited quite substantially because of its success with that homogenization. The general public has as well, I think, from the rise of compatible and standardized software. For many years, it was a sleeper that was slowly building momentum, and that standardization that occurred under Microsoft’s control made PCs mainstream.

Scott: And gradually, it became more important, until that model became the most prevalent one in the industry. In some ways, it’s analogous to the rise of the Internet, which existed but only quietly for a long time before becoming incredibly prevalent and important.

Con: That’s a good analogy. The Internet began 40 years ago, but it didn’t really enter public consciousness beyond the computer industry players and the technical cognoscenti until about the mid ’90s. Until maybe ‘93 or ‘94, most people had never heard of the Internet, and by then, it had been around for 25 years.

That’s an odd thing, and most people don’t really understand that it had been around a long time building up momentum among technical people before it suddenly reached into all corners of everyone’s lives. We don’t make a telephone call nowadays without going through some kind of Internet infrastructure in many countries around the world.

When I was finishing up my degree, I was also working for quite a substantial software distribution company, a very technical company. At the time, I was sharing a house with a system administrator who worked at the University that had just connected Australia to a Internet using a full IP protocol link. This was in the late ’80s.

I experienced Internet access at home because of my house mate’s access to the servers at the University, all of which were running some variation of a proprietary UNIX. That was a fairly early point in terms of adoption of the Internet.

That was spectacularly eye opening for me. I thought this technology and its implications were fantastic, and I believed that we needed to take it to as many people as possible. So, I took this notion to the company that I was working for, and they wouldn’t entertain it at all. They had no concept that anything coming from a group of bearded, sandal-wearing “ex-hippies” at a University could ever be viable in business.

So they said no and I said, “Thanks very much. I’ll quit and start my own business working on this technology.” Which was what I did, going on just about 19 years ago. At the time, it didn’t seem that the Internet would appear viable to the mainstream business world, the government sector, and so on.

Obviously, that changed, and the ethos and the culture of UNIX, which really came from the ’60s and ’70s in particular also rose in prominence, riding on the Internet’s coat-tails. In fact, even while AT&T was very heavily and visibly commercializing UNIX in the 80s, a great strand of UNIX development and culture was happening in the universities by that same group of hacky-sack playing, sandal-wearing bearded UNIX geeks.

These same people, literally, were the ones who built that layer of software that sits atop an operating system and makes it an Internet citizen: TCP stacks, Usenet transfer-servers and news readers, SMTP servers and mail clients etc.

Scott: What you’re describing is really a sort of rising counter-culture within the technology sector, while at the same time, a lot of it was actually government funded, especially by the U.S. Department of Defense and their DARPA project.

So right from the start, there was this sort of dynamic tension between the upstart nature of open source and the most established organizations, with both enabling each other. Even now, the same is true with some of the world’s largest tech companies contributing to open source and also reaping the benefits.

Con: That’s true, but notwithstanding the mutual enablement you mentioned, the spirit of information sharing going on at the time was completely contra to what was traditional in the proprietary world–certainly in the proprietary vendor UNIX world.

At the same time, there was a long period when a lot of the technologies that made the Internet possible had to be run on top of the operating systems that came from proprietary vendors: things like IRIX from Silicon Graphics (now SGI), SunOS back in the day, or Solaris as it’s called now from Sun, AT&T UNIX of course, IBM’s AIX, and so on.

To make these platforms good Internet citizens, you had to use this stack of software that allowed you to do sharing of files through FTP or Telnetting in from one system to another. Very powerful tools and techniques.

With time, the operating system portion of these Internet machines became less and less important. With the introduction of increasingly powerful commodity hardware, particularly the introduction of the 386 from Intel, it became possible to run 32-bit UNIX-like platforms on that commodity hardware. You didn’t have to spend tens of thousands to buy a proprietary server from companies like Sun, SGI, Motorola, or Hewlett Packard.

At the time, it became commonplace for $3,000 or $5,000 to buy something that you could run all your Internet software on. That all happened in the very early ’90s with the simultaneous growth of Linux and also the BSD platforms, when they eventually were ported to x86, particularly by a number of developers on the West Coast of the US.

Unfortunately the BSD world was hit with a very substantial lawsuit by AT&T against the University of California. It stymied the BSD system for several years, which was enough time for the Linux world to get ahead.

The introduction of commensurately powerful commodity hardware, running a viable and freely redistributable operating system was the shortest path to distance yourself from any particular UNIX vendor. You no longer had to acquire technology from a specific vendor to achieve Internet enablement.

That provided enough of a platform for these techie communities, which we called “hackers,” to thrive. “Hackers,” in the positive sense, as opposed to “crackers” who crack into systems and are malevolent. Hackers build systems, whereas crackers break into systems and wreak havoc.

The hacker developers in the very early ’90s helped propel this core of software that became known as open source, that made the Internet work. At the same time, they used the Internet as the mechanism by which they shared development responsibilities and ideas, in chat forums and so forth. IRC is a fantastic tool for development.

They also made this software available by distribution through the Internet. There is almost a genetic DNA double-helix strand between open source software on one side and the Internet on the other. The two cannot exist separately. You can’t functionally and competently build open source software in a meaningful way without something like the Internet, and the Internet could never have existed without something like open source software.

Scott: I’ve never really heard it framed that way, and it makes perfect sense. Up to this day, there are the IRC rooms on FreeNode, where depending on how big the project is, there might be 20 or 100 or 200 people who are just kind of connected to the room all day long as the software is being developed.

We talked to Justin Erenkrantz at the Apache Foundation, which of course is a core piece of technology for the Internet to even exist. Part of their ethos is if it didn’t happen in the mailing list, it didn’t happen.

The idea is that none of the development is happening in closed-off rooms. Maybe at IBM, they’re writing a piece of code that they want to link to Apache, but ultimately it all kind of gets cleared in the public eye.

It’s interesting to consider how things might have gone if BSD hadn’t been stalled. It might easily have become the dominant free and open source UNIX alternative. It’s got a very different license than Linux does, and you have to wonder how that would have affected the adoption of GPL and other things.

Con: It’s an open question, but I think the people who have looked at it fairly seriously would say that from a number of different perspectives and for a number of different reasons, Linux would probably still have come out ahead of the pack. One reason is the license.

It’s odd to think of it that way, but the license of Linux allows a large company to build software and to inject it into the Linux kernel, under the GPL, without fear of that software being taken up and used by competitors.

A prime example is that SGI contributed a developer-decade’s worth of code in the XFS journaled file-system to Linux under the GPL. At the time, the several of SGI’s competitors didn’t really have a serious, competent journaled file system.

If the target platform for the donation had been BSD UNIX derivative (say FreeBSD, or NextBSD), instead of Linux, it wouldn’t have been possible for SGI to have released the code under a BSD license, because competitors can pop up and say, “Ah, BSD license. Thanks very much for a fantastic journaled file system which we can embed within our own operating system. That’s just saved us a load of development. Now it makes us more competitive, and we’ve just leveraged your R&D. Thank you very much.”

So in many ways, the Linux license actually does assist in things like corporate code contributions. Because the potential competitors know that you can’t incorporate GPL code within software that you want to remain proprietary, it acts as a deterrent from purloining that code which various corporate or vendor bodies donate under the GPL.

That means that Vendor A can trust that Vendor B can’t jump in and grab the fantastic bits of contributed GPL code and inject it into their proprietary competing code and sell it in competition against Vendor A. That’s actually a great little warranty, if you like, which allows these disparate vendors to contribute that code into the common pool available for Linux.

There are hundreds of such vendors contributing code in different ways and in different extents, into the kernel, beyond the major players like IBM, Sun, Oracle, Red Hat, and so forth. There are a bunch of other companies that contribute other code as well.

As to why Linux took the lead; there are cultural differences between the Linux and BSD communities, too. Traditionally, the BSD world is much more buttoned-down and serious in terms of the acceptance of code contributions, although I think the Linux world has tightened up how some of its code is contributed nowadays.

Scott: And even though Linux has become dominant, it’s good that we have different platforms, including things like OpenSolaris. Variety is good, and I think there are risks in monoculture, even in something like Linux. It’s best not to put all our eggs in one basket.

Con: That’s true, although for decades, the industry has been looking towards having some kind of common platform. It did this with networks. If you go back 20 years, we had SNA from IBM, IPX from Novell, NetBIOS from Microsoft, and a dozen other competing network protocols. And along comes this open, unencumbered protocol which has also got, interestingly and importantly, open source reference implementations, that can be readily embedded into your own product range technology namely the Internet TCP/IP, and the UDP protocol suite.

Along these come, and essentially sweep everything else out of the way within 10 years. There is the corresponding potential for something like Linux, in connection with cousin alternatives like NextBSD, FreeBSD, and Open Solaris, particularly if they share enough reasonably-common, public, standard APIs. The desktops environments these are running, are pretty much standard across the board: things like Gnome and KDE are standard across these different operating systems, making them appear much the same to the casual user.

From the vendors viewpoint, there was therefore the potential for Linux to achieve something in the operating system space, similar to what the Internet achieved in the network protocol space; the Internet became the common standard that “everybody can trust, nobody needs to distrust”; the place that you could have a meeting of minds with your followers and your competitors alike. Linux could become the Switzerland of operating system platforms.

The Internet did that for networks. IBM could not really adopt DECnet, and DEC couldn’t adopt SNA. Well, as none of these industry players developed Linux, it becomes the neutral thing that they can all adopt to build products and solutions with, as long as they stick to the spirit of the community and the dictates of the license.

And you see a lot of that already in those areas where there isn’t an encumbrance (or expectacton) to adopting alternate operating system technology. Anything from mobile phones, to GPS systems, to DVD players, to Smart TV systems, to in-car computing systems. In all of these areas, to put it bluntly, people don’t necessarily expect to sit in front of a Windows interface; that expectation therefore doesn’t deter vendors from selecting Linux.

Users don’t expect to see a system menu with a Start button and the blue sky and green fields of an XP desktop in their GPS system, or on a phone or DVD player. All those product segments are very quickly becoming the Linux space, because there’s a relatively minimal amount of negative reaction from users when presented with that alternate user experience.

If they see something that performs how they want it to perform, that’s what they then use. Because of a number of different licensing and development costs and all sorts of stability reasons, many manufacturers are adopting Linux technology for devices. It’s obviously much harder for Linux to make headway into desktops, where people have been using a particular user-interface pattern for some years.

Scott: It also seems that eventually, economics will dictate change on the desktop. Once you get $100 computers, $150 operating systems become problematic.

Con: The netbook is that point of inflection for many people. At $200, you can go and buy one from major OEMs like Dell or HP. A $200 netbook is a more than “good enough” computer for many people. And at that point, you start asking questions about the viability of a $400 office suite or a $150 operating system.

And that’s probably the most likely path that increasing numbers of users will move, over time, to desktop Linux. It’s one of those things that might take years. After all, it took Unix decades to reach broad acceptance in the business community.

The same thing is true with the Internet. Remember, we’re not even 20 years into the Linux epoch.

Scott: Linux definitely seems to do very well on devices where you don’t place expectations on the operating system. TiVo is the classic example; many people don’t even think of it as having an operating system, let alone identifying which one it uses.

Con: It’s a tool; product makers need something that does the job well, at the best possible price, as long as you have an open and competitive market. Of course, there have always been questions in the IT space about that “open and competitive” aspect. Microsoft has gotten a lot of attention in recent years, and before them it was other major players, like IBM. People are always asking whether strings are being pulled in the background, strings that actually prevent competition.

There have been courtroom cases about this, but at the end of the day, it won’t really matter, because it’s impossible for any one company to maintain that “king of the roost” position forever. IBM knew this, and Microsoft probably also knows it as well.

And many of us who have been in the industry long enough remember the period when IBM had dwarfed all of the major players. The IBM universe was the universe, and if you didn’t do what IBM said, then you weren’t really playing in the IT game at all. That position has now been usurped by Microsoft.

My hope, which is probably shared by a number of people in the industry, is that if we do move to the post-Microsoft world, it won’t be controlled by a single, specific vendor. And if you look at it from a strategic point of view, realistically, the only platform that can deliver vendor-neutral, open, unencumbered computing for the decades to come, is something like Linux.

Scott: We’re sort of entering a new era. There used to be a lot of talk about whether open source companies can be competitive, and now we have begun to see a lot of acquisitions of open source companies by traditional companies. And we have begun also to see that when you buy an open source company, it isn’t exactly like acquiring a proprietary company.

Con: And that can surprise the people who acquire these companies, sometimes for what seems to be exorbitant amounts of money. They seem not to realize that, while they get the name and the trademark, the code base with most of these acquisitions remains open source, and is therefore harder to monetize.

And even if you buy the technology and decide to make it proprietary, that doesn’t stop someone else from taking the current version and continuing with the open source development. The right to fork the codebase is one of the intrinsic freedoms of open source software.

Scott: It seems that they are also acquiring the intellectual capital that is in the developers’ heads. If you buy MySQL, you get all of those developers, but the problem is that you have to actually keep them. If you’re Sun Microsystems, and you acquire MySQL, and then everybody leaves, what do you really have at the end of the day?

Con: That’s right, and it’s a non-trivial problem. Companies have to be very cautious about their interaction with open source communities. To date, I’ve seen only a handful of companies who have done this successfully, and generally, they have been companies that have been embedded within open source communities themselves.

After a non-open source vendor acquires an open source product, they need to convince the staff that the new parent values the way the staff has been working, and they want them to continue. The acquiring company has to work hard not to change the acquired company’s development culture and corporate culture.

There is nothing stopping, as you said, the developers from forming a separate company after the appropriate deadlines have passed in terms of contractual negotiations and non-compete clauses and so forth.

Once that time has passed, they can jump and form their own company with the same code and developer community. The user community will find out very quickly that they are the same smart people who created the technology in the first place, and chances are that those users will follow the original developers rather than the acquiring company.

This is something that those acquiring vendors need to be mindful of. They need to strategize accordingly and do their due-diligence. You can’t really do business as usual with open source software. It has to be done differently, but traditional companies can still be successful here.

Scott: IBM seems like they’ve done a very good job about figuring out how to derive a lot of revenue from things around open source, including services, implementations, and their own proprietary software that runs on top of Linux. They have also figured out where it makes sense for them to contribute to things like the Linux kernel.

They’re still a big corporation, and they’re still out there to please investors, but they seem to have kind of figured it out. On the other hand, Oracle’s database runs on Linux and some of that kind of stuff, but I wonder about the future of some of the technologies that Sun had been holding, like Java and MySQL.

I wonder what these things are going to look like two, three, four years from now in a company like Oracle, where you just have to imagine that there are tens of thousands of people who just fundamentally don’t get open source.

Con: I believe that many organizations, possibly most, embody the philosophy of the person at the top who runs it. IBM had major financial problems in the early ’90s, when it was losing billions of dollars every quarter. They brought in Louis Gerstner, who I can tell, even from an outsider’s perspective, did a fantastic job. One of the reasons he did a fantastic job is that he used to be a customer of IBM’s five years beforehand, when he was a fairly senior MIS player at American Express.

He got the rough side of IBM as customer, and he wanted to make sure that no other customers got that rough end of IBM. And he set about putting mechanisms in place to change IBM’s culture into the current one that IBM has. It was still a pretty big and byzantine machine, but it overcame much of the negative attitude and stifling process that it had 20 years ago.

It seems that Sam Palmisano is a master’s apprentice to Louis Gerstner, with much the same philosophy and the same outlook about how to deal with customers, competitors, and R&D, as well as what to invest in and how to make money by building solutions for customers.

That is a fairly different species from Oracle, where Larry Ellison exudes a very different type of personality from Louis Gerstner or Sam Palmisano. Oracle has traditionally been much more product and sales revenue focused, which makes their outlook and their positioning about how to use open source quite different from IBM.

Scott: On one hand, you sense that when these companies buy an open source company, they don’t understand entirely what they’re getting. On the other hand, you sense there’s an awareness, or at least this sense on these very large companies that they know they need to figure it out.

You see Microsoft dipping its toe into the water by contributing some code to the Linux kernel and some other stuff. Linus has even kind of defended them. It was interesting that Microsoft contributed integration components that let Linux run as a guest OS on top of their virtualization platform.

Immediately, lots of people shouted that it was self-serving. Linus responded by saying, essentially, “What do you think Dell’s doing when they contribute device drivers for their particular hardware?” That’s the whole point of open source: everybody gets to be completely self serving and in the process of being self serving, they build this great thing that everybody gets to use.

Con: That’s absolutely correct, and there’s no negative issue associated with that. The only reason for the pushback by the open source community is that in the previous decade or so, a broad mistrust of Microsoft has taken root.

In a perfect world, Microsoft becomes just another major industry player, like IBM, Oracle, HP, or what have you. In reality, though, many in the open source world still see Microsoft as having a bit too much control over what happens within the industry.

People will point to Microsoft and say they’re making mistakes and being silly and everything else. Microsoft are smart guys. They will sit down, and with mathematical precision they will itemize almost like a strategic battle plan for why and when they have to flip from having a positive or a negative stance on open source.

If there’s a mathematical point where they gain from distributing GPL drivers to allow for optimized hosting ofLinux on their virtualization platform under Windows Server, they will do it at that point. Not before and not after. And as you’re saying, that’s no different from how Dell or some other company would approach the problem.

Unless a vendor can visualize how they can gain strategically in specified areas by broadly supporting open source, they’re not likely to take a forward step. They will do it on an as-needed, case-by-case basis where it makes sense for their business.

If I were a shareholder at Microsoft, I would expect no less. The interesting thing is that, of course, Microsoft has run the numbers and calculated that they would be at a strategic disadvantage if they weren’t doing these injections of open source code and also normalizing relations with the open source community.

Scott: I don’t claim to know their inner thinking, but I would echo what you’re saying and observe that at some point, Sun Microsystems did something very similar. They had a proprietary Solaris operating system running on proprietary hardware, and at some point they decided to go with open source.

They did it very differently; they decided under Jonathan Schwartz to make a fundamental shift and get completely to open source in the time frame that they could, but they saw the writing on the wall. Proprietary UNIX systems were competing against freer, much lower cost Linux, which was just going to erode their position year after year.

IBM saw similarly that they had to get on board with Linux. They did the calculation and saw that they could still make a lot of money off of these other things, but that if they didn’t get on board and have their products running on Linux, their proprietary UNIX was just going to decrease in terms of share.

Con: All of these different firms have a particular strategic positioning. It also shifts with time and is impacted by the culture of the organization. Sun is a high-end engineering firm, and I imagine the high priests of the Solaris development scene looked down their noses at Linux and said, “Why on earth would we have to compete with this thing?”

But that’s an outlook which was there maybe 10 years ago or so, and now they can’t really make that kind of statement. The growth of quality and functionality within Linux far outstripped what you would get in terms of features and capabilities within Solaris, because of their different development models and licenses

And at the end of the day, you just can’t compete with Linux.

Scott: Right. You can’t compete with Dell and HP and all of these companies that are contributing to it. They are doing it in a very self-serving way, but you end up with a Linux that has an enormous number of drivers and supports a huge amount of hardware.

When Intel comes out with a new chip set, they contribute code to Linux to make sure that Linux takes full advantage of their new features and the chip set, while other operating systems, like Solaris, AIX, or HP-UX sort of schedule it. They get around to it when they get around to it.

It is very difficult for any one entity that wants to control all the code to compete with that sort of thing. We’ve talked a lot about these players that have been around for a long time, and then there’s this sort of new kid on the block, in the open source world, and that is Google. Google shows up and it becomes sort of the command line to the Internet.

It’s a text box and a button, and you don’t really have a lot of sense for how they are doing what they are doing but you get the sense that have figured out something with commodity hardware. They are not buying a bunch of Windows licenses. They are using some kind of UNIX-ish operating system.

They’re basically building a massive computer that spans the globe. They start not necessarily giving away their search algorithms, but they start contributing a lot of stuff to open source. They have GWT, they roll out Android, they’ve got a Chrome browser. They are talking about a Chrome operating system and these potentially very disruptive kind of things.

What does your crystal ball tell you about that?

Con: Google is in the position to have a very dramatic role in the next decade, for two reasons. One is that they have a fantastic name brand, which gives them the reach to maximize penetration into the mainstream. The average Joe or Jane is not that keen on acquiring technology, because technology is complex and potentially even scary.

Because of this, they want technology to be incredibly simple and from a big name vendor, so they feel confident that they can get it to work. Google can deliver that name brand clout, and of course they have enough money to be a serious competitor to anyone in the industry.

But, the question to ask is, “If you were Google, what is your end-game strategy? What are you trying to achieve?” From Google’s perspective, my guess is that they want to have computer terminals in as many places as possible to access ‘their’ Internet. So you will get your smart browser plugged into simple operating system terminals. You might have five of them in your household: you might find one on your desk, one on your phone, even one on your more powerful netbook, or notebook, or what have you.

But at the end of the day, they are looking for the standard utility model of computing. That’s where they have immense strength, because more than anybody, they know they have to maintain and make money by running hundreds of thousands of servers in the backroom and delivering a consumer-grade interface to as many people as they can.

Google will inject money into every possible technology or process they can, to help to remove the roadblocks towards that reality, and to erect bridges to make sure that that reality happens. Browsers have gotten pretty good, and increasingly, they’re all coming to the party with regard to supporting standards.

Google wants a complete operating system in the browser. They want to be able to run Quake 3 in a browser. Your browser does currently not run Quake 3, so Google make a browser that can, because they do not (in reality) want an operating system; they want the whole world to fit within the browser.

Of course this is the exact antithesis of what Microsoft wants.

Google also recognizes that they can increasingly tell people that they can use the Google web ‘operating system’: office suite, and mail and calendaring suites, all of which are online, free, and available for Mac, Windows, and Linux, all on your web browser.

However, they also still note and understand the industry’s maxim that “he who controls the desktop operating system can control the universe”, and this is what Netscape found to its detriment a dozen years ago when Microsoft decided to compete in the browser space. Microsoft controlled the desktop operating system, and so it could do things in such a way as to outmaneuver competitors’ applications that are running on top.

But from Google’s perspective, the end-point would be essentially zero-cost, friction-free software, on as many devices as possible, all linking back into the Googleplex in some way, shape, or form. One way to do this, of course, is to develop a current-generation OS that they can make available to as many low-cost, web tablet manufacturers as they can possibly take on board.

And these manufacturers will be making computing devices that will be increasingly low-cost and ubiquitous, so it will become essentially irrelevant what OS you’re running on this web tablet, as long as it is consumer-friendly, ultra-low cost, and can get you into the Internet, which of course means into Google’s arms.

Scott: I totally agree with your analysis, when I look at Android targeted at handheld devices, and maybe netbooks, as well as their plans around Chrome OS. I also have to think about HTML 5, which as you mentioned lets you run Quake in a web browser with storage, offline capabilities, and a lot of other stuff.

It’s a lot of separate efforts, but they are obviously all converging on this goal. I don’t know if this is the right characterization, but Google could become the next Microsoft to some degree. It becomes a ubiquitous platform. It’s not necessarily on your device, because whether your device is running Chrome, Android, or some other OS, it’s going to have a browser, and as more of these experiences become web based, Google could control the entry point to everything.

Con: As long as they are perceived as roughly living by that credo, “Don’t be evil,” that will work. I had this discussion a while back when someone asked me, “what will stop people from using Microsoft?” I responded, “How many years’ worth of Microsoft documents have you got in your PC hard disk?” He said “Like 12 years worth.”

And there you have it. How easy would it be to convert all those documents into something different? You can’t do it, because there are all of these macros, complex embedded formatting, etc. It can’t be done, so many are stuck with using Microsoft Office forever.

The corresponding question is, “How easily can you shift from using Gmail to Yahoo mail?” It would be a bit of a pain, but with a couple of hours’ work, you could bring across all of your contacts and copy down all of your mail.

And again, how easy would it be to use an alternate search engine? Well, you can do it pretty much instantly. You can start using Bing, or you can start using Yahoo Search. That’s the key difference between what Google does and what Microsoft has done. It’s a lot harder for Google to achieve serious lock-in.

They remain where they remain for as long as people perceive them to be doing a reasonable job, possibly a good job, and as long as people perceive them to be complying with their own personal ethos of how they think a business should work. Otherwise, it’s very easy for any consumer to say, “Well actually, no. I’m not using anything from Google. I get my mail from Yahoo. I get my online storage from Microsoft. I use FaceBook for my chat, and I don’t use Google search.”

The key reason, of course, that this is ease-of-exodus is possible is that to a large extent, Google understands that this is a double-edged sword for them. In order to be successful and to accelerate the adoption of these kinds of technology, Google has to use open standards. And by definition, a vendor cannot achieve lock-in through open standards.

So for its talk protocol, for example, it uses XMPP, which is the Internet-standard chatting protocol, the open source Jabber chatting platform. They don’t develop their own proprietary discussion protocol. It’s the same for mail, calendaring, and the list goes on and on.

Now, at no point are they in a position to be able to fully dictate terms on their software. They can say, at any point, “Well fine, we’re not bringing out new versions of this software.” But if their software is of enough interest to enough people in the community and to other vendors or technology companies, these will continue to maintain it, or provide viable migration paths to open standards-based alternatives.

Once again, it’s all made possible by open source software that uses open standards. This combination can’t put Google in any more of a strategic leverage position than any other vendor; as long as they are doing a good job developing Android or Chrome OS or the Chrome Browser, you will probably continue to come back to them.

At that point, it becomes just a standard consumer choice option. If Google does turn evil, consumers can say, “Look, Google’s turned evil, but I really like what they are doing, so I’ll go with them.” Or “It’s just as easy for me to go and use the newly named Bob’s Browser, forked from Chrome, from the group over in Cincinnati which is doing fantastic work. I don’t like Google any more.”

That’s the key fundamental difference. Google wanted to build a a viable client-side platform quickly and to make it as broadly-acceptable as possible, which introduces that double edged sword for them to contend with: Google has to use open standards and open source to maximally gain penetration, on the one hand. On the other hand, it can’t really achieve consumer lock-in.

Scott: You see a sort of meritocracy already playing out. When I go to a conference like OSCON, where it’s just all open source people there, I see an enormous number of Macintoshes, and I see an enormous number of iPhones. I know that the people there know that Apple is not an open source company. In fact, it’s very secretive. It likes to kind of surprise the marketplace with new products and this kind of stuff.

Yet people just want the products so bad that at the end of the day, they want the products for what they do. They want the iPhone for what it does. But there isn’t an enormous amount of lock in with an iPhone. It’s not like they couldn’t go buy a different phone tomorrow, and probably move their contacts over and move their music over and be in business.

Con: Apple is in a very enviable position. Although at the end of the day, Apple builds proprietary software, proprietary firmware, and proprietary hardware, people want what they offer.

I have been, for the past 13 years, a user of the Opera web browser, which is a proprietary piece of software. Now it’s free, although I used to have to pay for it. I love that web browser, but if Opera was not adhering to what I would perceive to be good business practices, it wouldn’t take me too long to move what I’m doing away from Opera onto Firefox, or onto Chrome OS or what have you.

So that’s how I would judge Apple in this instance. I don’t really use Apple operating systems, but people who do use them use Firefox and they use OpenOffice.org and whatnot. So from my perspective, these users get the OS environment that’s suited to them, but what interests me is that as long as Apple adheres to open and Internet standards, so that I can share material and can correspond with Apple users, I’m OK with Apple doing what they do, as they don’t have a chokehold on our industry in any real sense.

Scott: Well, that’s probably a great place to wrap up. Thanks for taking all the time to chat.

Con: Thank you for the opportunity. It’s been wonderful.

Bookmark this:

• Moblin and its transition from Intel to the Linux Foundation
• Linux as a multi-purpose OS across device types
• On the limitations to how many devices people really want
• Data synchronization capabilities and limitations
• Integrating open source ethos into a large company’s business model
• Public acceptance and skepticism of commercial involvement in open source

Sean Campbell: Dirk, give us a little bit of background on you, your role at Intel, and your experiences with open source.

Dirk Hohndel: I’m the chief Linux and open source technologist at Intel. I work in the Open Source Technology Center, which is part of the Software and Services Group. I’ve been doing free software and open source for about 20 years.

I started while I was at university, when I had the incredible fortune of getting involved in Linux at the end of 1991, with version 0.11. That made me one of the very early people who started contributing and getting involved.

I have been around free software and open source ever since, through a number of jobs. I have run a few open source projects either myself or together with other people, and I’ve been involved in quite a few other open source projects over the years.

I was CTO of SuSE from 1999 to 2001, where I started and ran their professional services business. I co-authored the first ever published book on Linux, and I’ve written a bunch of articles.

At Intel, my role is very much about making sure that Intel internally really understands open source and acts in a way that is compatible with the community. My charge is to help the company be a positive member and contributor in this space. At the same time, I’m trying to make sure that the community understands what we’re doing and where we’re going.

Sean: What can you tell us about the current state of Moblin, since it has moved under the auspices of the Linux Foundation? Is the focus likely to be moving away from Intel’s original vision?

Dirk: Moblin was conceived inside Intel, as a project that would bring a different user experience, to use the term very broadly. We identified the issue that there were no OSs that were focused on being connected to the Internet and using the services that are out there, such as social networks and so on.

We saw the need for an OS that could achieve that in a very natural way, hiding all the complexities from you, without being too restrictive. We wanted a user experience that is intuitive, easy to use, and visually interesting, while at the same time retaining the freedom that a Linux-based OS gives you.

It’s actually been almost three years since we began this effort now. We’ve gone through a number of iterations of how it should look and be implemented, and we eventually came to the conclusion that we really had to roll our own OS from scratch.

We felt that a very large proportion of the underbelly and infrastructure needed to be optimized for this use case and type of device to be really successful. We recognized that we couldn’t just create it as a patch kit on top of an existing Linux flavor; we really needed to develop it from the bottom up to focus on this environment.

Sean: How did the transition from Intel to the Linux Foundation come about?

Dirk: One of the things that happened is that we ran into a lot of people who were hesitant to contribute to Moblin because they saw it as Intel’s project. Even though they understood that the project was very important to us, they had concerns about what would happen if Intel lost interest.

We realized that the solution to that perception issue was actually pretty simple. From our perspective, Moblin was always an open source project, and we had planned from the beginning not to make it an Intel thing. To communicate that to the community, it seemed wise to bring it under the auspices of a neutral party that is beyond any doubt in the Linux community.

The Linux Foundation was a great choice, since it has a wonderful track record as being neutral and having the best interest of the Linux and open source community in mind.

It’s odd that this move created some fear in the market that we would disinvest; it’s actually the opposite. Over the last six months, we’ve increased our investment and put even more resources toward targeting different sub markets. So we’re doing a lot of work in this space, and I think we’re pretty close to having the first product based on Moblin in the market.

Sean: What is the status of Moblin 2 at the moment?

Dirk: I’m sure you’ll understand that I can’t disclose any of those plans–the marketing people would not find that funny–but I think it’s fairly close. We are currently releasing weekly or bi-weekly updates to our latest beta on moblin.org, so people can play with it. We are very much in a code freeze space, so we are not taking any new features right now. We’re just fixing the bugs that we see.

In parallel with that, we have actually started development on the next release, because of course, software is never done. You always have the next 20 things that you absolutely want to integrate, but we are freezing and serializing Moblin 2 right now.

I’m very excited, and I’m running the latest beta build every day. I think it’s a very impressive user experience, and it certainly fulfills our goal of making it easy and intuitive to do what we perceived and what our early testers perceived as a typical task that you would want to do with a netbook.

I should point out that the current version of Moblin 2 is focused on netbooks. We would make different decisions if we were targeting full-fledged desktops or touch-based devices like tablets.

Sean: If you consider portable devices that range in capability from maybe a smart phone at the low end to a netbook at the higher end, with others in between, like maybe the Kindle, how does Linux serve that full range?

Does it scale up to laptops, or do you think it has a sweet spot on that scale that maybe is the best place for it?

Dirk: If I put on my aluminum foil hat for a moment, of course, it’s the best operating system for anything and everything you could ever dream of. [laughter]

Having said that, and taking off my aluminum foil hat again, I think the market has been very clear over the last few years, in terms of where the majority of end users do and do not prefer it over other OS’s.

This is my personal assessment, because I don’t think there is a conclusive study on this anywhere, but my personal feeling is that for a general purpose PC, where users assume they will have a full office suite, image processing, and integrated UI, Linux has a much harder time succeeding in the market. That is the space where Microsoft Windows is the market leader, and Mac OS is a strong contender.

But as soon as you specialize things more, and as soon as you start using the term “devices” instead of “computers,” you see Linux-based OSs being much more successful. The Kindle, for example, runs Linux. It’s an extremely specialized device. Anybody who has tried to use the embarrassing attempt at a browser on the Kindle knows that this is not, by any stretch of the imagination, a general-purpose device.

Sean: Oh yes. I used that once. [laughs] That will probably be the last time.

Dirk: Consider things that people often don’t think of as computers. Think of your wireless access point, your cable modem, your set-top box, and your DVR. Those are very lightweight connected devices, and they almost certainly run Linux. In the area of cell phones, Android is obviously a hot topic right now, and it is also Linux-based.

That means that a key question around the netbook concerns whether it is a general purpose computer. To some people it is, and so you see Microsoft Windows running on more and more netbooks. To others of us, the netbook is a special-purpose device designed with the goal of being very affordable yet very robust, as well as easy to carry around. It targets surfing the web, connecting with friends, and social media networks. You might use a netbook for playing some media content, but not for general computing, and that is the space that we envision Moblin for.

Moblin fulfills a fixed set of purposes, and it does so very well, with a great user experience. It is distinct from the full PC model, with the goals of being light, easy, visually appealing, and fast. That distinction is why I think Linux makes a lot of sense in this space, because of how configurable and powerful it is, and because of how much you’re able to tailor it for your purposes.

Sean: On my desk right now, I have a MacBook, a Windows 7 T400S, an iPhone, and a Kindle. Obviously, that’s different from the average person whose computing equipment may consist of a phone and something bigger than that. There are limits to how much a person wants to carry around.

Do you see that threshold as being at maybe three or four devices? Maybe people want, on average, a smart phone, a netbook or other specialized computing device, and then a higher powered machine for productivity apps, editing photos, or whatever. How do you see those limitations shaping the future of a project like Moblin?

Dirk: I do not speak for Intel at all when it comes to big picture vision questions like this, but I can give you my personal opinion.

I think there is a continuum between the tiny phone through the bigger phone, the smart phone, the really scary big smart phone, the MID, the tiny netbook, the slightly bigger more powerful netbook, then the consumer thin and light CULV kind of devices, the notebook, the desktop replacement, the desktop, the gamer machine, and the Mac Pro workstation with two Xeons and 12GB of memory.

Where people draw the line in terms of which and how many devices they pick on this continuum is very much dependent on who you’re talking to. I have that eight-core Mac Pro, a Core2 Duo-based laptop, an Intel Atom-based 10-inch netbook, and a Celeron-based seven-inch netbook. I have a couple of Android phones and I have a Motorola Razr. There are seven or eight devices on that continuum, and even though I hopefully never have more than three of them with me at a time, some people would find that lineup to be absurd.

That’s the point, though–people make different choices in this area. I think that, as a vendor, it is important to embrace the fact that people like choices and to provide them alternatives that let them find the devices they like best.

For example, when Asus first started talking about the Eee PC two years ago, a lot of people told them that no one wants a diminutive laptop. Guess what? People were wrong, so I’m very happy that we invested in this and thought it was a great idea to provide the platform to do it on.

I don’t think it is our job to tell the end user which device is for what and how to use it. It turns out that they usually have their own ideas, and our job is to facilitate that.

Sean: Where do you see data synchronization playing into that vision? Sugar Sync now supports Android, and some additional solutions like Ubuntu One are trying to build out a sync story. But if a user really wants to live in a world where they can sync all their devices, they have some obstacles.

For example, a user might have a Windows or a Mac box, a mid-tier device like a netbook, and a phone. If they want even something as simple as a picture to show up on all of those devices, and to have an edit made on one show up on all of them, they are probably going to be frustrated.

What are either Intel’s or your personal thoughts on that, or what do you think the community is doing that’s interesting around that issue?

Dirk: This is a really hot topic, although I don’t think synchronization today is so much bound to your OS, even though there are a couple of providers who have a very jail-focused mentality in this space. They basically refuse to interoperate with devices that haven’t been made by them, which I find to be a very weird approach. I think that long term, that’s not going to be successful, but those companies are in the minority.

We do see that there are a number of services out there that are by design cross-OS, and by design simply a way to store and share data. I think these are the services that will define how we interoperate between our devices.

For example, and without any endorsement in this context, the way I store my personal contacts is that I have a central server that I connect to all of my devices. Whether it’s the big Mac, the Linux-based laptop, or my cell phone, they all get the contact information from one single place, and for me right now, that’s Google Contacts.

There are other services that would allow me to do the same thing, like you can use Funambol and create your own server. I think we’re going to see a number of different offerings tailored toward different user groups that provide the ability to access data in all of its forms across devices.

One consideration is that you have to have the ability to do this online and offline. If you don’t have the ability to do local work with your local data, you’re going to fail. We still have a terrifyingly bad existing wireless data infrastructure, by and large.

Here in the US, we pay among the highest rates for all wireless data service, and we have disappointingly spotty coverage in most places. If you don’t have the good fortune to live in Portland where you have WIMAX fast wireless wherever you go, you turn a corner and you’re back to GPRS, or to nothing as soon as you leave the metropolitan area.

Your application and data need to live on the device, and they need to sync with a central entity if and when they’re connected.

Sean: I agree that there are different solutions for those of us who are technically inclined, but would you also agree that right now, no one is fully solving that problem because there is a somewhat jailed model where some of the sync platforms exist?

In other words, if you have a Linux-based mid-range computing device, a higher end Mac device, and a phone from another provider, there is no proverbial grandma solution out there.

Dirk: I think my friends at Google will disagree with you, but I understand your concern. There isn’t a solution that is totally and seamlessly integrated out of the box with all of the different OSs, and market forces make that a pretty difficult problem.

It certainly appears that there are some tensions between the big players. Without commenting on any of these situations, anybody who has read the press in the last couple of months knows that there are some disagreements between, say, Microsoft, Google, and Apple over who controls what in this world.

Scott Swigart: I’d like to touch back on something from earlier in the conversation, which had to do with integrating open source thinking into a large company’s way of doing business. Of course, this isn’t at all unique to Intel.

Microsoft is trying to raise its open source IQ, Sun Microsystems worked hard at it for a number of years, and IBM has figured out to some degree where open source fits into its operations. Oracle is going to have a bit to digest around that, because they’ve brought on a lot of open source assets to a company that traditionally hasn’t had that.

In general, what have you found to be some of the steps along the path of a software company successfully embracing open source?

Dirk: I think there is a common problem that over the years, large companies get into patterns of behavior where assumptions are made about how problems can be solved, how interactions with the outside should happen, and so on.

All of that dictates cultures that range from an open approach where you publish your program manuals and you invite developers to join you, all the way to the secretive approach where people need to sign NDAs even to be allowed to see the contract that they have to sign, in order to be allowed to develop software using another NDA that is only available under NDA.

There are different approaches, and most software companies come to open source as a result of the value they can get from having open source components as part of their product or to use in conjunction with their product.

Very often, what happens is that the people who make the decision to go with the open source components look at this and say, “So what does it mean for us? How does this change the way we act?” The organization below them, meanwhile, tends to be very resistant to change.

Scott: How does a company address that successfully?

Dirk: The challenge is that if you want to gain a commercial advantage using open source components, just as with everything else that you do, you have to go beyond just following the copyrights, the licenses, the laws, and the rules. You have to understand them not only on a legal level, but also on a conceptual level that can inform you about how to successfully interact with the people involved.

Say you want to interact with Microsoft. That means you have to follow the license that Microsoft gives you and the rules that they put around their SDK. If you want to have a successful relationship with Microsoft, you most likely need to act in a way that Microsoft doesn’t find offensive.

The very same thing happens with the open source community. You need to follow the letter of the license and the copyright, but you also need to act in a way that is compatible with the spirit and the intentions of the community. This is a learning process that tends to take a while, because it’s not only a couple of decision makers who need to understand it.

It also needs to become the ingrained behavior of the company as a whole. A lot of companies come from a very proprietary-focused world, where everything is under NDA and based on company negotiations rather than developers freely talking to each other. Therefore, it very often happens that while the company as a whole, says, “Oh, yes, this is the way we want to go,” a lot of people in the trenches don’t understand it and have essentially a different approach to solving the problem.

Scott: How does that scenario play out at Intel?

Dirk: Over the past eight years that I’ve been with Intel, I think the company has changed very dramatically in the way it interacts with the open source community, and I think we have done very well. We are now one of the most respected vendors in the space, and we actually received an award earlier this year at the CEBIT Fair in Germany, as being the best hardware firm to work with.

We have a lot of very highly respected developers from a number of open source projects who work at Intel. I think most open source projects perceive us as being beneficial and a good force to engage with, which may have not been the case 10 years ago.

This is a process that takes a while, and that takes commitment from an executive level down, but it is something that I think each and every company can be successful with. I’m very excited to say that Microsoft certainly has taken some great steps in this space. Their cooperation with the Samba team over the last couple of years has been very successful and has had very good results in improving interoperability, which helps everyone. Their contributions to the Linux kernel a few weeks ago helped Linux be a better guest in the HyperV environment.

There are a lot of areas where Microsoft has their own open source projects, and different companies have different degrees in which they progress on this path from being completely proprietary to understanding the heterogeneity of the world and successfully engaging with the open source development community.

Scott: Particularly if you go back three or four years ago, it seems that many areas of the open source community had a very suspicious and skeptical view of commercial companies. When a company decided to embrace open source, they were at great risk of having any misstep they made seen as a “gotcha” moment.

There were always people waiting to dismiss them as big, evil companies that weren’t really serious about open source. Do you feel that enough companies have now changed their direction and stance toward open source that the degree of suspicion has lessened? To what extent do companies still need to be cautious about that issue as they start down the open source path?

Dirk: Let me give you a slightly oblique answer and then bring it back on point. I think the Daily Show’s Jon Stewart is actually one of the very few people on TV who asks the hard questions of people who need it, and something he said a few years ago has stuck with me. He said that there are very few people with truly extreme positions in the world, proportionally, but they seem to be the ones dominating too many conversations.

I think it’s often true that, whether you’re talking about the public debate on health care right now, software development models, or the actions of companies, the people that the press reports about and who get noticed tend to be the people on the extremes.

The pragmatic moderates that just want to get work done are too busy to spend hours and hours screaming and yelling and telling the world how terrible it is. Whether it’s politics, economics, or technology, there is always a very vocal minority with extreme opinions that get a lot of attention even though they do not represent the consensus in the community.

I think that the majority of the people who do the actual work of open source, as opposed to the people who have all day to blog and comment on Slashdot, are extremely positive to each and every company that comes around and contributes. They try to help them around the pitfalls, and they try to make sure things are done right. By and large, they just try to embrace anybody who wants to join the open source community.

I’ll give you an example. Gregg K.H. at Novell is doing a tremendous job of embracing companies that want to have Linux drivers but just don’t know how. If you have a driver that doesn’t quite follow Linux standards, that is kind of awkward, ugly, or broken, he’s willing to put it into the Linux staging tree as a starting point. He has a bunch of developers who will take documentation, under NDA if necessary, and work to create the drivers and bring more companies into the fold.

The same is happening in many other projects. There is a very strong outreach there, of trying to get more and more people involved. There are a devout few who will complain and whine and be unhappy with what’s happening, but the reality is that the majority of the people actually doing the work are very excited to get the companies engaged. They’re very tolerant of the fact that there is a steep learning curve, and they’re willing to go out of their way to get as many companies involved as possible.

Scott: We’re running a bit low on time, but that’s a great place to end.

Dirk: Thanks–this was a fun conversation.

Bookmark this:

• Freespace products and the air pointer market
• Bringing order to the expanding world of digital content in the living room
• Building new devices for human-computer interaction
• Distributing open source libraries to spur innovation
• The math and alorithms behind motion tracking and control
• The Loop: a next-generation pointing device

Scott Swigart: To get us started, can you please take a moment to introduce yourselves?

Chad Lucien: Sure. I’m the vice president of Freespace products here at Hillcrest Labs, and my responsibility is the general business management of the Freespace division. Hillcrest Labs has two different product groups. Ours is centered around air pointing and motion control technology, and the other business unit is centered around our application software platform for consumer electronics devices.

I’ve been in this role for about 18 months, but I’ve been at Hillcrest for just over five years, in a mix of strategy and business roles.

With respect to the Freespace division, as the general manager, I’m responsible for setting the direction of the business unit, setting the product strategy, and engaging with our customers and partners at various levels.

Parag Sheth: I am the vice president of corporate marketing at Hillcrest Labs. I have been at Hillcrest for about three years now. I’m primarily responsible for marketing, but I also run business development in Europe.

Scott: Give us a little bit of background about what the Freespace products are.

Chad: Freespace is a six degrees of freedom motion-control solution. The primary application that we have been marketing toward is air pointing remote control. We have customers such as Universal Electronics, which has designed and is in production with a couple of different remote controls that use the technology. They’re marketing their remote controls to cable, satellite, and IPTV operators as well as consumer electronics companies.

We also license that technology to Logitech, and so Freespace is being used in the MX Air mouse, which is effectively a motion sensing piece of software that interacts with off-the-shelf MEMS inertial sensors and creates one of the best air pointing solutions available on the market.

It’s incredibly accurate, and it’s very stable from a user perspective. As a user, you can use a device enabled with Freespace in any orientation. So if you’re laying down on the couch using a Freespace remote control to watch television, you don’t necessarily have to point it at the screen or hold it in any particular position.

Whenever you move your hand up towards the ceiling, the cursor will move up on the screen, and every time you move left, it’ll move left on the screen. You can be holding it upside down, and it will still work the same way.

Freespace is also licensed by other companies that use the technology in their products, including Kodak, who use Freespace in their Kodak HD Theater Player, and ZillionTV. Freespace is also used in our own Loop pointer, an in air mouse for the television, which we should make sure to discuss at more length.

Scott: Talk about some use cases. I think Wii remotes really made this type of thing mainstream. It seems like before that, this technology was a little more special-purpose.

You mentioned cable operators rolling it out to their customers, so I could see it being used to navigate menus and things like that, but talk a little bit about some of the things that somebody like me might be using it for versus maybe some of the more exotic uses.

Chad: Absolutely. The real impetus for developing the technology was that we saw that there was going to be a need, which has since played out, to control vast libraries of content on your television set sitting in your living room.

As the team here thought through what the consumer interaction was going to be with those large content libraries, we realized that the traditional up/down/left/right control mechanism for remote controls coupled to text lists and hierarchical and grid based menuing systems were not going to give consumers a good experience.

When you start to present tens of thousands or even hundreds of thousands of different pieces of content, you need to optimize the efficiency of the interaction between the user and the content source.

We chose pointing coupled to a visual interface as the optimal method. That’s because with visual imagery, you can actually put much more information on a screen at one time than with text.

Imagine if you put 50 DVD covers on a screen. The amount of information you’re conveying relative to what’s typically in a VOD interface of maybe eight or 10 at the most text-based titles is quite significant.

But then if you put 50 different selectable items on a screen, trying to wade through those arrow by arrow, click by click, you may have a dozen different button clicks to get to one particular piece of content.

Abstracting that interaction and using a pointer, one simple movement and a single click can allow a user to access any piece of content on the screen. We really see that as the optimal way to bring Internet delivered and video on demand content into the living room.

Scott: I can relate to that. Spelling out the name of a show on a TiVo, you have to go up, down, left, right, letter by letter, and it’s very cumbersome.

Like you said, the home entertainment experience is moving in the direction that, year after year, there’s more and more content at your fingertips available through your TV. There’s really an explosion in the amount of stuff that you have to navigate.

Parag: Right, and the other part of that is the advent of Internet content coming to the television. Because Internet content is largely nonlinear, having an up-down button on the remote would not be sufficient. The presence of many different kinds of Internet content plays a big role in making pointing a more appropriate solution for the TV screen.

Scott: Are there other, more exotic uses? I am thinking about the computing platforms in the movie “Minority Report,” where people are directly interacting with a lot of information. Are there any specific case studies that you can talk about?

Chad: There’s a company called Kopin that has announced a product called Golden-i, which is a head-worn computing device running a full PC inside. They’re using Freespace as a module embedded into the headset as a head tracker. The user looks into a monocular display, and they can actually use their head movements to pan around the display, as one application example.

There have been several other companies that are interested in integrating our modules into other wearable computing systems. One area where we’ve seen a lot of interest is by the military, which wants to build wearable computers for Special Ops, where people may not have their hands free to touch a computing device.

They’re using multiple motion trackers on the lens so the wearer can interact with the computer and translate hand signals, for example, back to the rest of their group. We’ve seen plenty of interest in other areas, as well, including the medical community and the fitness community.

Once you have a full six-axis inertial tracking system, there are a lot of things that you can do with that information. Our product strategy is to make integrating the technology into any target device as easy as possible.

One of the ways that we sell Freespace, as I mentioned, is as a module, which is actually about the size of a US quarter. You can purchase that module and integrate it into your own device and software.

Scott: It looks like you’ve developed an open source library to make it easier to develop against your hardware technology. Talk a little bit about that piece of it.

Chad: Our business model is to license the Freespace technology, so the more remote controls, game controllers, wearable computers, medical devices, etc. that are produced using our technology, the better off we are. Initially, we launched Freespace as a pointing solution, and it was, effectively, a mouse replacement. It’s relatively easy from a development standpoint to integrate a USB HID mouse into any range of platforms.

But, over the last several years, we’ve added a lot of new functionality to that. For example, we’ve added linear acceleration, angular velocity, and angular position measurements. We’ve also enabled the ability to remotely upgrade firmware in the field.

These additional functions are not specified within the typical USB HID, so we’ve created this open source library to include all of the unique Freespace-oriented functions, as an enabling measure.

Any developer who’s used to working with libraries for Linux or drivers for Windows platforms should be able to take that information, incorporate it into their driver environment, and quickly integrate it into their systems.

Scott: Why did you take an open source route versus releasing a proprietary SDK for developers to use, which is probably a more conventional approach, in many ways?

Chad: We started to see an increase in the interest from companies doing lots of different and unique things. I think Kopin’s a perfect example; we had never really envisioned a device like their Golden-i product.

We felt that releasing this into the open source community would enable us to really construct and capture the benefit of developers who are out there creating new things that we had not yet conceived of.

As with most open source projects, our hope is that this will give our customer base access to a lot of the creative thinking in the broader development community. We hope that if somebody brings up a new application for Freespace, they’ll go ahead and contribute that code back, and the next customer will be able to pick up on it and use it.

We are also seeing that the device maker and the application softwar