• Building limitations into Maven that enhance consistency and efficiency
• Encouraging a paradigm shift in place of incremental change with new tools
• Fostering collaboration, even among competing organizations
• Avoiding the temptation of trying to fill every role at a young company
• Bridging the gap between various programming languages with future versions of Maven

Scott Swigart: To start, could you take a minute to introduce yourself?

Jason van Zyl: I come from a physics background, but I’ve been developing software for almost 20 years. When I was working in printing and publishing in New York, about 14 years ago, I started getting into open source, and I’ve been making my living from open source for almost 15 years now. It’s culminated in working on Maven and turning everything Maven-related into a product line, and that’s where I am today.

I’ve been part of the Apache Software Foundation for almost 10 years now, and I started the Apache Maven project in 2001. The oldest version was released in 2002, and then Maven 2 was released in 2005. In 2007, I started Sonatype to help invest in the community. In 2008, we became a product-based company and got venture funding.

Scott: My understanding of Maven is that it’s a build system. Talk a little bit about Maven, what it does well, and your thinking about the need to build a better mousetrap in this area.

Jason: To start, I would say that the largest competitors to Maven are large ad hoc systems that are created inside organizations. The primary difference between Maven and something like Ant or Make is that, while those are great tools, there’s no pattern or structure to how you actually use them.

Maven goes a step further, by bringing standards and patterns to a build system: most organizations tend to fall into a set of patterns for developing applications. For .NET, there are standard patterns from Microsoft for building applications. If you’re using Java, there are standard J2EE patterns and there are even design patterns for creating applications.

Maven takes that concept and applies it at the development infrastructure level. As a result, if you were to look at 100 projects built with Make or 100 projects built with Ant, you would find that they are very different. If you look at 100 Maven projects, they’re almost entirely identical in structure. While there might be differences in how they are customized, at a very basic level, if you know how to build one Maven project, you can build all Maven projects. Look at 100 different Ant projects, and each project is a unique “creation”. Over time, these 100 Ant projects will diverge from one another.

Scott: Talk a little about how that plays out, in terms of value in day-to-day operations.

Jason: If you were to take someone off the street without any experience and show them your Maven build internally, 99 times out of 100, they’d be able to build it on the first try.

Maven’s approach addresses some of the problems that came from the explosion of tools that were used in development infrastructure. It had become very hard, even within the same organization, to have a standard, cohesive structure shared among all projects. Even within a single company, you would have inefficiencies. Differences would start to emerge over time with one group using one project structure and build tool and another group using a different structure.

If you expand the perspective, and look at collaboration between different organizations, these inefficiencies become even more apparent. We have more than 2,000 very large clients who talk to one another as part of our technical advisory board. They use the same terms about how to build, test, and deploy software, because Maven provides a platform upon which other tools can build.

It’s impossible to build a platform on top of an ad hoc structure, whereas it’s very easy to build a platform on top of something that has a consistent model. That’s where Maven is different than most other build frameworks or build tools. It has a standard set of lifecycle phases that it passes through. It isn’t just a random assortment of tools, it is an entire factory, and our products and approach provide an end-to-end solution for software engineering.

Every component that is made with Maven is made in the same way, whether it’s a .NET assembly, a .jar file, or a .war file. You step through the same phases of compiling, testing, gathering resources, doing integration testing, and installing or deploying into a repository.

Many of the core ideas for Maven come from “A Pattern Language” by Christopher Alexander. He was an architect who tried to capture the patterns that made cities productive – patterns that enabled a healthy city infrastrucutre, where people can start businesses, communicate efficiently, and so on. I took Alexander’s ideas and applied them to questions about how you construct software and how you develop at an infrastructure level.

When I arrived at Apache 10 years ago, I worked on five or six different projects, and we were finding that people weren’t volunteering to build the software. They would show up, and then they couldn’t build it. We kept losing volunteers, because they didn’t want to spend their time messing around with the build. They wanted to devote their energy to actually creating code.

That’s when I took the version of Maven that I had had for a couple of years before that and checked it into Apache software foundation. It started in the Turbine project, which was probably the first Java framework that was ever created, long before Struts. It graduated into its own profitable project at Apache in 2002.

Scott: The first time I bumped into design patterns, which seem to be at the core of what you were talking about, I was at a software conference down in San Francisco, and one of the gang of four was giving a presentation on design patterns.

He was talking about how they were sitting around lamenting the notion that Smalltalk developers seem to solve the same problem the same way, because there were design patterns baked into the language, whereas C++ developers might solve the same problems in a whole bunch of different (and often horrific) ways, because they could go off in any direction they want.

As a result, there was a big push about factoring problems in terms of the patterns that you were looking at. It sounds like that’s really entrenched in Maven: that when you’re building software, there are certain patterns that repeatedly arise, and if they’re codified in the build system, then the same things are done the same way, even across a wide group of people who never talk to each other.

Jason: Maven is opinionated, and I designed to be this way. I don’t think it’s possible to have a tool work for such a large number of people unless you restrict some of the options and provide some guidelines.

Some people at first railed against Maven conventions and the fact that you have to do things a certain way, but if you follow those patterns, you generally can get a software project up and running in a day.

That’s evidenced by the traffic from Maven central, which has become the de facto standard central repository for Maven artifacts in the open source community. We get on the order of 250 or 300 million hits a month against it. In 2008, we had about 1.9 million unique visitors to Maven central. In 2009, we had almost four million.

So, even though Maven’s opinionated, it seems to work for a lot of people. Even though you’re restricted in some of the things you do, it provides more value by limiting what you can do than it would in providing ultimate freedom to give you as much rope as you want.

Cities that are designed well usually have structural patterns in common. Look at cities that don’t have much infrastructure or don’t copy other cities. Lots of cities in North America do that, and they have horrible infrastructures. A lot of developers and organizations tend not to look at what’s happened in the past, and they often repeat mistakes over and over again.

We have worked with very large organizations to address this issue. For example, Intuit, SAP, and Cisco work openly with us, and they’ve adopted Maven. We’ve actually had people from these organizations talk to one another about what their infrastructures look like, and really, they’re not all that different when it comes down to it.

We’ve been able to collect a lot of large companies because they want consistent patterns. They want to be able to hire resources off the street. They want to be able to move their developers from project to project without forcing them to relearn things over and over again.

That’s the tradeoff for what’s perceived at first as a limited set of freedoms. We actually get a whole tool chain that can help you produce a piece of software from end to end, and generally get a theme up and running and have continuous integration, testing, and deployment all working in a few days.

Scott: The challenge is always when there’s a paradigm shift, because you end up with people who are simply trying to use the new tool that is designed to operate under a different paradigm, and they’re trying to use it the old way.

People try to use C++ without embracing the object-oriented model, or they move from a pointer-based style to garbage kind of development. If someone is coming from Ant, Make, or some home-brewed build system into Maven, I assume it’s challenging for them to step back and take a few days and learn how to work within its flow, instead of in spite of it.

It sounds like you have found a way to help people be successful at that. The shoulder on the learning curve is low enough that people are willingly adopting it, which helps to create its own momentum.

Jason: We recommend that people approach Maven initially with a smaller project to understand the basic principles. People do run into problems when, as you say, they try to wedge Maven into an old model.

That’s why, if someone’s first attempt with Maven is a version of another project, they’re generally not successful.

Scott: There’s almost a pattern around learning a new paradigm. I remember when .NET came out, a lot of organizations tried to learn the new technology by taking their largest, most complex piece of Java software and porting it to .NET.

That’s actually the worst way to go about learning the new technology. The best way is to choose a new, low-priority project and build it from scratch. Doing that a number of times before taking on a bigger project will save a lot of heartache and make the whole experience more successful.

Jason: We’ve adopted certain strategies with the larger companies we work with. You can’t do a bank conversion on an organization that has 4,000 developers; it’s just not possible. Instead, we’ve developed a pattern over time to encompass how projects using different build systems can interoperate at the Maven repository level.

So, even if you’re not building with Maven, as long as your project can publish artifacts to a Maven repository and other projects can consume those artifacts from the Maven repository, you can make it work. There are Ant tasks for dealing with Maven, as well as Ivy, and there are other build systems like Gradle and Buildr that all talk to Maven repositories.

Usually, the companies that we go into are interested in how components are being promoted around various systems. For example, how does a development team “promote” a particular build to a release repository, and how does development interact with quality assurance and an operation team that is responsible for support an an application in production. These organizations tend to move slowly, they leave projects alone that are delivering, and they publish and consume to Maven repositories.

We try to help companies adopt that strategy. Even if they don’t adopt Maven as a build tool at first, they can still immediately benefit from repository management.

Scott: I see more of that sort of model in the business world, and I think that’s a good thing in general. Instead of organizations dictating to others, they adopt a stance more like they’re offering a service, even though there’s also an element of competition to it. After all, they’re competing with the existing build system

I think it’s interesting to see different parts of a company organized around service delivery. There may still be lots of fiefdoms of power, but there’s also an impetus toward cooperative interaction.

Jason: There’s often a consistent solution, but generally what happens is a pattern. Someone likes the idea of Maven, and they want a common reporting and repository infrastructure. They will often become a spearhead within the organization.

People see how they build their projects, and almost every continuous integration tool and every passing framework integrates with it, and you get all of this at a very low cost. You have to learn something, but essentially, every other organization in the world can contribute to Maven, and thousands of corporations have contributed.

Eventually, people within the organization see this, and they set up a small Maven build. Once it starts within an organization, it’s usually spreads like wildfire.

Scott: You also mentioned that there’s the opportunity for a lot of cross-company collaboration. There’s an opportunity for Cisco to collaborate with SAP, in terms of using Maven, because in general, companies don’t see the build system itself as a competitive differentiator, even though the thing they’re building might be.

I see that as a prime area for open source to fly. It’s kind of like in the software-as-a-service model, where the intellectual property is their competitive advantage, but the underlying web server isn’t, whether it’s Apache or IIS or whatever.

It’s the code that they’re running on top of it that differentiates the companies, and so Maven makes perfect sense. Like you said, most companies don’t view their make files or their Ant files as the differentiator.

Still, I’m sure you’ve run into some companies that are resistant to that idea. I remember working with a company that had built its own remote object protocol; they reinvented COM objects and that sort of thing.

They really felt like it made sense for them to maintain this large, proprietary system of plumbing. They couldn’t hire anybody off the street who knew it, of course, but certain people within the company were so invested in it that they were really reluctant to let it go.

Do you run into that? You mentioned that it’s common for companies to have their own home-grown build systems. Talk a little bit about the politics involved there.

Jason: We’ve gone into organizations where someone asked for our help, and we immediately get into arguments with others who are very resistant to our being there. It’s one of the last frontiers in an organization, because there are existing patterns for almost every other task. A lot of times, with J2EE applications, for example, there aren’t too many other tasks that vary. A lot of good engineers like making infrastructure, because it’s hard to do. They often get very attached to the systems they create, and I’ve been in places where people think of them as their babies. It’s really hard to dislodge them.

In some places, it’s taken a year to get through all of the politics, but we try not be antagonistic. We tell them that they can integrate into the process slowly, and we try to show them the benefits. We often try to convince them that contributing to Maven is a better avenue for working on this stuff than trying to keep their own system around. There’s generally resistance at first, but there are good, logically sound arguments for that approach. If people are going to spend their energy developing infrastructure, they’ll probably like having people using the things they make.

Ultimately, if you contribute to Maven, you’re going to have two million people using your stuff, as opposed to maybe a hundred people using an internal system. We often say that writing build tools is fun, until people other than your friends start to use it. Then it becomes a drag, because you end up having to support 1,000 people.

One problem that we run into in almost all of these organizations is that, if your build system doesn’t support the functionality required to code something in your application, you wind up having to change your build infrastructure at the same time you change your application. If both of those things change at the same time, and something goes wrong, it’s just a big finger pointing session. One of the points we often make is that by using a framework that many other people are also using, if something goes wrong in your application and tests fail while the build infrastructure remains the same, you can be pretty certain that it’s your application that’s wrong.

The cost of maintaining your own build infrastructure can be substantial. One company we work with has estimated conservatively that over five years, failed projects and lost developer productivity because of poorly executed infrastructure have cost them $400 million USD. Now that is real money, and I think companies are starting to appreciate not just the dollar cost of maintaining custom development infrastructure but the lost opportunity for collaborating with a large community using a standard process.

Scott: That does make sense when you consider that code is inventory, and similarly to the retail business, its best to minimize the amount of inventory you keep around, because you always incur costs just from having it around.

The ideal for a development organization is to maintain only the code that is core to your business. The build system just isn’t the type of code you want to maintain, because it simply amounts to overhead.

Jason: A lot of people think it looks easy, but I would draw an analogy to how, as a developer, I used to regard sales people. It looked simple enough to me, until I knew more about what their job actually entails, and it turns out that it’s a very hard, complex job.

Some developers think that, in terms of infrastructure, they can just whack something out that’s good enough to meet their needs in a day or two. As soon as they start needing other features, they see that it’s just as complicated a task as writing an application.

People often underestimate this complexity and really get themselves into trouble, especially because the patterns for developing infrastructure are not the same as for developing an application.

In a lot of large organizations, it’s not developers who run the builds. It’s done by people who have a lot of scripting experience or come from a configuration management background. They’re aggregating a lot of components that come in from different places, making sure configurations work, and validating deployments, and it’s just not the same type of thing.

This is where we’re almost coming full circle, now back to a lot of these dynamic language based build tools. They’re using Groovy and Ruby, and a build system created by a developer ends up being a 4,000 line Ruby script, which doesn’t really help anybody. You have to continuously invest in maintaining this 4,000 line Ruby script if that is the path you choose.

I would point anyone interested in creating build systems to read a little bit of history about designing cities. These things all collapse when you don’t invest in them, because infrastructure requires investment and conscious planning if you want it to work. Fully healthy, functioning infrastructures do not just emerge. They have large degrees of entropy, and you have to take care of them.

Scott: Let’s switch over for a second and talk about the Sonatype company. You spend a certain amount of time consulting with Fortune 500s who, like you said, may have 40 different build systems. There’s a lot of opportunity for building efficiency there.

It sounds like a lot of what you do is to coach those companies through adopting things like Maven. But talk a little more about Sonatype and what it does on a day-to-day basis.

Jason: Prior to 2008, we did a lot of consulting and training. When Brian Fox and I, who is the first employee of Sonatype, decided we wanted to pursue making Maven-based products, we soon realized it was very hard to actually do system training at the same time as trying to develop the products.

In 2008, we took venture capital, and we moved away from consulting at that point. We still do strategic consulting for the largest companies in the world, who basically help us develop the software for others. If the software works for Cisco, then we know it’s going to work for other people.

But we soon turned into a product company. We invest a lot of our time, energy, and money on the open source side, including the Maven project itself; into Eclipse, which is our Maven integration for the Eclipse platform; and Nexus, which is our Maven repository manager.

The strategy for the company is that we have an essentially open core, so we support all the open source versions of these tools. On top of that, we have a commercial stack of products that we’ve been selling for two years, and that’s what we actually make our money on.

We have commercial versions of Nexus, like Nexus Professional, and we have the commercial version of our Eclipse integration called Maven Studio for Eclipse. So, we are primarily in the business now of shepherding the open source projects and creating commercial extensions to them.

We have a lot of consulting partners now, and we funnel 99 percent of the consulting work we get asked for onto our partners. We’re very focused on keeping the open source projects healthy and then developing products on top of them.

Scott: In a lot of open source projects, things get built because there’s somebody who’s scratched their own itch. They developed something in open source that was novel and had real benefits over the things that came before it.

When functionality like that starts to develop a following, the people who wrote some of the initial code often wrap a company around it. That provides some stability for the project, because if the company’s profitable, you can bet that the underlying project is going to get development effort.

At the same time, there are a lot of different skills required to run and grow a business that are not necessarily natural to the software engineer. What do you say to the 20 year old out there in college right now who’s got some open source code they are thinking of growing a business around?

Jason: I would say to find someone who has a lot of experience, and I happened to do by chance. I took funding from Hummer Windblad, solely because I met a guy named Will Price, who is now the CEO of Widgetbox.

I was lucky that I found him. He gave me a tremendous amount of advice related to starting and structuring a business. He told me that at some point in my life, I may be able to run the company by myself, but that I should not make the mistake of thinking that I understood what I needed to know initially.

If you’re good at technology, you probably can learn how to run a technology company, but if you’re going to start a business, you need to find someone who understands how to grow it and how to have it function at a management level.

We are just at the point now where we’re hiring a new CEO. We hired a great board that has helped me manage the company to this point. Really, a key requirement is figuring out what you’re not good at, and most technical people are not good at running companies.

Having an under-appreciation for the other parts of the organization is the easiest way to run it into the ground. We were very lucky that our board of directors found great salespeople and marketing people for us.

I learned to defer to those people in situations where I don’t have any experience. Recognize where you have no experience, and don’t try to think you can learn it on the fly, because you can’t.

Scott: In my experience growing a business, there’s the tendency to want to guard your baby, since you as the founder have put your heart and soul into the company. I’ve also noticed some small business owners who are afraid to hire somebody smarter than them.

They feel like they need to be the expert at the technology and at understanding the customers, and they may extend that to think they need to be the expert at employee benefits, employee agreements, and running projects.

They’re a little bit afraid to hire a project manager who’s smarter than them, or a sales guy who might make more than the business owner by generating an enormous amount of sales revenue. Many people, especially at the beginning, lack the perspective that it’s not a detriment to the business if you hire a sales guy who is way better than you ever imagined. [laughs]

Jason: We’ve been lucky there. I’ve had mentors at VCs who are great. We have Mark Gorenberg from Hummer Winblad on our board, and Paul Levine, who’s from Morgenthaler and was the guy who started Atria software.

They’ve been great mentors, and they have always said that you need to find the right people for the work that needs to be done. Nothing significant is accomplished by one person; it’s always a team.

They drilled that in to me, although it didn’t take long, since I had been working with open source for a long time. Having a successful community and project takes hundreds of contributors, and it wasn’t that far of a stretch to use that analogy in a business.

Still, I was categorically lucky to have these people on the board who have lifetimes of experience. That’s the single most important piece of advice I would give anyone starting a company: don’t assume you know anything, and look for people who can help you almost immediately if you want your company to work.

Scott: If you’re an engineering person, another thing to realize is that you don’t necessarily want to do some of the jobs that are needed to run a business well. You have to ask yourself whether you want to spend your time thinking about how you could do automated deployments to Amazon’s EC2 cluster, or whether you want to review a new version of the 35-page employee handbook.

There are people who really enjoy that and are good at that, so get them to free you up to do the things on the engineering side that are really going to move the company forward and that, frankly, you may like a whole lot better.

Jason: You also have to consider that doing things outside your sphere of expertise means that it will take you far longer than someone who specializes in that area, and the result will probably not be as good, either.

Small business owners need to learn that pretty quickly, or they put their companies in jeopardy. You have to be very cognizant of cash burn when you’re starting a company, and if you pay attention, it immediately becomes apparent that there are lots of things you shouldn’t be spending your time doing.

Scott: Well, we’re coming up on the hour, so I’d like to hand the microphone to you and ask what interesting things are in the future of Maven and Sonatype?

Jason: The polyglot Maven project is an attempt to enable people to use dynamic languages with Maven. Normally, you would use an XML file to describe your Maven project, but we have some very interesting collaborations with people like Charles Nutter from the Jruby project. About a month ago, we took Maven central, which had about 100,000 data artifacts in it, and we did something in our Nexus project that dynamically turns a Maven repository into a Ruby Gems repository. So, we turned Maven central into the largest Ruby Gems repository on the face of the planet.

That was fairly interesting work, in terms of allowing Jruby developers writing applications in Ruby for the JVM to leverage every Java library that exists in Maven central. We’re trying to work on some of these polygot approaches to bridge the gap between people using Ruby, Clojure, and Scala. So people writing applications in these languages can benefit from Maven and repository managers like Nexus. We’ve created a facility where they can use their favorite language as a front end to Maven.

We’re also working a lot on holistic solutions. We plan to release a stack based on Maven, Eclipse, Nexus, and Hudson. Most of our clients ask us for end-to-end solutions. They don’t want point solutions, and if they’re paying us for support, if something goes wrong in a part of the system, they want help with the whole system.

We’re actually going to be releasing a commercial development-infrastructure stack, and that’s really exciting for us. That’s being driven almost entirely by the needs of our clients. Some of the largest companies you can think of are helping us drive the features that are being put into the stack.

Maven 3 is about a month away from being released. We’ve gone to great lengths to make it fully backward compatible, which will allow users to easily pick up new functionality if they need it. There’s a lot of stuff going on.

Scott: Awesome. Thanks for talking today.

Jason: Great. Thanks a lot, Scott.

Bookmark this:

• Meeting the challenges associated with building a database and storage engine
• Contrasting document-oriented databases with relational ones
• Trade-offs relative to working with relational databases
• Smoothing the paradigm shift to schema-less and document-oriented databases
• The role of CouchApp and the Couchio company in the world of CouchDB

Scott Swigart: To start, please introduce yourself and talk a little bit about the CouchDB project and the Couchio company.

Damien Katz: I am the founder of the CouchDB project, and I’ve been in the industry now for 15 years. I started in the software industry on LotusNotes, the back-end database model of which is largely the inspiration for CouchDB, although Lotus Notes was a much larger project than CouchDB is.

About five years ago, I was done with a startup and decided I didn’t want to take a normal job. Relational databases were everywhere at that point, and MySQL was getting insanely popular. I had a lot of fondness for the LotusNotes platform, although it was very much the ugly duckling.

I saw all of its warts, but I also saw that it had some beauty, and I wanted to take that beautiful part and bring that into the modern open source web. I quit my paid software career and moved my family from Massachusetts to North Carolina and lived off of savings and occasional contract work while I started the CouchDB project.

I started it in C++ and XML; there was no Erlang or anything like that. Eventually, we threw away the C++ codebase, and eventually we moved to JSON, JavaScript, and Erlang. We’ve gone through quite a lot of iterations to get to where we are.

We just founded Couchio, which is the company wrapped around CouchDB, last December. I moved my family out to California in January, and we’re doing the whole startup thing in Silicon Valley. We’re focusing on services and support and hosting right now.

Scott: Taking on developing a database and a storage engine is no small task. In fact, it’s one of the hardest things to do in computer science. In addition to considering the functionality, there are also performance and scalability considerations that become really important. Talk a little bit about taking on that challenge.

Damien: You’re right that there are a lot of issues to worry about with a database server, such as performance, concurrency, and reliability. When somebody writes something, does it actually get written? And when somebody read something, was it supposed to be read? Was it actually committed by someone else?

At the root of these concerns are the ACID properties: atomic, consistent, isolated, and durable. It’s very difficult to accomplish that while also achieving the performance and reliability characteristics you want. It takes a lot of careful planning.

Doing it improperly can easily produce results that seem to perform well until you hit some sort of threshold where you run out of resources on the computer and everything sort of falls off. Or the computer doesn’t shut down properly and you might corrupt all your data.

There are a whole lot of challenges there that you have to really think about, and there are a surprisingly large number of ways that you can address those issues. CouchDB takes a fairly novel approach compared to most database engines.

Actually, not all database engines do actually address those issues. Some just decide to kind of punt by saying that you need to run on a very reliable server or on multiple machines for failover.

There are all sorts of different ways to directly solve these issues, as long as you really think about it carefully early on. Early in the CouchDB development cycle, it seems like I spent weeks or months just thinking about the design from a very high-level point of view.

I spent huge amounts of time planning how the software would interact with the clients and the disk, how multiple clients would interact with all that, and coming up with a simple design that would be reliable and provide good performance.

Scott: Another point of interest is that CouchDB is a document-oriented database. It’s far more common to be familiar with relational databases, so talk a little bit about this kind of database and the tradeoffs between it and your traditional relational database.

Damien: The traditional relational database, based on tables and rigid schemas, is an old design that’s very well studied and optimized. When you can break your design out into schemas and carefully size each column and table to match your data, you can make amazingly fast solutions.

But we often don’t have that sort of insight into our data model ahead of time when we’re developing an application. And frequently our data model just won’t fit into that kind of schema. The document model is an alternative, where the data is self contained, and documents fit into that.

To understand the model, it helps to think of business cards. Most business cards contain the same information: a person’s name, phone number, email address, possibly their actual physical mailing address, and other bits of information such as a professional title or the company name.

These cards are all documents, and they have close similarities as well as differences. This is an example of something that doesn’t necessarily fit very well into a relational database. Of course, business cards are relatively simple, and you might be able to figure out all the possible iterations and put them into a relational database.

But there are other types of documents where you don’t have that luxury ahead of time, and it can be especially difficult with something like content. That sort of data is ideal for a document database, where you’re going to have fields and attributes that vary broadly from document to document.

It doesn’t mean the documents themselves are completely different from one another, and usually they actually have a great deal in common. Usually, all the documents have a common set of attributes, but they also have unique attributes that vary document to document. Those are the types of things that relational databases have a difficult time with, but that document databases are ideal for.

Scott: From a programming standpoint, how do developers deal with that? To consider your business card example, it’s obviously simple to code against the stuff you’re expecting, but like you said, the whole point of a document-centered database is that it can deal with the more freeform stuff as well.

How do you see application developers dealing with unknown, free-form sets of fields?

Damien: Frequently, in a document oriented app, you just want to capture the information at first, and you don’t necessarily even know how you want to report on it. The simplest approach is just to add the fields that you want to capture into the documents, and then later on figure out what to do with it.

For example, you might have captured a large set of foreign tax IDs, and then six months later, you decide how to create a report on all those content records. Therefore, it’s important to make it easy to get the data into the database, so you don’t have to adjust the schema, you don’t have to do an alter table or anything. You just stick the data in there and it’s captured.

Later on, you can create a new view without having to alter any of the data in the database, without having to alter any of the existing logic to create something that will show you all the documents that have these foreign tax IDs.

That’s an example of something that can be kind of difficult to do in a relational database. With a document database, the application doesn’t actually have to modify a schema somewhere in order to do this; it just basically sticks the attributes right in the document.

Scott: Obviously, relational databases can do joins, and there are other things that they’re good at, but there are trade-offs. You have to consider what your data looks like to decide which model fits best with your application.

It also seems that document-oriented databases have strengths when it comes to distributed processing in the cloud. Parallelism used to mean multiple threads inside one computer, but more and more, it refers to nodes.

Tens, hundreds, or thousands of nodes each crunch on a piece of data, and then the results are put back together on the back end. Talk about a little bit about how you see development changing as it moves to the cloud and where CouchDB fits in.

Damien: With relational databases, one of the difficulties is that, with a fixed schema, it’s difficult to have multiple instances of the same database in different places, particularly with a master/master type of setup.

A master/slave type of setup is the standard for relational databases across multiple machines. All the changes go into a single master, and you have multiple slaves to alleviate the read load. That works fine with relational databases, and you can also do that in CouchDB and other document databases.

On the other hand, in document databases, you can also have a master/master model, with geographically distributed databases that can be independently updated and can replicate their updates with each other.

With relational databases, you can do master/master setups with very careful planning of your data model ahead of time. You have to do a subset of the relation model to make sure that you don’t get into an invariant condition, and then you can do master/master.

Using relational databases, though, you can’t change the schema on one side in a master/master setup without a tremendous amount of planning and some downtime on each side. When you change the schema on one node, you create a conflict of schemas and data. You have to figure out how to translate the differences.

With a document-oriented database, each document contains its own schema, so those conflicts don’t arise. Instead, if two people edit the same document, that conflict is detected automatically by the system and can be resolved by the user on an automated basis.

Relational technologies are designed to have a single master at any given time that is the authoritative master state of the data. Without a whole lot of planning, you have to build something on top of the relational model in order to get around that.

In the document model, it’s simple to open it up to multiple machines and move the data around. The data is independent rather than interrelated.

Document databases allow data to move between machines, within a single cluster or across clusters much more easily. Of course, that makes cloud computing much easier than with relational databases. It’s easier to partition new machines, bring new machines online, or move them across data centers.

Scott: Schema-less databases and document-oriented databases are a paradigm shift, and it provides bigger programmer challenges than something like picking up a new language. I would liken it to the transition from procedural to object oriented programming.

When people try to pick up a functional programming language for the first time, they have to grasp an entirely new way of approaching the problem. I would imagine that there is a tendency by some people to try to force it to work like a relational database.

What are some of the key concepts that people need to wrap their heads around in order to use CouchDB the way it’s meant to be used, rather than shoehorning it into working the old way?

Damien: The first thing to consider is the tendency to normalize your data excessively. While we aren’t a relational database, it is possible instead of putting a customer’s name into an order record, to use a customer ID, and that might actually be a better way in CouchDB.

Still, you could go too far and make everything relational. For example, when a customer buys something from a store, the receipt doesn’t have the store’s tax ID number on it, right? They give you the actual name of the store and the address and the phone number.

Those things could change later on. The store could change names, it could change ownership, it could move, or it could change phone numbers, and any piece of information on that receipt could potentially become out of date.

That receipt still has value to us, as a record of a transaction at a point in time. The relational paradigm says don’t do that, we need to normalize everything. The document paradigm says a receipt is perfectly fine, and we can put the name and phone number right on the receipt.

CouchDB encourages you to think more in terms of the receipt, but it can also allow you to be more relational. You don’t have to be entirely non-relational and de-normalized. The challenge for developers to work their head around is where to find that sweet spot, so they can decide optimally where to normalize and denormalize in CouchDB.

That’s actually the challenge in a relational database, too. It is very easy to over-normalize a relational database. As a result, it can become difficult to work with, all the queries can become unwieldy because you’re doing way too many joins, and everything can become more obfuscated than it should be.

There is a similar issue with CouchDB, in the sense that you can go over-normalized or under-normalized, although in CouchDB, you generally want to lean more toward a de-normalized form.

Scott: I think that data warehousing foreshadowed this transition to some extent, in the sense that, in certain scenarios you have to embrace denormalization. On one hand, you have to be concerned with breaking things down granularly and having tables and sub-tables, and joining all kinds of stuff together.

But to actually get performance, the data warehouse has to have a pre-denormalized view, to let you do that sort of stuff.

Document centric database in some ways maps to object oriented bit better, right? A document is sort of a hierarchical object; if you think of an HTML page, it’s got a title, it’s got a body, and it’s got a bunch of elements in it that are defined on one hand but on the other hand they can kind of be in any order and so forth.

Talk a little bit how a document-centric database is a more natural fit for an object-oriented programmer than this sort of data-mapping layer where we’ve always had to take our objects, shatter them into tables, push them out into a relational database, and then kind of reform the hierarchy as we pull stuff out and get it back into objects.

Damien: I’ve always felt that, at its heart, CouchDB is really an object database. The reason I don’t call it an object database is that it’s a very overloaded term in the industry, and people have a set of expectations that go along with object databases.

Nowadays, there really aren’t that many popular object databases, but back in the day, Gemstone had some popularity. Maybe it’s still popular these days, but I haven’t heard much about it. This was the idea of a transparent object-persistence layer for a programming language. Gemstone was for Smalltalk, there was Zope for Python, and I know there were others for other languages.

These are essentially databases to store programming objects natively right into a database without any sort of real translation layer. CouchDB, on the other hand, is not a seamless, persistent layer where you get your objects and then you have different methods and everything also gets stored in there.

That’s the biggest difference, but there are still objects. They just don’t have the methods, and they’re not in an object-oriented programming database.

I think one of the big reasons object databases didn’t catch on is that they’re married too closely to the languages they are designed to work with. That’s convenient when you’re working with the language, but the data frequently outlives the application or the project, and you also need to share the data with other applications or other projects.

With CouchDB and JSON, you can use it from any language, but if you’re using something like Gemstone from Smalltalk to instantiate one of those objects, you’re going to have to call it from Smalltalk. If you’re going to access it from something else, you are going to have to write an access wrapper.

That becomes a big obstacle to re-using that data from another application. I think that’s one of the big reasons that object databases never really caught on and never had a chance against SQL which is its own simple language and so it didn’t have that problem.

It also plays much more easily with a variety of programming languages than any single-object database does. CouchDB’s popularity owes in part to the fact that it can play very easily with a wide range of programming languages. And it opens up your database to a lot more applications and being able to share the data, which is very important.

Scott: As cloud development matures and evolves, what do you see as the challenges, opportunities and future direction for CouchDB?

Damien: The large data stuff hasn’t really been our primary focus at CouchDB. That’s not to say that we haven’t focused there at all, and we have always designed the CouchDB data model so it can scale up.

But our focus has been primarily on making sure that we can build applications very easily, very rapidly, so they can replicate. Building really large scale applications has always been a secondary goal for CouchDB.

We make it very easy to launch web apps, scale them up or scale them down, and not worry too much about their reliability. You can just bang them out, get them working, and deploy them. That’s primarily our interest.

Scott: Talk a little bit about Couchio and what the company does.

Damien: Right now we’re providing services and support. Our first customer that we signed up for support is Canonical, and we’re helping them with their Ubuntu One service. We are just about to launch our hosting service. We are in private beta, and we hope to initiate public beta soon.

Our focus with hosting is going to be on low-friction hosting, to get people started with as little work as possible. We aren’t focused on large-scale hosting, for people with huge data sets and things like that.

We eventually want to get into application hosting. There’s a thing called CouchApps. CouchDB, being an HTTP server, can host applications directly, so you can write applications and forward the HTML, CSS, and JavaScript through CouchDB.

When you point your browser at it, the browser comes alive and starts the JavaScript. It becomes interactive as you query and update the server and everything. When everything is served from CouchDB, that’s a CouchApp and you can replicate it around, just like the data.

We’re building these CouchApps for things like mail, timelines, scheduling, and document management. We want to provide hosting and support for these CouchApps, so users don’t necessarily even know that these CouchApps are built on top of CouchDB, they just know that their applications are being made to run their business.

And the benefit of these CouchApps is that they can run them in the cloud, they can run them in the browser, and they can run them on a small server in their office. The whole time, we have them backed up and they’re available in the cloud via replication.

Their data’s always with them, it’s installed locally on their laptops or on their phones, and it’s always available using CouchDB for replication. So if their network connection goes down, or it’s really slow, or if they’re traveling on an airplane, they always have access to their data.

Moreover, they’re always accessing it through the same browser interface, and regardless of the network conditions, it’s always a fast, low-wait connection, because it’s local.

Scott: This notion of a CouchApp, where the app is stored in the database, is fascinating, and it makes sense because the database can store anything. And it’s automatically replicated, so if you needed to stand up more nodes or whatever behind your load balancer or those kinds of things, it’s a piece of cake.

Damien: Databases have this concept of design documents, which are like regular documents, except they contain information about how to index the database. The MapReduce JavaScript indexes are specified in these design documents, and they can also contain the HTML, the CSS, the images, and whatever they need for the applications.

All of those can be stored in a single design document, and these replicate around with the data. You can also use apps to put them up on the server cloud, so you point your browser at it and it’s just like a regular web browser app. We have several examples of these.

Mozilla has written one called Raindrop that’s basically a mail-style app. We’ve written a chat app and a calendar app.

You run these in your browser, and they’re just like regular web apps. The really cool thing is that once you install CouchDB locally, you can replicate these down and still run them even if you’re disconnected from the network. You have the exact same experience.

We want to get the browser plugins, so you can integrate them straight in the browser. In the CouchApps, there would be the option to use it offline, which would cause it to replicate locally.

The browser would still deliver the same user experience, like a local proxy. They would still see in the URL that they’re hitting the remote, but it would intercept all those requests and hit the local replica.

Any changes they make to the local replica immediately get replicated up to the cloud, and any changes somebody makes to the cloud version are replicated down to the local one, so you always have the most up to date version locally.

It could also be an appliance that’s rented to somebody. There’s this thing called the Shiva plug, which is a really small Linux plug that’s about the size of an Airport Express. It runs Linux, and it has a hard drive and a little web server. We’re going to put CouchDB on those types of devices.

Then people can rent these plugs that have the server and their application on there. Their cloud provider will mail it to them, they plug it into the wall, and it automatically connects to the cloud CouchDB instance.

They would get a really high bandwidth, low latency experience for these apps, because they would be served straight from the plug, which is also constantly replicating with the cloud. This solves problems for small businesses that are running off of a cable modem, and their telco has oversold.

A lot of people run off of DSL or cable modem, and frequently throughout the day, it gets slow or stops working altogether. The customers don’t have access to the network for whatever reason, so they’re completely in a holding pattern until the network comes back up. The plug makes it so the data is always right there locally.

Scott: That’s a great explanation, because it makes it clear to people how this technology can actually help them in their business.

Damien: The idea behind that little device is that now you have a little tiny server that’s always tethered to the cloud. You can serve applications to everybody in the office, and nobody has to install anything.

You’ve essentially replaced the whole Microsoft Office package. Whereas previously, you’d have to buy Microsoft Office and install Office Server and Exchange and all sorts of different things in order to deal with the fact that you can’t have everything in the cloud because your network connection is just too unreliable.

This bridges that world, so now you’ve got this low cost tiny plug and everything’s higher speed because it’s local, and then you’ve also got everything in the cloud, so that everything’s backed up, professional administered, and up to date. And if the plug dies, everybody in the office just accesses the cloud version while you wait for Fed Ex to deliver you a brand new plug.

Scott: I want to be sensitive to the time. Are there any closing thoughts from your side?

Damien: We’ve just gone to version 0.11, which is our release candidate for 1.0. We’ve already been very stable as a database engine, and right now what we’re actually trying to stabilize on is our program interface, and we’re about ready to go.

Scott: Thanks for taking the time to talk today.

Damien: Thank you.

Bookmark this:

• Determining what content to feature at OSCON
• Transitions in the open source community as a whole
• Business-strategy considerations associated with open source
• Selecting participants to present at OSCON
• Interrelationships between NoSQL, cloud, and mobile shape the future
• The role of OSCON in the larger sphere of technology

Scott Swigart: To start us off, please take a second to introduce yourself and your involvement with OSCON.

Edd Dumbill: Sure. I have now been the program co-chair of OSCON for three years, and my colleague Allison Randal is the other program co-chair. I took over from the legendary Nat Torkington.

I wear two hats at O’Reilly. In addition to being program co-chair for OSCON, I’m also responsible for the software that runs the conferences, so I kind of get to hack on both sides of the fence.

Before I started working with O’Reilly, I had done a couple of startups and been an open source hacker for a while. I contributed to the GNOME project and was a Debian developer, so I’ve been in and around the open source world for quite a long time.

Scott: How many years has OSCON been going now?

Edd: This is the twelfth year, I believe.

Scott: Obviously, you want to keep it fresh every year, so how do you track what the trends and interesting movements are in open source?

Edd: That’s a great question, and it’s actually the part of the job I enjoy most. We go about it in two ways. The first is we’re lucky to have a great response to our call for participation. This year we had well over 700 proposals.

Considering that we could only accept around 120 of those, we unfortunately had to turn away a lot of really great stuff, but the spectrum of responses paints a pretty good picture of what’s trending and what’s not.

The other thing is that O’Reilly tracks projects and the industry as a whole day in, day out. An example of that effect is the health IT track, which we’re doing at OSCON this year. We identified health technology as an area that’s ripe for the kind of collaboration that comes with open source.

We’re betting on that, and we opened up a separate call for participation for health IT, which we’ve had a great response to. We hope not only to cover trends there, but also to be part of getting new conversations and collaborations going.

In a similar vein, we’ve also set up two summit days in parallel with our tutorials. The first summit is an open source and cloud summit, where we’re gathering a lot of leaders to discuss the place of open source in the cloud, as well as what the future holds for emerging standards and collaboration.

At the hacker level, we’re doing a summit on the Scala language, which is an up and coming JVM language that has both enterprise and hacker credibility. It’s been used in places like Twitter and FourSquare, places where there are a lot of scalability concerns. We’re organizing a half-day Scala tutorial on Monday, and then a full day of Scala-related talks on Tuesday.

To come back to your question, we observe and try to shape some trends. NoSQL databases will be featured strongly this year. There are also things we identify as up and coming, and we try to give them space to grow and provide a forum for our community to get to know each other.

Scott: Five or ten years ago, it felt like open source was trending toward a separate, parallel universe. We would have an open source operating system with an open source web server on it. There would be open source office productivity tools, open source development environments, open source compilers, and it was all very distinct from the proprietary side of the industry.

Looking at OSCON now, there’s stuff about Android and iPhone, and even though Android is open source, it’s not really community developed. There’s also a lot of material at OSCON about cloud infrastructure, and that area involves a lot of proprietary players, whether it’s VMware or Amazon. Amazon built an infrastructure using open source, but they’re not exactly open.

It seems that a lot of proprietary stuff shows up in direct conjunction with open source now. There’s even beginning to be a thriving open source ecosystem around those proprietary offerings, whether it’s iPhone or whether it’s a certain cloud provider or something like that.

You mentioned that your involvement with open source stretches back a ways, into some Debian development and that kind of stuff. How have you seen the perception of open source and proprietary software evolving over that time?

Edd: I think there’s always been a spectrum, and that spectrum still exists today. At one end, you have the free software movement, which wants a complete stack of completely free software. At the other end is corporate open source, where the control is still very much with the organization, but they’re using open source as a tool. And there’s a spectrum that lies in between.

O’Reilly has always been pragmatic, recognizing that while the free software movement is a very important thing, you don’t have to subscribe to it to get value from open source. The key to open source and sharing source, and also really open data and open standards, is that there’s a lot of value to be had without subscribing to either extreme.

Scott: How does that play out with transitions like the current one toward the cloud, which is really causing some new types of convergence between the open source and proprietary worlds?

Edd: One of the really important questions facing open source right now is with software as a service, what is the place of open source? Previously, if my text editor was broken, I could just develop it and hack it, add a new feature and submit the patch. What happens when I can’t do that?

You’re seeing a slow uptake in the popularity of the Affero GPL license. Not only does it require you to publish the source of you make any changes to it, but, but if you were running this as a service, then you must publish your source.

A poster child project for this is the StatusNet project that Identi.ca runs on top of, but we also saw an announcement this week that Sugar CRM v6 will be AGPL. There is a free software reaction to the web services, and it will be interesting to see how that develops.

We also have Stormy Peters giving a keynote on that topic, and she’s also hosting a Birds of a Feather session and a bunch of lightning talks on it. I expect OSCON will be a forum this year where we can talk about our reaction to cloud and open source.

With regard to what you’re saying about how open source has transitioned away from free software idealism and so on, corporations are catching on more and more to the fact that open source is a very useful business tool. Tim O’Reilly made the point that you can see open source as a tool used by the second player in the market.

Taking phones as an example, the iPhone has a lot of dominance with a completely closed platform. The key for Android is that it’s open source, so they’re getting a competitive advantage by opening up. We are seeing that particularly in the mobile segment, but in other places too.

Open source is becoming a business tactic, and OSCON will have a lot of content to help guide businesses.

Scott: We’ve interviewed people from a lot of companies that are wrapped around open source projects. We have found it to be quite common that somebody had an idea, built an open source project, and did the care and feeding of it for a while.

Then, when they got a lot of people using it, they launched a company around it to provide professional services, customization, support, and so on. It may be a bit of a stretch to say that most popular open source projects have some kind of company kind of wrapped around them, contributing code, but sometimes it seems that way.

What are your thoughts on that? Can you give us a little bit of preview about the bullet in the conference description, “use open source effectively as part of your business strategy?”

Edd: In the business track, we’re targeting two kinds of audiences. The first is corporations that are thinking about open sourcing their code bases. OSCON is a great place for people to get oriented into the open source platform.

The other audience is businesses who are using open source. You can leave yourself open to a number of problems if you’re not actively tracking your use of open source and the associated licensing. Those issues can be especially complex in acquisition scenarios.

In general, open source developers need to pay more attention to matters of law. We have a session from Bradley Kuhn and Karen Sandler, which is proving very popular already, called “The Bare Essentials of Legal Issues for Developers.”

The goal of that session is to teach what developers need to know about trademarks, copyright, and patents. People can’t just release open source code and then forget about it. The future of their project is vulnerable on all three of those fronts, and people must not be ignorant about it.

Scott: At this point, it seems to be inevitable for a software company to run into open source, whereas ten years ago, everything they were using might have been proprietary.

To look at the really big examples, IBM obviously interacts with open source and contributes to it a lot. Oracle just acquired Sun, so there’s a ton of open source in their portfolio. Even Microsoft now contributes to the Linux kernel and has a whole bunch of little projects they’ve released as open source.

If your company gets really huge, it will almost certainly interact with projects and contribute code, but more along the lines of what you’re alluding to, your developers are just going to want to use it.

The difference between an Apache license and a GPL license can have huge ramifications if you’re bundling the code with something you’re distributing.

Edd: Absolutely, and companies need to be aware of that. There’s another aspect that we’re also covering. I think business and community in the open source world are very much interlinked. You can’t really use open source in isolation from its communities.

We have a whole community track that discusses things like how to manage a community full of contributors, how to be an effective leader in those places, and how to foster diversity.

We have a fascinating panel that Leslie Hawthorn’s put together about financial incentives in open source. Leslie ran the Google Summer of Code Project, and she just recently left Google. I’m really looking forward to that, just to hear the panel talk about what works and what doesn’t work and how you can effectively use money to accelerate and encourage open source projects.

Scott: It’s a fascinating topic, and a lot of open source projects have failed because a company thought it was going to be easy. They just threw the code out there, but the community didn’t come, or maybe it fractured. There are a lot of very smart people working with open source, and a certain percentage are very sure that what they believe is absolutely right.

Community managers need to address the issues that come along with strong personalities, and they need to incent people to do work that nobody really feels like doing, even though it’s critical. Getting community contributions requires a different mindset than when you have a hundred developers on staff.

There’s also a whole domain of expertise associated with deciding strategic issues like whether to follow the aquarium model or the community model.

Edd: That’s true, and the community as a whole is only now finding out a lot of those things. There’s only a handful of projects that you can look to for a ten-year span of experience, and some of them have gone through pretty radical overhauls.

There’s another presentation from Dave Beckett, who works for Yahoo! now. He’s been maintaining infrastructural projects like the PNG since 1994. His presentation is called “Mature Open Source: Making Your Software Last a Decade.”

What’s fascinating about him is that he’s just one guy. As opposed to the corporate side with incentives and whatnot, he started off more or less as a solo hacker and kept his projects going. As time has gone on, he has slowly expanded the pool of participants and so on.

I’m glad that we’re hearing the story from both ends. The corporate side is talking about taking their projects open or funding projects, and the lone hacker side is talking about how to stick with something and slowly build participation without going crazy.

Scott: OSCON is in a fairly unique position to really understand open source developers, because you’ve been running this conference for 12 years now.

Open source has evolved with corporations getting more involved, but there’s still that core constituency of open source developers. Are you guys ever surprised by what resonates or doesn’t resonate with those developers?

Edd: When you put together a conference, you’ve got to have several things in mind. First and foremost, people are paying to come there, so it’s got to be useful and relevant. Over and above that, it has to educate and challenge as well.

As program chairs, we play our part both in recognizing people in the community who’ve got something different and challenging to say and inviting people who we think will stretch people’s minds and thinking in our keynotes.

Scott: So you’re not shying away from controversy, in other words. If you think somebody has a really intriguing point of view, you’re OK with it being somewhat controversial?

Edd: I think so. For example, we have been sponsored by Microsoft, and they used our conference to announce their open source initiatives. When they first did that, it was the most frosty, hostile reception that they probably feared.

[laughter]

Still, although there was controversy, the OSCON crowd is a civilized and human one. In fact, I’ve never known a warmer bunch of people to work with, and to be among for a week. Nevertheless, because we’re geeks, we debate with a certain degree of ferocity.

To their credit, Microsoft stuck with it, and by year’s end, they started to deliver on a variety of fronts.

Sean Campbell: You obviously have a defined process that involves committees and so on to determine which people are asked to present at OSCON, and I imagine that there are certain best practices to get invited as part of that process.

Without naming names, obviously, are there two or three things that you see a lot of people do that either help or hurt their efforts to appear as speakers or panelists at the conference?

Edd: First, we certainly like to hear from the principals themselves. It’s generally a disadvantage if a PR company submits a proposal on your behalf, for instance. When we deal with the actual speaker, that person has much more of a personal investment, and we have a clearer understanding of what’s going on.

We do sometimes accept things that are submitted by PR, and that’s the way certain companies prefer to operate. The reason that it typically puts them at a disadvantage is that when a geek writes for other geeks, it tends to speak more directly to the audience. If it’s written by somebody else, it doesn’t quite grab you in the same way.

The second thing is that we like to see a fair amount of detail. We have 700-plus proposals, and part of the way we can be persuaded that you know what you’re talking about is that you’re able to write in a coherent manner and you give detail about the subject matter.

Not every superstar hacker is a superstar presenter, so the ability to present your thoughts clearly, even in just a description and abstract, is a pretty key indicator for us. We also look for the subject matter to be fresh and concrete, rather than being based on buzzwords.

For established projects, we tend to get a lot of proposals that describe “the state of X” or “what’s new with X in 2010.” That’s all well and good, but frankly, you can read that information off a blog, and we tend to de-emphasize those.

In the past, we’ve put them in lightning talks, where maybe 16 projects got five minutes each to say what was up and coming, but we’re not even doing that this year. These days everyone can read a blog or a changelog or a tweet or whatever. We’re looking for a lot more practical experience, or somebody to put forward an interesting new spin or viewpoint.

Sean: Someone who’s used to giving monologues or who is just too sensitive may not be a good panelist. For a panel, you may want someone who is a bit of a contrarian to make things interesting, without being so far in that direction that the conversation seems fractured.

How do you balance looking for someone to participate on a panel versus an individual speaking slot?

Edd: I think you said it pretty well, and to be honest, we don’t actually accept a huge number of panels. It’s very difficult to do a good panel, and I think some of it comes down to the personal preferences of the chairs.

Neither Allison nor I like to do too many panels, largely because if they don’t go well, they don’t tend to be very useful means of communicating to people.

Sean: How do people get involved in the actual planning side of the conference? Particularly because O’Reilly is a for-profit corporation, what is the experience like for someone who is not an O’Reilly employee and wants to participate? What channels do you direct that kind of feedback to?

Edd: We have a variety of channels, and we do rely very heavily on community contributions, with the first point of contributing obviously being through the call for proposals. A lot of longstanding friends of the conference contribute in that way.

We try to listen to the largest possible swath of people in and around the community, including our company. We do have a formal program committee made up of 20 to 30 folks active in the various orbits that OSCON covers.

We’re always looking for people to invite onto that committee who are active in their particular part of the open source world or interested in how to make OSCON better.

It’s a broad conference, so Allison and I as chairs need to rely heavily on the expertise of people in various areas. We have a Perl sub-committee, and people in Python, PHP, Postgres, and so on. We do some degree of self organizing, and they have little committees of reviewers who review all the talks.

We try to make sure that every single proposal is seen by at least two people, and very often a lot more than that. Everything is read and considered and voted on.

Scott: To call out a technical point, I see that you’ve included NoSQL as a topic in cloud. How intersected do you feel like those two are? In other words, would we have a big interest in NoSQL if it wasn’t for the cloud?

Edd: That’s an interesting point. We have three broad topic areas that we see as the growth areas this year: NoSQL, cloud, and mobile. All of them are interrelated, and to understand that interrelation, you need to dial back a few years, when we were talking about the LAMP stack.

Then we were talking about web applications, and the web sews all these topics together. The question of whether we would be as interested in NoSQL if we didn’t have the cloud is interesting from the standpoint that they both stem from the massively distributed and popular information network that we all use, called the web.

Perhaps the aspect that brought NoSQL to the biggest audience is the need to scale web applications and to be able to build inverted indices from regular SQL databases for performance reasons.

The vast majority of NoSQL applications sit side-by-side with an SQL database. There’s a collaboration and a cooperation going on there, and it’s a hybrid architecture.

Another interesting spin to NoSQL concerns graph databases. They have been around for a good while, but the rise of social networking on the web has finally given us a graph a lot of people care about. Again, you’ve got web applications driving that side of it, too.

Mobile is now quite interlinked with everything else. It is no longer an option, whether to add a mobile interface to your application. It’s kind of expected, rather than a value-add, if you’re launching any kind of commercial service.

Looking at the overall package as a bunch of thin clients and a bunch of chunky servers at the back end, it’s clear that mobile and cloud go hand in hand. If the client can do less, then the server has to do more.

All these issues are growing together, and the other very popular thin client, of course, is the web browser. One must not underestimate the importance of the HTML5 feature set. You’ve basically got feature-equivalent thin clients in the mobile world and the actual desktop.

One of the bets we’re making this year is that if you’re looking to target everywhere, including Android, iPhone, and the rest, then web technologies are the sane choice, as well as the most open route.

A lot of people are rightly worried about developing solely for an Apple platform, when they can change the terms and conditions, and blow what you’re doing out of the water.

Scott: In another conversation earlier today, someone observed that people get too hung up about building iPhone applications with their proprietary SDK, Objective C and that kind of stuff. Most of those applications could be built as web applications, and they can even run locally on the phone, with local storage.

There seems to be an awareness problem, where people aren’t really considering that option. Do you think that web-based approach is going to catch on?

Edd: I think this is the year when people are going to figure it out, and we’re actually working on that internally at O’Reilly at the moment. We develop a bunch of apps, and a lot of our books have been successful in that format on the App Store.

We think we need to provide more mobile interfaces to the things we do, and personally, I think the conference needs to be one of those.

Developers can either put up with a lot of risk and delay from the App Store, or they can do a perfectly great app using just the mobile browser. The poster child for this is Google’s mail, Gmail, which delivers a great experience on the iPhone.

Scott: PhoneGap makes a web app to run local on Android, iPhone, and all that kind of stuff. It looks just like an app, but the whole thing’s built using HTML5.

Edd: We have folks presenting at OSCON on that, and I mentioned several of the cross-platform talks about that sort of thing.

Scott: You mentioned that this is going to be the year when this model takes off. Since O’Reilly has the ability to put a spotlight on things, talk a little about the notion that, “With great power comes great responsibility.”

You have observed that using proprietary SDKs and so on can require developers to start over from scratch in order to go from iPhone to Android, with very little ability for code reuse. How concerned do you think you need to be with regard to companies’ terms of service and so on, when you advocate for a certain development approach?

Edd: Honestly, a lot of those things come down to personal values. The broad picture is that we’re obviously very much in favor of openness, transparency, and so on, but we’re not breaking with vendors consistently.

We will stand up and make a big deal of it when it matters, like Tim O’Reilly’s famous spat with Jeff Bezos’ one-click patent, for instance. We also stood up and spoke out directly in favor of Microsoft when they were being hit with a silly patent dispute concerning their web browser.

I don’t know that the conflict between various mobile platforms is necessarily any different from Windows versus Linux versus Mac, in the sense that all these different platforms have different terms of engagement, and open source plays a role in each.

As far as mobile’s concerned, we’re here to help equip developers, so we have a great range of books on iPhone, Android, web standards in HTML5, and so on. Our role is really one of educating, rather than taking a stand on how people should develop.

Scott: I want to be sensitive to the time. Any final thoughts about OSCON that you’d like people to be aware of?

Edd: I’d encourage everyone who reads this to look through the OSCON program. There’s a wealth of topics that we’ve not been able to touch on here. For instance, we have a fantastic hardware track, covering Arduino, things like the Beagle Board, and plug computing apps.

It’s become very easy to prototype, and there are a lot of fun things we can put out in the world to process data, connect to the cloud, and monitor and augment reality. That’s a fascinating area for us.

We have an operations track,, which is very much following the rise of developer operations as a hybrid discipline, and something else the cloud brings with it.

You can’t really run software for the cloud without taking its architecture into account. When you have ten thousand instances of a machine, you need to allow for that, so the Dev Ops discipline is very much growing.

There’s so much there, I could go on forever.

Scott: It’s obviously a great conference. These are technical people, but at the same time, OSCON really tracks trends, conversations, and debates. Having Microsoft come in was a controversial thing to do, but you guys felt that it was important. It seems that you really live by a philosophy of not shying away from controversial topics.

Edd: That’s right. Community means a great deal to us, and we regard it as an enormous privilege to have this conference. The attendees and speakers each year are what makes it a success, and it’s absurdly gratifying to stand on stage and welcome people.

It’s a fantastic collection of people who are inspiring, entertaining, and controversial. I really couldn’t think of a better job.

Scott: Thanks for taking the time to chat.

Edd: Thank you.

Bookmark this:

Video streaming has grown way beyond YouTube and Hulu. Today, broadcasters such as ABC, NBC, and the Comedy Channel are streaming full shows straight to viewers. Netflix is living up to it’s name, and Amazon offers a huge library of movies on-demand. And open-source has moved well beyond the infrastructure layer, with open source video platforms like Kaltura.  In this interview, we chat with Shay David, co-founder of Kaltura.

The rise of video as a key form of web content

The open source projects and business model behind Kaltura

The role of the cloud in the future of video editing

Key vectors for the spread of video content on the web

Solving the format-selection and openness problems of web video

What the future holds for IP-based video

Scott Swigart: To get us started, could you introduce yourself and Kaltura?

Shay David: I am one of the co-founders of Kaltura, and I run business and community development. Kaltura is the first open source video platform, meaning that we have a set of tools, technologies, best practices, and services that allow people to do anything and everything with video, including adding video to any site.

We empower the entire video stack with a set of open source building blocks, including ingestion, metadata normalization, player development, playlist development, content moderation, scheduling (for both live and on-demand video feeds), monetization, and analytics. Those building blocks are the foundations of video applications.

People are using Kaltura for video publishing, goods and asset management, distance education, and enterprise collaboration, among many other uses. Pretty much across every industry, we feel that we are becoming part of every web experience.

Scott: Why do you think video is becoming so important right now, whereas it wasn’t as widespread in the past?

Shay: One of the reasons is that the means of production are so available. If you look five years back, it took a lot more to be a media company or a studio. People needed to have a satellite van, $60,000 cameras, and whatnot.

Today, two billion people carry the means of production in their pockets with smart phones, and more and more of the population are equipped with broadband. The capacity to produce and to consume video is very prevalent.

This creates a whole paradigmatic shift in the set of technologies that are necessary, because, unlike text, video is complicated. A lot of things that we know how to do with text pretty well are not well understood for video; we need the means to handle, store, and stream very large files, transcode one format to another, and manage advertising and analytics.

Scott: Do you feel that there’s more opportunity for analytics in video? For example, if somebody goes to web page, you can determine how long they spent on the web page, but you don’t necessarily know what was interesting to them.

With video, on the other hand, I imagine you might be able to know what portions they watched, what they skipped, and so on. How are the opportunities around analytics different for video than other types of content?

Shay: Analytics actually encompass two very different types of information. The first is audience measurement, such as who is looking at the content, which countries and demographics they come from, and so on. That type is well understood.

The second set of data is media analytics, which consists of the types of things you were mentioning, like drop-off points, popular pieces of content, level of engagement, etc., which is a much less understood science. Part of what we bring to the table is a whole set of technologies around that.

Using this information begins with identifying the data you actually want to collect and the events on the player side that are associated with such data. Next, you have to decide how to close the loop back into your production systems and determine how you will use the information. Just measuring it without acting on it is only half the task.

Scott: The first time I got a sense for video analytics was the whole Janet Jackson Super Bowl thing. Since with Tivo, you can just hit a button and go back eight seconds, the Tivo company reported that the eight-second rewind function was used at that point more than at any other time in the history of their service.

I thought, “Oh my goodness, Tivo knows every time I back up eight seconds. I had no idea they collected that kind of telemetry information.” That opened my eyes to some of the things that you can track with video.

Shay: Absolutely, and that’s just the beginning. When we talk about social media and interactive media around video, we can go well beyond basic audience measurement, but we have to be sensitive to privacy concerns too.

The Nielsen paradigm is about people-metering, and making some gross assumptions about who’s watching each channel in very large categories. And that was fine when we were talking about a broadcast paradigm, but who are the people watching? How many of them are there? How many couch potatoes are in front of the screen?

When you’re dealing with interactivity, there’s a whole new set of measurements to be concerned with, because that’s what the advertisers or publishers ultimately care about.

Scott: Talk a little bit about the open source projects that are underneath what you do as a company, how you engage, and how the development of that works.

Shay: Kaltura is open source through and through, and we run everything on the LAMP stack. So underneath, for production, we use Linux, Apache, MySQL, and PHP. Our entire backend is in PHP, and the front end is a combination of technologies.

We want to maximize availability to publishers, so we provide a set of code in Flash, a set of code in HTML 5, and more recently, a Silverlight front end. It’s completely multi-platform; I run it on my Mac, our developers sometimes develop on Windows, and production is based on Linux. The code itself is a set of extensions on top of Apache.

Scott: Did you choose PHP for the presentation layer just because you wanted your customers to be able to easily modify it?

Shay: Absolutely, and PHP is also the back end. A lot of the presentation layer, for the most part, is the graphics hub. You could use a variety of libraries when presenting content in PHP, .NET, or Java. Ruby, too, is becoming more and more popular. The idea is just to maximize publisher freedom.

A publisher can take the entire set of functionality and just plug it into whatever kind of system they like. One of the things that we’re proud of is a lot of pre-made integration kits, so if you want Kaltura within whatever context, you can most probably go to our newly released Application Exchange, which is a marketplace for all the extensions, plugins, and applications that Kaltura, our partners, and our developer community create.

You can find kits that we have already pre-integrated with Drupal, Joomla, MediaWiki, WordPress, and many other systems like that, regardless of the operating system or the language you’re using for the front end. That takes the pain out of the integration.

Sean Campbell: Walk me through how you make money. When I look at the way you describe the five layers of your open source stack, you’re leveraging a lot of open source projects, but you don’t seem to have a fully packaged, open source version of the solution.

Shay: We make money in three ways. We bring to market the best of breed open source platform, where people that understand open source and have the development wherewithal and capacity to do it can go to our community site, kaltura.org, download the software, install it on their own, and never buy a single service from us.

In exchange for that, we expect them to play nicely with the open source rules. Any derivative work has to be contributed to the community, and they have to follow rules about attribution. Licensing for the code is under the Affero-GPL. So if they want to produce web services, they’re still regulated by the open source licenses.

From that group of people, we may never make any money. We have 60,000 publishers in our network, and the majority of them never pay us license fees. They use the software and contribute code or bug reports, or just become part of an active community. Some of them opt in to receive ads.

On the other hand, we make money from that group by selling them professional services, such as maintenance, support, installation, application development, and things like that. That’s one revenue bucket.

The second revenue bucket is a service business whereby people that understand the open source game still don’t have the capacity to do it, so they pay us to provide them software and services.

Using the same stack of code, we offer a hosted service where we host their files, we run the transcoding of the files, and we run the applications that are going to be integrated into their stack in a hosted manner.

The third revenue bucket comes from service providers. To hosting companies, cloud companies, telcos, system integrators and the like, the platform is also available as a self-hosted solution, under a commercial license with full support and maintenance from Kaltura.

Sean: I’ve heard a lot of people say that there are certain things they wouldn’t do with a cloud-based app, and a common thing people tend to bring up is heavy multimedia editing. It requires significant CPU, and you want to have full fidelity solutions like FinalCut Pro or even key solutions like iMovie or FinalCut Express or Pic Premiere on Windows.

On top of that, the open source community has at times struggled with desktop Linux OSs like Canonical to really get good embedded tools that would rival some of those toolsets. You guys have obviously spent some time on the editing side, and it looks like you have done really great work.

Talk a bit about the limits you have to deal with and work around, in terms of people’s user experience with uploading and editing.

Shay: I think that’s challenging, but we have a few solutions for that. One is that we do offer what we call the Self-hosted Edition, which has the capacity to run the hardened version locally. If you’re a service provider, an enterprise, or a university and you want to run this locally, you don’t necessarily have to run it from the cloud.

We give you a hardened version that you can install on your premises, and you own ROOT. You can control it, and the files are close to the destination so you don’t have to operate over VPN.

Depending on the details of the deployment, you might be able to enjoy economies of scale without having to have the complexity of cloud. We see a lot of adoption of this model in higher education, where many institutions have robust on-campus infrastructure.

I think you’re correct to point out that it’s only a matter of time until you can pretty much do everything remotely. People like Sun have been saying for ages that the “network is the computer,” meaning that, at some point in time, if the network is fast enough, you don’t really care where the CPU is.

In that sense, I think that what we have is a beginning. It’s not ready to replace high-end, broadcast-grade editing solutions any time soon, but people are using it for production, and you can run really complex work-flows around it.

Where cloud-based editing becomes essential is when you’re starting to think about collaboration. On one hand, if I’m editing my own files that I just uploaded from my camera to my computer, I have little motivation to run everything on the cloud, other than the benefits of automatic updates, auto-backup, and so on.

On the other hand, a group of students working on a class project doesn’t really have the option of doing it offline. In the context of enterprise collaboration or training, the need arises to gather information from multiple sources, and potentially from multiple geographical locations.

Therefore, you are almost mandated to have a network solution. For a network solution like that, the first tier replicates the broadcast model online. A provider has a library of content and wants to capture data out there. But that is just the starting point.

As we’re moving into the new realm of video, it’s about interactive media; it’s about collecting information from multiple sources and being able to transmit it. That’s the place where I think we are seeing a lot of adoption of the more interesting concepts around editing, remixing, annotation, content manipulation, and metadata extraction.

In my mind, all those fall in the same category of advance network services that have to do with collaboration.

Sean: People obviously love social media streams and the collaboration associated with them, but it’s somewhat uni-directional. Normally, I upload my video, and people watch it. It gets more interesting when they remix it or do something interactive with it. Some musicians actually like that their fans are doing that.

That creates an interactive world where, by lifting all that weight off of physical drives in everybody’s house and office, we enable new use cases. If it really is all up there and you’ve got good editing tools, you can blend and create environments and communities.

There’s plenty of video on the Web, but it’s mostly a broadcasting-type model. I’m hearing you say that you might be able to enable a certain level of community around video as a content stream, given the tools and the environment you provide.

Shay: Absolutely. I think that revolution is impending, and I think the first step we’re going to see at a large scale is on Wikipedia. Using this technology, we’re providing Wikipedia the same type of media collaboration that they know from text.

Some people are going to upload, some people are going to edit, some people are going to clip, and some people are going to transmit. It’s a completely open system that allows people to collect multiple media files, sequence them, edit them, and translate them, all in an open way.

It’s very exciting stuff, and two years of work are coming to fruition. The first phase just went live a couple of weeks ago. People are contributing content, and then later in the year, all the sequencing and remixing software is going to go live. That’s going to be very significant.

It’s been a great year for HTML5 video. Safari, Chrome, and Firefox already support it, and a few weeks ago, Microsoft announced that a new version of IE is going to support it. Google announced that it has open-sourced the VP8 codec. That kind of opens the way to creating a truly open HTML5 solution.

Sean: It seems that one of your potential business development strategies would be to look at properties on the Web that are already successful but also text-heavy, which would naturally lend themselves to a video overlay.

I imagine you also have your eyes on other text-heavy properties that are driven by community contributions.

Shay: True, but I think the opportunity goes beyond publishers like Wikipedia in the classic sense of publishing. It’s not only text-heavy content that is aimed to end users; it’s also other places where there is mainly text.

Think about university learning-management systems (LMSs) that are 95 percent text right now and maybe a PDF and an image here and there. Think about Enterprise Resource Planning (ERP) or Customer Relationship Management (CRM) software, and all the other places that couldn’t use rich media because it was too complicated. Now we can change all that.

We are giving the market a set of technologies that are enabled for rich media and video. I’ve seen statistics that show that video is going to be more than 90 percent of the traffic on the web. I think that’s a conservative estimate; I think it’s going to be somewhere like 95 to 99 percent of the traffic in the world in terms of number of bits.

For large education institutions or enterprises, this requires a huge shift in thinking. For these guys, it’s not about all the libraries, but being able to integrate into the platform that’s going to give them all the tools, because now, if you don’t do it properly, all of a sudden you’re going to choke on it.

Imagine you’re a university campus of 50,000 students and you have an LMS that’s running great. You have a new film professor who assigns his 300 students homework to upload movies to a campus web site. If they do that before you provide for it, 300 students each uploading half-gigabyte files will cause the system to collapse. True story.

In scenarios like that, you really have to think about the infrastructure. You have to use a new set of tools. That’s just one example.

Scott: You mentioned HTML5. For the longest time, as videos started to emerge on the Web, it all gravitated toward Flash, but now, the industry seems to be moving toward HTML5. Google is pushing it, Microsoft’s going to support it, and of course, Apple has declared war on Flash.

I can’t play at least 85 percent of the videos on the web using my iPad, because they’re all Flash-based. What are you seeing in terms of emerging video formats, and what opportunities do you think that creates?

Shay: I think a lot of the war is driven by adoption and devices. There’s no question that Flash is installed on a large majority of PCs and Macs, but that’s not the case on mobile devices. Flash doesn’t have any adoption on IP-enabled TVs, set-top boxes, Windows Mobile phones, Android phones, iPhones, iPads, and iPod Touches.

Only a relatively small portion of devices can support Flash, and while its absolute adoption numbers might be increasing because the PC market is still growing, it has declined as a percentage of the overall ’screens’ out there.

I think there is an opportunity for open formats to take Flash’s place, and HTML5 is one of those formats. HTML5 has always been a codeword in that sense, because it does not define the codec or the transfer protocols for video, so we’re going to see a lot of bifurcation in the market moving forward.

The promise of HTML5 is that it allows developers to write once and publish anywhere. You can include video tags on your page, and it’s the device’s responsibility and the browser’s responsibility to figure out the codec and the transfers that render the video properly. In that sense it’s a huge step forward, but I don’t think Flash is going to disappear overnight.

It’s a paradigm shift that is going to take a few years, and I think Flash has a lot of things going for it. We work very closely with both Microsoft and Adobe to maximize the benefits to publishers. The biggest opportunity it presents for us is that today, publishers are faced with a bifurcated audience, because they have to address the iPhone, the iPad, Android, set-top boxes, and PCs.

They need a set of tools that allows them to simplify writing and publishing their application without having to think about all the devices, and that represents to us a big opportunity to provide an infrastructure that allows developers to write just once. Our libraries are going to do the heavy lifting for them in order to deploy the app with the appropriate protocols.

Sean: Anybody who’s produced video for the Web, even to put a video on their blog, is immediately faced with choosing a bitrate, encoding, and whether to put it up as an MOV file, Flash, a WMV file, or whatever.

People trying to create interesting content don’t really want to worry about all those technical details, and there is no right choice anyway. It really depends on what a particular user has on their machine.

Like you said, if they’re using a mobile phone one format’s right, and if they’re using a Mac or a Windows PC, a different format might be optimal. I think you’re right that there’s a huge opportunity to deal with those issues and offload the content developer from having to worry about them.

Shay: Precisely. I think that no single company can solve the challenge alone. We’re big believers in creating standards in the market, interoperability, and distributed control.

We’re a founding member of an organization called the Open Video Alliance. That’s an umbrella organization that is bringing together all the people who care about open video. We had a wonderful Open Video Conference last year where more than 900 people showed up to talk about these issues, and 4,000 more were on the live feed.

This year it is going to happen again on October 1 and 2. I’m inviting the developer community and the people who care about it to actually come and participate and just vote with their feet, helping solve that problem on the ground.

We are also involved with the industry resource website called HTML5Video.org, which allows people that have interest to showcase the technology, explore the libraries, play around with it, and experience the potential of HTML5.

Scott: That’s great. Any closing thoughts from your end?

Shay: It’s an exciting time for this market, and we are maybe at the end of the first inning, with eight more innings to go. I think this is the same order of explosion that we saw with the first wave of the Web 10-12 years ago, with text.

I think next year, more than half of the TVs are going to be IP enabled, and half of the homes are connected to broadband. Two billion people have access to the production process, and imagination is their only limitation.

I think 2010 is the year of HTML5, and we are going to see massive adoption. 2011 and 2012 are going to be all about productization and adoption of new business models around video. We’re nearing a state where the plumbing of video, so to speak, is done, and the beautiful gardens will soon spring into full bloom.

Publishers no longer have to spend human resources on the infrastructure. The problems are beginning to be well understood—and solved with technology like what Kaltura brings to the market. Now people can start focusing on innovation and actually building the applications and what follows around it.

The Application Exchange we just launched recently is an excellent example of that, which allows people to think about higher-level applications, instead of plumbing.

How could video drive better distance education experiences? How could it drive better enterprise collaboration? How could it drive better publishing and media experiences? I think that’s the future we’re looking at.

Scott: Thanks so much for taking the time to chat.

Shay: Thank you. For more information check out:

• www.kaltura.com
• www.kaltura.org
• openvideoalliance.org
• html5video.org

Bookmark this:

The genesis of ReplayDIRECTOR: recording and playback for software

Managing the architectural complexity of simulating multi-level applications

Implementing the test and debug environment, and the results that follow

Handling the economic pressures of being a startup, especially in a down economy

The value of open source in an entrepreneurial setting

Scott Swigart: To start us off, take a second to introduce yourself and talk a little bit about Replay Solutions.

Jonathan Lindo: I’m one of the co-founders and CEO of Replay Solutions. We’re a software company that has developed a very unique technology that helps to automate a large portion of the software life cycle for anybody who is building or managing software applications.

My background is technical. I’ve been a software engineer for over 15 years, before founding this company in 2004. Throughout my own career, I’ve felt the pains and the need for a technology like ReplayDIRECTOR, which is our core product.

We’ve experienced a lot of the challenges and difficulties that folks are facing even more today, in terms of delivering high quality applications for the new types of environments in which people are now deploying applications.

Scott: Unpack that a little bit. Explain what ReplayDIRECTOR does and a little bit about how it does it.

Jonathan: Let me start off with a bit of a story to tell you how we got to where we are as a company. My co-founder, Jeff Daudel, who is our CTO, and I have worked together for over a decade, most recently at a company called Muse Corporation.

That company was building a very ambitious software project to deliver the next generation of the Internet, including a lot of new technologies like 3D graphics, multi-user browsing, spatialized audio, and video, in a 3D environment.

The idea was that companies like Paramount and Sony that have a lot of content could develop very rich 3D websites, while making them so simple from a user perspective that your mom could very easily access that content in a very compelling way.

Building that required a lot of different components. There was a highly distributed, heterogeneous back end, with a lot of databases talking to application servers.

What we found as we were trying to bring that product to market was that, with our large group of QA testers and beta testers, we were drowning in the sea of bugs and issues that were being reported. We were spending almost all of our time simply trying to reproduce and fix these problems that were being reported, as opposed to actually improving and shipping the product.

This was in the late ’90s, and TiVo had just started to become popular. Both Jeff and I were early adopters and big fans of that technology, and we still are. We both thought it would be great if we could take a similar concept to TiVo’s recording and replay capability to software development.

When a problem was experienced in the field or during testing, we would have a recording of it as it first occurred, which would allow us to easily trace back to the exact root cause. That was the genesis of the idea, and how it works is actually pretty simple. We basically hook into the software application as it’s running and record all of the sources of input that could potentially affect the software.

By doing so, we provide just enough data to allow the software to replay and actually reproduce, at the source code level, the exact root cause of any error. We reproduce the error in such a way that it’s actually debuggable. You can attach your debugger, like Visual Studio or Eclipse, step through the code, and see the root cause of the problem, right down to the responsible line of code.

Scott: It seems that there would be a lot of technical challenges to building that, in terms of capturing all those events, all those inputs, into the software, and being able to play it back. A number of things spring to mind.

Let’s say that a user logs into their online banking site, tries to pay a bill, and gets some kind of an error. You can’t literally just replay that transaction over and over, taking money out of their account and giving it to the power company each time. How do you address cases like that, where an error can’t be replayed in the production environment?

Jonathan: In your online banking example, a browser talks to a web server, and there’s also an application server, a database, and probably some authentication servers running. All of that interaction will result in a lot of different transactions taking place between the various components of your application.

That set of transactions speaks to the incredible complexity that now exists in application-deployment environments, which becomes even more pronounced with deployment into the very distributed environments associated with cloud infrastructures.

Our technology basically isolates each of those components as they’re running in the system. Let’s take the example of the application server. What we do is actually monitor the activity on that application server in a very lightweight, seamless way, and allow just those actions that occurred on the application server to be recorded and replayed.

The result of that approach is you can take the recording of the problem that we just described, an error in an online banking system, and you can bring that problem out of the actual production environment and put it onto a developer’s or a QA person’s workstation.

Without having to add access to the database, the authentication servers, or any of the other components, including the client itself, you can actually replay the problem and see it occur at a source code level. It’s not necessary to actually execute the transaction again.

When you replay, you’re actually running in a virtualized environment, so part of what we actually do is application virtualization. VMware and Microsoft are very popular for doing OS virtualization, and we bring that up the stack and virtualize at the application layer. That allows us to run software in a virtualized mode without impacting any of the surrounding components that existed when the recording took place.

Scott: Do I understand correctly, then, that you’re creating a virtualized test instance of the application server? When input comes in, the application server makes a call out to try to talk to the database and retrieve records. The replay is basically just replaying back the data that it got when it did that in the production environment.

From the app server’s perspective, it thinks it called to the database, and it thinks it got data back, but it’s really just getting a replay of that. It’s been transplanted into your test environment, but it still thinks that it’s in the production system and is behaving as though it were.

Jonathan: That’s a perfect way to describe it. We inject the same inputs back into the application and simulate the surrounding components such as the database and any clients that existed when that issue took place.

You can imagine that, if you had 1,000 clients connected to your application, one of the big benefits of a solution like ReplayDIRECTOR is that you don’t need to generate that load again. The replay will have captured all of that activity from those 1,000 clients, and you simply need to press “play.”

Scott: Do you run into issues when something needs to authenticate? My understanding is that some security systems are specifically designed not to let you replay a response of an access token, since that could allow a malicious user to intercept an authentication request as a means to spoof or hack the system.

I’m just kind of trying to poke around the edges, but do you occasionally run into some scenarios that you can’t replay, because of issues like that?

Jonathan: If you think about your earlier description, where the database is not required on replay, it’s the same for any other server, such as an authentication server. We are actually simulating and making the application think that there’s an authentication server present during replay.

When that request goes out to the authentication server to authenticate, our system will respond, in place of the authentication server, to say, “Yes, you’re OK. Keep going.” That’s what allows you to take that recording out of the environment in which it occurred and replay it anywhere.

Scott: I’m picturing little agents or something similar that do this lightweight monitoring. They are able to capture these inputs and outputs in your production environment, so you can take them back to the lab.

How does your hosted solution plug into that? That is, what’s the difference between doing this on premise versus hosted?

Jonathan: We’re really excited to have announced with Version 3.0 of ReplayDIRECTOR that we are now offering cloud hosting as part of our solution. What that means is, as you described, you’ve got agents that run on your application servers that are recording, either 24 by 7 or however you’d like to set that up.

The data those recordings generate is being sent to the ReplayDIRECTOR server, which can be run behind your firewall and is used to store and manage those recordings. It also provides a web-based interface that lets people share, collaborate, and access those recordings.

In the cloud-hosted solution we’re now offering, the ReplayDIRECTOR servers will run on our premises. We provide that as a hosted service, so the recordings will be stored securely on our servers. That’s the component that allows folks to get up and running really quickly, without having to download and install a server on their own.

Scott: I imagine that these recordings could result in extremely large amounts of data, depending on what’s flowing through the system and what’s required to service a request. Sometimes applications aren’t very efficient, and they’ll make a request to a database that returns a lot more data than it actually needs, and the application will parse through it, rather than correctly pushing the work to the back end.

Especially if you’re going to be pushing this data up to a hosted service, it seems like there must be some kind of tuning you can do to control what’s captured, how long it’s kept, and those sorts of things.

Jonathan: One of the nice things about our technology is that it’s very simple to get, configure, install, and start to use. It’s basically a very binary type, and there’s not a lot of tuning required. We see that as a very good thing.

We’re able to do that because the recording system itself is extremely lightweight. It can be run all the time, even in a production environment, and it’s there when you need it.

We’ve gone through a lot of research and development effort over the last six years, building up this technology so that it’s extremely lightweight and records as little data as humanly possible. The data it does record is actually compressed and encrypted as well, which further reduces the data size that’s generated.

How long the recordings are kept is also tunable. If you want to allocate 500 gigabytes of disk for your recordings, that’s often going to be enough to store more than 30 days worth of recordings. You can also tune it to keep a rolling window of the last seven days, for example.

Scott: Obviously, the primary scenario is that an error happens in production, or there’s a potential bug, and you need to track it down. Have you had people use it for other things? I could imagine, for example, that people could use it forensically to find out what happened when their web server got hacked.

Jonathan: There are really four areas where this technology has been used successfully. The first is in problem reproduction and resolution. The second is in security, as you mentioned, which allows people to go back and look at any security issue, as it occurred, with as much detail as they would like.

Third, this is a very powerful technology to analyze and resolve performance issues. Finally, it allows any kind of compliance-related issue to be verified, reproduced, and audited. We’ve had applications of our technology outside of problem resolution and with good success.

Scott: Without divulging anything about your customers, obviously, what can you tell us about the spectrum of uses for this technology, maybe from small startups to relatively massive implementations that require a lot more scale?

Jonathan: Let me give you an example of a very large software company that has a lot of products out in the field with thousands of customers. They had a product that was used on a daily basis by large software teams, and every couple of months, they’d get a report of a bug that was corrupting a database.

It was so bad that people would have to go back and actually restore the database from the last backup that they’d made. This was the kind of problem that they could never reproduce in their pre-production environment. They ended up making seven attempted fixes to this problem, and one after another, all seven failed in the field, and another problem report would come in with a corrupted database.

When they started using Replay, they captured the problem with a recording, which allowed them to reproduce the issue in literally about 10 minutes. All they had to do was to access the recording, press “play,” and watch the problem happen again. Their next fix nailed that bug, which it turned out had been in their system for over two years.

Scott: Talk a little bit about the business side of the company. These are interesting times, as they say, economically. What can you tell some of our readers who may aspire to starting a company about your lessons learned and that sort of thing?

Jonathan: In 2004, having identified this as a major pain point for us and for most people in our industry, we decided to found a company. We went out with a prototype of the technology, early on, and we started working with companies directly.

We were fortunate enough to be able to use some of our contacts to start working directly with companies before we even considered raising money or building up the company. At that time, it was just me and my business partner, Jeff.

We built up a small stable of pretty notable customers, including Microsoft, NVIDIA, and Electronic Arts. We were able to use those relationships to bootstrap the company and continue our research and development.

A lot of man years of research and development have gone into this technology, including about a dozen patents, two of which have now been issued. We’re very excited to have reached this point, but it certainly was a bit of a road.

In 2006, we had built the company up to a handful of folks. We had reached profitability and then decided to go and look for an opportunity to grow the company. We did so by partnering with two venture capital firms in Silicon Valley. One was Hummer Winblad, and the second was Partech.

With them, we were able to grow the company and really expand the breadth of our technology, to build a more ubiquitous solution that could be used by even bigger teams than in the past. We’ve continued that research, and we actually raised our second round of funding in 2008, with Sigma Partners and UV Partners.

Since then, the economy has really started to rebound, and we’re seeing the benefits of that. We had a record 2009, and we have started off very well in 2010. We’re hiring, we’ve expanded our office space, and we intend to meet or exceed our targets for the year.

We’re certainly seeing some good signs in the economy. People are coming back and looking at investing in technology that will help them be more efficient in the new environment in which we find ourselves.

People are looking for solutions to help them with the present complexity of the software environment, which includes cloud computing, and we’re one of the new players on the scene to really address those kinds of issues.

From a business standpoint, people are looking to be lean, mean, and efficient, but they are also looking to be strategic, and we’re benefiting from that as well. Some of our large customers in the financial services space are really taking that attitude, and it’s been good.

Scott: Looking back at the very beginning of your company, there are a couple of tensions that I think any startup faces. That’s especially true if you’re going after a lot of large enterprise customers, and I’d be interested to hear how you dealt with them.

The first of these problems I’ve seen is that a big company might be very interested in your technology, but they could be afraid that your little company won’t be around in a year. They may refuse to make a big investment in it that includes implementation, training people, and so on, because you’re simply not big enough yet.

The other common problem is that they may think the base technology is great, but your product may be missing one or more features that they really need. At that point, you have a conflict in staying true to your vision for the product, while at the same time servicing that kind of one-off feature requests. They may take your product in a direction that’s not strategically beneficial to your company as a whole.

Jonathan: There’s no question that we have felt those pressures during the course of our company’s lifetime. We’ve approached that first question, about convincing large organizations to make a financial and workflow investment, by building our software to be extremely seamless and lightweight.

That means that people don’t really have to make significant changes, if they have to make any at all, to the way they build and manage their software. We’re very complementary to the existing workflows, technologies, and tools that folks are using today.

We have painstakingly taken the approach to keep our technology as transparent as possible so it fits into existing workflows. That means that, if they did need to remove our technology from their workflow, it would not cause things to break.

Obviously, it would make their jobs more difficult not having Replay, but it’s not something that they’re betting the farm on, in terms of the way that they go about their daily work flows.

Your second point concerns how to react and adjust to pressure for custom development or feature requests that might not be strategic for our company. We’ve approached that, again, by building our technology to be lightweight and seamless, but also to have a low entry point.

By having a broad appeal, and by virtue of our technology being scalable and robust, we can afford to take on smaller customers, rather than just the $500,000 deals. Our low entry point in terms of price means that we win out on volume and don’t necessarily have to bend over backwards to win specific deals.

We take a long-term view in terms of how we engage with our customers. People might start off by deploying to their pre-production teams in one or two departments, and then they might get comfortable with it and grow with the product.

Scott: I’d also like to talk a bit about your approach of bootstrapping the company for a couple of years, and then going out and looking for venture funding. It seems to me that I have heard a lot of people talking about VC funding really having dried up in the 2007-2008 timeframe.

I think there are a lot of people out there who think that they can walk into a VC with a set of slides to show off a great idea and fairly easily get funding. What advice can you share about walking down that path of bootstrapping versus VC, as well as the right point to switch from one to the other?

Jonathan: Walking into a venture capital office with a slide deck is a very rough hill to climb, unless your last three companies were extraordinarily successful. It’s also the case that today, people can get very substantial infrastructures online with very little capital investment.

It is really a new world in terms of being able to build substantial modern technology, get it online, and start building a customer base or a community around it with very few resources. Then you can start thinking about venture capital or other forms of investments, or not.

There are options in 2010 that didn’t exist in 2004, in terms of your ability to go out with a small team and do incredible things. Our premise was to prove to ourselves that our idea was really viable before anything else. We did that by building up a base of about 15 customers that were really getting value from our technology.

Once we had done that, we felt like we had a proof of concept on which to build a profitable company, so we decided to see if we could expand at a more rapid pace. At some point, you do have to be concerned about other companies coming up and chasing down those technologies.

It is a bit of a balance, and I think today, VCs are more risk-averse then they have ever been before. It is a little bit more difficult to go and attract venture financing, but on the flipside, it is also much easier to do more with less.

I think it is a great time to be thinking like an entrepreneur, and if you have a great idea, go out there and try to make something happen. If you look at what has happened with the iPhone App Store, the Google Apps Store, and soon Microsoft’s Windows Mobile 7, there are many opportunities to go and build new technology and bring it to market.

Scott: The majority of people we talk to are involved with open source projects, or their business is wrapped around an open source project. There is the perception in some quarters that open source is trendy, and there is more interest in a product that has an open source component to it, versus something fully proprietary.

Obviously, you have been successful during a pretty tough economic time, so what is your view about the perceived necessity to have some piece of the puzzle be open?

Jonathan: I think open source makes a tremendous amount of sense for certain sectors, although I am not convinced that the problem of how to monetize successful open source projects has been completely solved.

I think that is an ongoing area of experimentation and research. There certainly have been some good successes with open source companies that have been acquired, but in terms of really generating a lot of revenue in that space, I think there is still a lot of work to be done.

Because our technology is so new, and there really isn’t a lot to compare it to out there, we have decided so far not to open source the core component, although we do open source pieces of it, such as our plug in.

That is certainly an area we are looking at constantly, and we are big fans of the open source community. We actually are providing our software free of charge to many open source groups, and certainly we will continue to do that and expand our engagement with the open source community. In terms of open sourcing our technology, I am still looking for some good business reasons and benefits to the company to do so.

Scott: Well, we have covered a lot of great ground. Are there any closing thoughts from your end?

Jonathan: It is an exciting time to be developing software and building new technologies. We like to think of ourselves as a great example of a company that is taking advantage of the new paradigm in software. Our technology, we think, is really helping folks to capitalize on that, so we’re hoping that people will check out ReplayDIRECTOR and see what it can do for them.

Scott: Great. Thanks for taking some time to chat.

Jonathan: Thank you.

Bookmark this:

Defining the role of static analysis in the software industry

Targeting static analysis where it can have the most benefit

Interacting with the open source community

The role of automated defect analysis in commercial software development

Areas for future study and development

Scott Swigart: Why don’t you introduce yourself and your role at Coverity?

Andy Chou: I’m chief architect at Coverity. I really have many different hats, but basically, I’m responsible for the technology direction of our products. Coverity was founded seven years ago, and we build products that help companies improve their software integrity. Our flagship product performs static analysis to find defects in source code automatically.

This is an interesting space, because we do software development ourselves, but we also analyze other peoples’ software. We get to see a wide variety of processes, tools, and products.

We started out as a research project at Stanford University. The potential of static analysis technology was clear after we spent one weekend in 2000 and discovered hundreds of defects in the Linux kernel.

In 2003, we commercialized the technology and bootstrapped the company for four years without much external funding. Since then we got our first round of VC funding, and we have been expanding internationally and growing our product line since then.

Scott: Various open source projects use static analysis, and I know that Coverity has scanned something like 280 open source projects and provided those results back. Some projects more than others have worked to root out the issues found.

Before we go there, though, talk a little more about integrity scanning and higher-level issues related to software quality and design.

Andy: One of the things we discovered as we started to expand is that companies are interested in more than just the code in terms of software integrity. They’re interested in how the code is designed. They’re interested in how it’s assembled by the build process. They’re interested in how it’s tested and whether there’s anything that can be found by automatic tools during QA.

We have different modules that together form the Coverity Integrity Center. It’s our combined offering that addresses different parts of the development life cycle.

There are a lot of tools in the Application Lifecycle Management space that focus on streamlining and automating processes in software development, like tracking bug tickets that come in or doing source control management.

We consider ourselves to be in an adjacent space because we analyze the artifacts of software development and really dig into how they are put together and what they are doing. For example, a source control system treats source code as nothing more than a pile of text. We treat source code at a more semantic level, by parsing it and analyzing its behavior on specific paths.

Scott: When you’re talking about analyzing code, you have to consider the broad range of programming languages out there: C#, Java, PHP, and all kinds of other stuff. When you’re getting more into the design and build process, even though there are artifacts, there are also build instruction files, for lack of a better term, and things like that to consider.

How did you choose where to invest? Back in the day, organizations used a fairly discrete set of technologies, whether it was .NET, Java, or C++. As time went on, we got Ruby on Rails, different kind of cloud platforms and services, development for handhelds, and whole ecosystems growing up around those.

I assume that to support each one of those requires a separate investment from you, so how do you decide where you’re going to focus your resources? It seems like it would take a crystal ball to determine what’s going to be around for a long time.

Andy: We focus primarily on areas where software integrity is essential, for example safety-critical or business-critical systems. Often, this means embedded software. For example, software that’s going into a device, into a plane or car, or into a router, where it can be very, very expensive to fix a problem (think recall, or flying an engineer on-site to debug a problem).

The amount of software that’s in these devices is growing tremendously. I think it’s not really widely understood that software systems have become truly enormous over the past decade or so. For our customer base, a million lines of code is not considered very large.

If you printed out a million lines of code, it would be a six foot high stack of paper. It’s far too much for a single person to truly comprehend all the details. It’s hard to imagine any human product on that scale that has no errors in it. Some of the larger individual code bases we work with run into the tens of millions of lines of code.

And by the way, that’s just one variant of that code base. We all know that in real software development, there are many branches of development that share a great deal of the code. We also have products put together by sharing different pieces of software across teams, and the complexity grows tremendously.

It’s always a challenge to understand where to invest in order to maximize your impact, but we believe that listening to our customers is of paramount importance. They tell us which types of their software are most critical, and the most critical stuff is where automated scanning is most valuable.

Scott: You mentioned embedded devices, which makes a lot of sense, since they take a long time to develop, and they are very difficult to patch. You can push a patch to some types of embedded devices, but there others that don’t have a network connection, and it basically takes a product recall to make changes.

Cars are a great example that I think bring the fact home to the general public that software is everywhere.

Andy: The average person probably thinks of software as something they interact with when they turn on their PC, but the reality is that most of the software we all interact with on a day-to-day basis is not on a desktop computer.

Some of it is in your car, your mobile phone has tons of software on it, and you use software almost every time you purchase something. Furthermore, when you get a piece of information off the Internet, you use many layers of software. It’s not just on your computer and the website you’re going to, but also on all the infrastructure in between – the routers, switches, wireless access points, firewalls, and database servers. There’s an incredible amount of software that supports mundane everyday activities, and that’s not really well understood.

At Coverity, we think in terms of the software supply chain. When companies put together applications or devices, they’re no longer developing all of it themselves. So much software development is about pulling together components from different suppliers (including open source), integrating it together, and customizing it.

Scott: Let’s switch over and talk a little bit about the work you do with open source projects.

Andy: We worked with the Department of Homeland Security (DHS) putting together a scanning service for open source projects called Coverity Scan. That effort began in 2006, and today (early 2010) we scan over 300 different open source packages and have found over 11,000 defects that have been fixed by the open source community.

The DHS was looking for a way to have a big impact on software integrity across a spectrum of different kinds of open source projects. They saw that static analysis technology could apply to millions of lines of code and provide a quick way to harden open source infrastructure. Open source is used by virtually every company and government agency.

The economics of using shared code is compelling. But the downside is that defects in the shared infrastructure propagate everywhere. I think the DHS was concerned about this growing ubiquity, and what could be done to accelerate fixing more defects earlier.

Scan has been really successful in terms of raising the awareness of what static analysis can do.

Scott: You bring up a good point about evangelizing the benefits of static analysis. I think for a long time in the open source community, there was the notion that once code has been posted to a mailing list and looked at by a lot of people, that scrutiny will find all of the bugs.

When Coverity started scanning projects and turning these reports over to the maintainers, some projects probably really appreciated it, while others may have been more inclined to dismiss the results as a bunch of false positives. Some people would hesitate to believe that a program could really understand their code.

How have you seen the thinking about the benefits of static analysis evolve among open source projects over the past three years or so?

Andy: There has been a greater embrace of static analysis, both within the open source community and among commercial software makers. Part of the reason is that static analyzers have gotten better. They’re starting to think about software more like a person would and less like a machine enforcing rules blindly.

Part of why we think static analysis is being adopted more is that it has begun to provide enough information to understand the problems that could arise if you don’t fix a particular bug. That makes it more valuable and clearer that there is a real issue.

Some open source communities have really taken the results and run with them, fixing virtually everything that’s found by our tool, whereas others are a lot slower to adopt it. We’ve found a high correlation between the projects that do static analysis and doing other things like extensive unit testing that also help with quality.

Scott: In other words, certain projects broadly adopt tools and techniques to raise quality as part of their culture. Those measures might include static analysis, unit testing, and maybe other things.

Andy: And they see static analysis as a useful technique that can address some of the gaps left by other techniques, in terms of achieving coverage of all of the code without necessarily having test cases that cover everything or missing things that a developer might overlook.

Static analysis is good at looking at all of the different paths in the software, which people are not so good at.

Still, it’s been very difficult to get companies and open source adopters to really incorporate static analysis deeply into their daily workflows. We have found that, if you incorporate testing for defects into a nightly build process, for example, it really helps.

In that scenario, the developers are talking about defects that are fresh and about code that they just wrote, which really brings up the relevance level. Even for commercial software development, that matters quite a lot.

For a lot of companies that use our software, although not all, they see the initial set of defects that come out of the tool and they say, “Well, it’s too much work to fix all of those right away, so we’ll baseline that and fix everything that comes in new after today.”

Scott: That makes sense. Like you said, if they’re running it over a million or ten million lines of code, and thousands and thousands of defects come out, they might say, “Well, the software works today, and we’re not getting a lot of complaints about the quality, but we want to continue to raise the bar.”

It also seems like a pretty good educational tool. You check your code in before you go home at night, and in the morning, there are some defects waiting for you. That could very quickly help you change your own development practices to avoid defects.

Andy: We recently published a paper “A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World” in the Communications of the ACM that described some of the lessons we’ve learned over the past few years in terms of making static analysis work in a commercial setting.

One of the continuing lessons we’ve learned is that incorporating a new technology like static analysis into real, commercial software development really requires you to understand how people work and some of the complexities of their environment.

One example is that in our forthcoming Coverity 5 version, there’s going to be support for multiple code branches and identifying defects in shared code, which I think almost every software development shop has. They have a release version, a development version, perhaps a branch where they’re doing advanced work, and perhaps branches for customers that have special sets of requirements.

That is a very common scenario, but all of those different branches are variants of the same code base. It’s the same when you have different targets. You may have a Windows version, a Linux version, and versions for other target platforms.

Even in embedded applications, each device may have a different CPU or set of devices and, therefore, a different layer on the bottom to interface with the low level components. Some phones have cameras and some have Bluetooth, but not all of them. The variants between these different products and branches are part of the real life situation in software development.

Supporting that well requires more than scanning a code base and finding a bunch of bugs. It means digging into where else the same bug may arise. If I understand your branching structure and the ways in which you put together all of the software components into different products, I can tell you which products might be impacted by this bug.

That can be really useful. If you have a bug and you fix it in one branch, you may not realize that it’s also present in others. If you don’t also find it there and tell the developer about it when it’s timely, you may miss out on an opportunity to save quite a lot of time.

Scott: You mentioned that you’ve got an upcoming release, Coverity 5. Give us a little bit of a preview, in terms of what is coming in that version, and even some of the interesting problems that you’re thinking about beyond that.

Andy: One of the features we’re excited about in Coverity 5 is defect impact mapping. It gives developers a quick way to see what the most likely high-impact defects are. Instead of thousands of defects in a flat list, you can focus in on those that are most likely to be critical, such as those that involve things like memory corruption and memory leaks, which can be very expensive to find and to fix.

When you combine insight into which defects are critical with which defects occur in multiple code branches or streams, it’s a huge step forward in understanding the impact of defects on the business.

A third feature is that we’ve mapped all the results that we find to the Common Weakness Enumeration, the CWE, which is an industry standard way of talking about software vulnerabilities. It’s something that our customers have been asking for, because it allows them to understand defects not in terms of how Coverity classifies them, but also in terms of how the industry is talking about them.

Scott: You’ve talked about a lot of ways of finding and analyzing defects. What are some of the hard problems that are left? I think that, in terms of compute problems, this is probably right up there which speech recognition and automatic language translation, in terms of being a really hard computer science problem.

Andy: In our space, every time you turn around you face an NP-complete or undecidable problem. What we’ve found is that you really need to be careful about the trade-offs made when finding defects. You simply can’t find all defects with a low false positive rate, while scaling to millions of lines of code while remaining almost completely automatic.

The balance between these many factors is something that you have to strike very, very carefully, because a tool needs to work for very large code bases as well as very small code bases, and on code bases written using different styles.

One of the challenges we continually face is how to make software analysis something that really customizes itself for the particular code base that’s being scanned. You should treat code bases like patients, in the sense that you don’t treat each patient exactly the same way.

If you look at their history, the processes used to develop them, and the idioms used in the code base, you may come up with a different set of diagnoses about what’s wrong. That’s a relatively new field. We’ve done some of the work in our products, but I think there’s a lot of room to grow in that dimension.

If you’re talking about a web application, for example, security may be of paramount importance, but if you’re talking about embedded device, you may have a different set of concerns. The blind scanners that just look for a list of problems don’t necessarily have that context, so they may look for things that are lot less relevant, just because they don’t understand what you’re trying to do with this application, how you developed the software, and what idioms you’re using.

That’s one area for future work. From a top down point of view, I am interested in pushing software analysis more deeply into the work flow and incorporating information from different sources.

Correlating the information between different systems that track bugs and track changes in source code is another interesting area for the future. There’s tremendous value in converging information from many different sources and using that to better determine whether software is ready to release, to gauge its quality, and to determine which teams are doing a good job.

Scott: This is definitely an interesting space, and it’s good to see it becoming more mainstream. We’ve covered most of the stuff I had on my agenda, so do you have any closing thoughts?

Andy: Despite the growth in the past few years, this space is still kind of nascent. I think the market is large for tools that are starting to integrate information from different software components together.

One of the things that we’re looking at is how to increase the quality of the supply chain, in addition to just individual software projects. That’s an area that we think will require some automated analysis. It’s too expensive to manually look at code bases that are being produced by third parties or by suppliers.

There’s no common regulation or standard by which software quality is measured today, so going beyond just finding defects, how do you determine what is high quality software? I think there’s an open question out there, because today, there is no real metric for quality.

There’s a lot of work to be done in that area, and that’s going to make for an interesting future.

Scott: Great. Thanks for taking time to chat. This has been a great interview.

Andy: Thank you.

Bookmark this:

• Breadth of open source business intelligence offerings from Jaspersoft
• The rise of information as the key asset owned by many businesses
• The role of the open source model in Jaspersoft’s success
• Transition in business intelligence toward smaller companies and less expensive solutions
• The role of cloud computing in the future of business and government
• How the business intelligence layer fits into the larger cloud computing model

Scott Swigart: Please introduce yourself and Jaspersoft, as well as the company’s relationship with open source.

Brian Gentile: I’m President and Chief Executive Officer of Jaspersoft, which, we’re happy to say, has emerged as the world’s most widely used business intelligence software. We’ve very successfully used the open source model of development and distribution to reach this point. It allows our software to be, in a sense, downloaded and put into use by virtuallly anyone who has the acumen or even interest to solve some data problem using a BI tool for reporting and analysis.

Today we sponsor and promote five major business intelligence projects that are also signature products. JasperReports is the world’s most widely used reporting engine and library, and iReport is a companion report design and development product that sits on top of JasperReports.

JasperServer is a repository-based report and analysis server that provides scheduling, authentication, and all sorts of server-based business intelligence functions. JasperAnalysis is a full relational OLAP environment that works side-by-side with JasperServer and provides multi-dimensional capabilities through the ROLAP engine.

Lastly, we sponsor the JasperETL Project, which allows for very sophisticated data integration, enabling extract, transform, and load functions in creating a data warehouse.

The combination of all five projects we refer to as the Jaspersoft Business Intelligence Suite, and we take it to market first in community releases, which means freely downloadable, mostly GPL-licensed releases, and second, in the form of commercial editions, which we refer to as the Professional Edition and the Enterprise Edition.

These two commercial editions have features and capabilities that go above and beyond the community-oriented project releases. So for our commercial customers, there is a feature-based reason to potentially subscribe to and pay for the professional or enterprise editions of our suite.

Scott: In its early days, a lot of open source was infrastructure-oriented, and in many cases, it was written for developers, by developers, such as the Linux kernel and the Apache web server. Now, we’re seeing success with companies like yours that offer software that operates at a higher level, along with open source ERP systems, CRM systems, content-management systems, and those kinds of things.

Talk a little bit about the trajectory of open source evolution. Why is now a good time in history for these higher-level products to emerge in open source, where it used to be more infrastructure?

Brian: Let me use some numbers that provide clear testimony that the world and its development model has surely changed. With our business intelligence platform, we are now approaching volumes that are close to those infrastructure open source projects that preceded us. For instance, we surpassed the 10 million download mark in January 2010, and we’re closing in on 10.5 million downloads today.

Right now, we have a new community member registering at the Jasper website about every five minutes. There are more than 125,000 registered members, and we have more than 70,000 forum posts. Forum posts are where our community interacts, so they’re helping one another every time they post something to solve one another’s problems, answer each other’s questions, and so on.

We now have what is known clearly as the largest ecosystem in all of BI, because of the partners, contractors, community members, and developers that are participating. It really is fundamentally changing the way software is built.

Just as the Spring Framework, Hibernate, and a variety of other development tools have completely transformed and enabled the LAMP stack and the new world of web development, Jaspersoft now is the BI layer in that world.

The volumes and the kind of development that is occurring, the kind of people that are developing and creating applications for which BI now becomes a web-based layer, has expanded dramatically over the last few years. And it has expanded because of the existence of all these open source products and different layers.

Therefore, it makes all the sense in the world that open source software has marched up the stack. Jaspersoft dwells at a level in between middleware and applications. We’re not quite an application, but we’re certainly more than middleware. Our providing the business intelligence layer within a web application frees the developer to focus in his core competencies at the application level. And, assembling applications rapidly based on the new web model and using open source tools whenever they’re available is clearly the new design point.

Scott: Historically, the need for content management systems, CRM systems, and ERP systems goes back a long way, so it seems obvious in hindsight that open source alternatives have become available for them. There’s a relatively longstanding need that is relatively well understood.

CMS and CRM are slightly newer, but even those go back a decade in terms of proprietary solutions. Business intelligence, on the other hand, seems to be driven by a different force. It seems like we’re more addicted to data than we’ve ever been.

More and more, the standard seems to be just to save everything. Save every search query that anyone enters, and save every detail about every interaction with the customer. Just as an oilfield or a super collider produces terabytes of data in a really short time, now even mundane business tasks produce terabytes of data, too.

This addiction to data has driven the need to have really good business intelligence, because it’s just too much to look at and make sense out of any other way.

Brian: For the last 20 or so years, information and its use have constituted a large percentage of a company’s relative value. 100 years ago, perhaps 90 percent of the company’s value was based on its ability to manipulate land, labor, and capital–the fundamental factors of production.

As we moved into the late last century, information technology began to drive an important part of the economy as we became so much more capable of using it effectively. Today, the management of information makes up the vast majority of the value of many companies.

For example, one could argue that nearly 100 percent of the value of a financial services firm is in the information that they command and maintain. Vast amounts of data exist because that data is increasingly valuable, even for an old guard steel mill or manufacturing company.

That trend makes it very important to efficiently manage and manipulate that information. Value is created from insights gleaned from that information, by increasingly large numbers of people. The data needs to be distributed, decentralized, and democratized so it can be used in the largest possible variety of ways, as quickly as possible.

That is a fundamental reason why open source and lighter weight, faster tools for business intelligence have become so popular, and their popularity will go nowhere but up over the course of the next decade.

Scott: Concerning the democratization of data, not that long ago, companies had canned reports that they printed on a big line printer on a weekly, monthly, or quarterly basis. Someone had determined ahead of time what data and reports were useful, in a fairly inflexible way, and they went into a binder on a shelf.

Today, there’s the understanding that the company can’t fully predict how that data might be used. Individuals may have very specific problems that they need to solve, and to do so, they may use data in a unique way that is not apparent or useful to anyone else.

Brian: That phenomenon is exactly why Gartner estimates that about 40 percent of the budgets for business intelligence are now pushed out to the business units, either consciously or unconsciously.

Those who really understand the data and need to make use of it for better business decisions and faster insights are not in IT, and they’re not at headquarters. They need to be able to make decisions about which tools they’re going to use, how they’re going to put them in place, and which data they’re need to access and manipulate.

We’re seeing a fundamental shift toward working with departments and enterprises who are now creating software and using the tools inside of that software at a faster pace than we’ve ever seen before.

It’s all being done using web techniques and thin-client tools, and they’re doing it in days and weeks rather than months and quarters, for tens of thousand of dollars rather than millions of dollars. That trend is the next stage of the democratization of data, and it is largely enabled by open source.

Scott: Open source has clearly been successful for Jaspersoft, and as you said, you are getting millions of downloads. What is it about the open source nature of the foundation that has made your approach so successful?

Brian: Based on surveys that I’ve participated in and read, first, the innovation and the capabilities available in open source software are simply not found in their proprietary alternatives, if those proprietary alternatives even exist.

Second, open source affords companies flexibility. Some enjoy the fact that they have the source code, some enjoy the fact that they have access to lots of other people who are using it and can share ideas and improve it, and all of them like the idea that they have an alternative to a proprietary player, again, if one exists in the category.

Many of the senior people in these organizations that have been successful in using open source would go on to say that it prevents them from feeling locked in to a proprietary vendor who can constantly increase prices or put pressure on them to do things that they might not otherwise want to do. If there is any lock-in in open source, it is surely at a much lower cost and a much lower threshold level.

Today, because of those factors, we’re seeing what I’ve referred to as the mainstreaming of or the mainstream use of open source. As a result, we are seeing real depth in the use of open source, even among those who are not on the bleeding edge of technology.

One of the best examples is the public sector’s adoption of open source. Governments around the world, by definition and by charter, have never really been on the edge or in the vanguard in their use of information technologies. And now public sector entities are adopting and using open source now in big numbers.

The clear advantages occur in terms of innovation, flexibility, and avoiding lock in. Notice that I didn’t even talk about low costs, which would probably come fourth or fifth in importance for those enterprises that have been most successful with open source.

Scott: My experience with government customers has been that one of the reasons they love open source is that their procurement process is so onerous. The ability to use a free, open source tool that doesn’t require approval, a competitive bidding process, RFPs, and all that kind of stuff is very attractive to them. They can have an idea this morning, download some stuff this afternoon, and have a prototype mocked up by this evening.

Obviously, though, you guys have got to make money, and obviously the community of users is going to be larger than the community of paying users. But talk about how a government user translates from free to fee, in a way that kind of keeps your business going and lets you keep contributing back to the underlying projects.

Brian: Their use of the free, open source tools is no different than in private enterprise, in the sense that they can, just as you said, download something and get started on a prototype or a proof of concept on their own time. There’s nothing about the licensing techniques that would prevent a public sector employee any differently than a private sector one.

The good news is that the public sector’s big enough that there plenty of employees willing to do what they need to do to get their jobs done better, and that’s a good thing for taxpayers all around.

But then your second point is the procurement process, which I cannot deny is certainly more laborious than in a private enterprise, although I can point to some of our most difficult contract discussions and lengthy procurement processes actually coming from the private world and not the public sector. There are many private-sector companies that would give any government a run for its money in terms of ridiculous procurement processes.

Your point is accurate, though; it does take longer on average in the public sector. As an open source company, we have to have the ability and patience to work through those issues so we can get on purchasing schedules and so forth. It’s all about making the purchase process easier for them. It takes time, but it’s absolutely worthwhile.

And, that’s the right thing to do for public sector spending in general. I’ve written plenty about the need for governments to embrace and use open source wherever it exists. Using our taxpayer dollars more efficiently than they would be otherwise is definitely worth the effort, even though it can be difficult.

Scott: Different companies that have an open source foundation unpack the difference between the community version and the paid version differently. Do your customers find value mostly in features they want that are only available in the enterprise version, or is it more the support peace of mind?

Brian: There are three or four major reasons that a customer would choose to move to a commercial, paid version instead of the community version, and I’ll list them in no particular order, because it’s going to vary by customer.

The first reason is that there are features and capabilities in the commercial versions that are not in the community version, and those features and capabilities may make the paid product better aligned with what they need to do.

Our community versions are clearly capable products, so what we’ve chosen to put in the professional or commercial versions is very significant. Any commercial company, I think, would look at our commercial products and say, “Yeah, that’s absolutely worth paying for.”

The second reason is the license. Some customers are not comfortable with the license that governs the open source code. Usually, it’s in the range of some relatively permissive license, like the LGPL or Apache or something like that, to a relatively restrictive one, like the GPL.

Each of those licenses comes with terms that make it less interesting for some companies to put the software into use. They would rather have standard, commercial terms around the software and its use. By going to a commercial version, customers have a commercial license which includes the indemnification and protection they want to go along with it.

The third reason is professional support. The forums and the support that you get from working with the Jaspersoft community are pretty robust, but if your use of Jaspersoft is relatively mission critical, you probably want the comfort of knowing that you have a support team that’s very intimate with the code and available to you 24/7.

Scott: Another area that’s interesting to me is the notion that business intelligence used to be used by the largest companies, and it was sold to them by very large software companies. These were very expensive solutions and implementations.

Talk a little bit about the need for BI in the small and medium size business sector. What kinds of examples have you seen where BI has become increasingly critical to relatively small companies?

Brian: This is one of the most important and fundamental shifts that has occurred in the last 10 years: the size of the company now has almost nothing to do with the size of the data problems. We have many small to mid-size companies that need to analyze enormous volumes of data, in very sophisticated ways and in almost real time.

One of my favorite examples in our portfolio of customers is a small company called CrowdStar, which creates Facebook games.

They need to analyze, in almost real-time, 20 million records of data at least every day to understand patterns of movement and player activity within their games. They can modify those games during the course of a day to enable more activity, which for them equates to more use and more payments.

They analyze all that data using Jaspersoft in the cloud, which they pay an incredibly small amount of money for. CrowdStar is not a very big company, but they use big volumes of data, and there is literally almost no other way they could afford to do this without Jaspersoft in the cloud.

I think that’s a quintessential example. There are others that are less dramatic, perhaps, but they are equally important in the magnitude of data that needs to be analyzed and the size of the company that is behind it.

This trend is causing all kinds of technological shifts, and it has triggered what I refer to as “therefores” in the world of business and technology. By “therefores”, I simply mean that they are a second order effect, caused by the sea of data so commonly barraging any size enterprise. Examples of these “therefores” are increased interest in columnar orientation analytic data stores, proactive analysis, and predictive-like analysis. There’s much more interest in elegant design of responsive mechanisms for data analysis, such as dashboarding methods of quickly drilling into data, even through multiple dimensions.

It’s created a need for fast, successful small projects. It has pushed, as we described earlier, more of the emphasis out to the business unit levels, so they can have these small projects around where the data is best known and understood, and they can accomplish things more quickly.

So this host of business and technology “therefores” has come from the enormous amount of data now that is being analyzed by almost everyone.

Scott: There are so many different trends converging that let an entrepreneur go from an idea to a product in less time, with less capital expense than ever before. It used to be that, if you had an idea for software, step one was finding an office. Step two was buying servers. Step three was hiring a bunch of people.

That all burned through a lot of cash trying to get a product built, before you even had something that your first customer could look at. Today, you don’t have to buy a server. Even before the cloud, hosting providers offered dedicated servers, and then they started offering virtual servers.

Then clouds came online like Amazon EC2, and you started to see the platform as a service plays like Google App Engine and Azure, and you see a lot of the traditional hosting providers also becoming cloud providers to some degree, building on top of a virtualization infrastructure.

You can put a $2,000, or $5,000, or $10,000 project out on Elance or something like that, or build it yourself if you’ve got programming capability. You can be a company of five people, 10 people, 20 people who relatively quickly layer on top of another infrastructure like Facebook or something that has millions of users. Your user base can grow at a fairly steep exponential curve, having millions of users and enormous amounts of data.

There is also a huge amount of open source data, or data that you can subscribe to. There is a website that’s built on Google App Engine called Walk Score that you put in an address and it tell you how walkable a neighborhood is. It taps into all kinds of different streams of data that they don’t own.

It’s not their data, and they didn’t have to collect it, but it’s syndicated and accessible data. Whether that’s baseball stats, map and geo-location data, or the data that you might need to use business intelligence on, it increasingly might not even be your data. You might be looking for correlations or you might be relating data in novel ways that nobody else has bothered to put together.

What do you see as the future of these trends?

Brian: Some of the changes in the last few years have been profound, and the great amplifier of this phenomenon, going forward, is of course the cloud. If we fast forward five or seven years, it will look ridiculous that we built all these data centers. Everybody having their own data center will seem as absurd as everybody having their own power plant.

The ability to find tremendous efficiency, and deliver computing power on demand, instant on, as it’s needed, is one of the most important things happening right now. So many technologies that are transfromational, we tend to overestimate their impact in the near-term and underestimate their impact in the long-term.

All that is why Jaspersoft has been working in the cloud for more than a year. It’s not because there is yet a huge demand for business intelligence in the cloud. It’s because I know fundamentally that we have to be there so we can learn, adapt, and ensure that our BI platform can be delivered as a service through the cloud even more easily than it can be downloaded today.

My favorite example of all this is the U.S. federal government, which manages slightly more than 1,000 data centers today. If you’re a U.S. taxpayer, your money is going in a big way to help continue the running and management of more than 1,000 data centers.

Let’s just say, for fun, they could cut that number in half, which should be, by the way, an incredibly modest goal. It should actually come down to about a couple of dozen data centers. But let’s just say they could cut it in half by virtualizing and taking into the cloud a substantial percentage of the infrastructure platforms (hardware and software) that is common, or should be common, across every government agency.

Fortunately, that is a goal that the new federal government CIO and CTO are working on right now. It’s a high priority on their agenda, and I don’t think it can happen fast enough. I’m hoping that they do it in a way that is highly successful, and I’m hoping it results in a 90 percent reduction in those data centers in the course of the next decade or so.

You could use that same example across much of the private-sector landscape. Private clouds are going to have a huge impact. I think the public cloud has probably been overrated, but it will have an impact in a longer term way. The methods of building and creating private clouds is going to be one of the most important things that have come along since the invention of the personal computer and the Internet.

Scott: When I think of private clouds, I envision a company that still has a data center, but it’s just providing cloud-like services to the organization. A guy I talked to one time said, “A cloud is basically a data center with an API.”

You can run, start, stop, and clone instances, and there’s the notion of IT being more of a service provider to the organization. How does that square with the way you’re using the term “private cloud?”

Brian: Back to the government example, that’s all true, but what Vivek Kundra should be creating is a single, enormous private cloud for the U.S. government that should allow the reduction of 1,000 data centers to, say, 50. This would allow agencies to share these resources simply and capably.

And, this Government cloud should be private, so nobody unauthorized is able to tap into it. It should be a U.S. government agency cloud-based infrastructure that is shareable and provisionable for each agency, but it’s managed by some central authority, and enormous in scale.

Scott: The idea being that, even if you look at the government doing a census and tax time for the IRS and all of these peak loads, the entire capacity of the thousand data centers are nowhere near fully utilized.

So you could consolidate that all down to 50 data centers and still have lots of headroom, but without the risk that if a public cloud provider gets compromised, Los Alamos would be potentially compromised.

Brian: That’s exactly right. The present duplication of resources is remarkable, and cutting a lot of that out is where the cost savings would come from, more so than the virtualized hardware and software, and it would allow for the fungible provisioning of computer resources based upon different agencies’ needs for it.

Scott: Certain things are holding organizations back from going to the cloud. Part of the hesitation is that they just don’t trust the technology yet, but there are other factors as well. Some organizations are held back by the difficulty of extending their identity and security boundaries or issues around compliance requirements.

How does that impact Jaspersoft and business intelligence, given your stated goal of having it run as well in the cloud as it does if you download it and run it on premises? What are some of the specific things that you are tackling to make sure that you’re ready, as some of the network security trust issues get solved, and people start adopting the cloud in a more mainstream way?

Brian: We’re avoiding those use cases where those preventatives are in place, to be honest. It’s one of those things where gravity finds its own level or water runs downhill. The use cases come to us. The customers that don’t have those obstacles in their way and that have problems that can be solved with BI in the cloud are the ones that are finding us now.

The kinds of things we’re seeing come to us now in the cloud pretty commonly occur around three or four major project types. You are right that those project types do not involve industries or processes where there are major restrictions of security or data flow, or authentication issues like HIPAA-based requirements, and so on. Those are not the use cases being solved for now.

Our logic is to solve the green field problems where the marriage of cloud and BI make good sense today, and CrowdStar is a great example (one I offered earlier). Over time, the benefits of using instant-on, highly provisionable cloud-based services like BI are going to become even more efficient.

Even those industries and sectors that are highly regulated or highly concerned with security will find ways eventually to adapt and provide secure provisioning and whatever’s necessary to make those services available for these more difficult use cases.

If we were to target these security-minded use cases now, we would be foolish. Instead, we’re going to target and work with those entities, companies, and sectors that don’t have those restrictions or problems, solving mostly green field problems today. Then, over time, as the benefits emerge more clearly, we’ll surely see even the laggard use by those industries that are highly regulated.

Scott: As everybody works on that problem and they figure out a way to do HIPAA in the cloud, what are some of the things that Jaspersoft needs to do, or is doing and needs to continue to invest in to be a great BI platform in the cloud?

Brian: Truthfully, I don’t think we need to do anything differently than we’re doing today as a great BI platform. I think the issues that will need to be solved with the cloud are going to be separate from the BI layer.

Some things we’re doing in general to help us to be a better participant in highly regulated urgent care industries include things like marrying BI to processes and collaboration environments.

Those are probably the two most requested emerging trends that we’re seeing, where we have partners that are building Jaspersoft into industry-specific BPM software, to take some time out of processing and add analytics to it, so that there are huge gains in the output and the results.

One of our good partners is named HandySoft. It’s created a remarkable environment using Jaspersoft and its own BPM expertise to fully automate the process flow of secure documents that need to travel inside the Nuclear Regulatory Commission for the maintenance and management of nuclear materials all around the world.

Of course, that’s a very secure, highly authenticated environment. We’re solving the challenges there today with HandySoft for the NRC. They’re actually providing a hosting service for the NRC, very cost effectively.

I think someday that will be available generically in the cloud. We have a bunch of hurdles to cross before that can happen, but there’s no change needed to the BI software, really. What’s needed will be changes to the cloud infrastructure, the perception of the cloud, and how it behaves in order to satisfy uses beyond what I described for the NRC.

Scott: There are a couple of different cloud models, and I would imagine that infrastructure as a service is particularly interesting to you. I’m referring to the model where somebody essentially has a virtual machine, and they can put whatever they want on it in terms of an OS and a stack, and they can put Jaspersoft’s stuff on it.

On the other end of the spectrum, there are things like Google App Engine, where the developers really insulate it from even knowing that there’s a machine underneath them, an OS, a file system, and that kind of stuff. It can call out to something running on another cloud provider, but it seems like there wouldn’t be a lot of partner opportunity for a company like yours.

And then in the middle, there are things like Engine Yard, where companies are making things like Ruby on Rails or these other development frameworks into platform as a service offerings that can run on an arbitrary infrastructure as a service.

What do you see as the opportunities associated with some of these different models?

Brian: I think that many of these emerging application infrastructures will include Jaspersoft as the BI reporting and analytics layer, so that if you’re choosing from a smorgasbord of functionality in your platform as a service environment you could very well choose Jaspersoft as your reporting and analytic layer to go become part of your solution.

Our job is to ensure that the Jaspersoft platform is inside of all these environments, which is precisely why we’re doing what we’re doing with both the platform fundamentally and the cloud work that we’ve done.

Scott: In some ways, it’s almost like an OEM relationship, where you make the case to the platform or the service providers why their customers may want Jaspersoft as a BI solution they can choose from.

Brian: In fact, a little more than 60 percent of our business is in arrangements that are OEM-embedded uses of our tool, for a lot of reasons. That substantial customer base that we have who are embedding Jaspersoft, makes me not want to compete with them, but to help them succeed by attracting more customers.

I don’t want to, for instance, host Jaspersoft and try to compete with the very customers I’m selling to; that is, our OEM customers. I would much rather allow them to do it in a way that is probably application- or domain-specific, and which adds their expertise to our BI platform – and includes our expertise, which is all about building a great BI platform.

Scott: This has been a great interview. We’re running short of time, so are there any closing thoughts you’d like to add?

Brian: The kind of innovation that we’ve represented in the last few years and the substantial community that we’ve built around our open source heritage is about to take some important leaps forward.

We have two major releases planned this calendar year, and we have significant announcements with our community that will show just how big and broad it is. I think you’ll see a variety of projects that use Jaspersoft as the open source engine across stand alone and embedded implementations that will prove we really are the alternative now to the aged, proprietary vendors of the past.

Our ability to innovate rapidly and steadily, and our ability to work with the community is going to continue to be a substantial differentiator for us. It’s going to be a lot of fun, and a year from now, our progress is going to be shocking. There are plenty of surprises ahead over the course of the next 12 months.

Scott: Thanks for taking the time to talk today.

Brian: Scott, thank you so much. This has been great.

Bookmark this:

• The development of the Zend Framework and Zend the company
• On the rise in popularity and the lasting success of PHP
• The history and future of PHP adoption and innovation
• The evolution of the relationship between PHP and Microsoft
• The future of cloud computing as it intersects with PHP

Scott Swigart: First, could you please introduce yourself, and tell us a little about Zend Framework?

Zeev Suraski: Sure. I’m co-founder and CTO for Zend Technologies, one of the principal contributors to PHP. I’m also responsible for the Zend Framework project, and I lead the R&D organization at Zend.

PHP is obviously at the base of our technology stack. My colleague Andi Gutmans and I became involved with PHP as students back in 1997. We actually used PHP/FI v2 two beta, which was written by Rasmus Lerdorf.

We ended up rewriting it from scratch, and that was the beginning of the PHP project in a form similar to what it is today: a community project with a lot of different contributors at many different levels. We’ve been involved ever since.

I spearheaded the development of PHP 4, and Andi spearheaded the development of PHP 5. And five is where we are today, with six on the horizon but not quite baked yet.

Zend Framework is an open source project that we started in 2005 with a mandate to provide infrastructure classes for PHP applications. Zend Framework was definitely not the first framework to exist for PHP, but in a relatively short amount of time it became one of the most popular, and it’s probably the most popular in commercial environments.

Scott: What are some of the factors that have helped make Zend Framework popular?

Zeev: Developers appreciate the fact that Zend Framework is designed to work well with other frameworks so they can use it for the parts of a project that they want to, and they are free to use other frameworks or custom code for other parts of that same project.

It’s an easy choice to make, because a developer can get started with Zend Framework and then use it more extensively as he or she sees fit.

Interestingly, just last week, there was a conference in France called Symfony Live, which is the yearly conference of Symfony, another very popular PHP framework. They announced that, in the next major version of Symfony, they’re going to use parts of Zend Framework as the basis, instead of reinventing the wheel again.

I think, in a nutshell, that gives a pretty good indication of how far Zend Framework has come in terms of becoming somewhat of a standard for building PHP applications, so that even other frameworks sometimes decide to rely on it.

Scott: How does Zend the company fit into the equation?

Zeev: Andi and I founded the company in 1999, with a broad mandate to provide solutions around PHP. That’s kind of the mile-high view of the company.

At ground level, there are several interesting facets of business. First, we are very deeply involved in the open source projects that I mentioned. That includes PHP itself, for which we’re still major contributors, and we’re probably the biggest contributor and maintainer of the engine, called the Zend Engine.

In addition to that, there’s the Zend Framework, a community project for which we are leading the way and contributing alongside countless other contributors outside the company.

There are some other open source projects we’re involved with, like PDT, the Eclipse PHP Development Tools project that we’re leading, and a couple of others, but those are the major ones.

Zend also provides commercial software on top of the open source software. For instance, for development, we offer Zend Studio, which is an advanced version of PDT with a broader set of features.

On the server side, we offer Zend Server, which provides an easy-to-use, easy-to-install version of PHP, plus quite a few additional features around performance optimization, proactive monitoring and root-cause analysis, job-queuing, scalability, and so on. We offer both a free and a commercial version of Zend Server for Windows, Linux and IBM i. The free version is not entirely open source, though, so it’s not in the open source line.

Scott: Can you tell the story of how you got involved in PHP originally?

Zeev: OK. In 1997, Andi and I were students in the Technion, a technology university in Israel. While in school, both of us were also working part-time at Internet service providers. Early on, we found that we preferred the practical aspects of our work in the ISPs over our more theoretical studies.

Obviously, as part of the curriculum, we had to develop several projects, and one of the projects we decided to take on was to develop a shopping cart application. Today, one can choose from various shopping carts available to just download and maybe hack a bit to make it one’s own; but back in 1997, this kind of work was considered cutting-edge technology.

We were the only two students who asked our instructor for permission to use PHP/FI as our language for developing the project. All of the other students asked for Perl, which was the undisputed king of the web at that time.

I was familiar with PHP/FI through my part-time job, and thought it would be easier than Perl with its headaches, so Andi and I decided to use it. It wasn’t easy to convince our instructor to let us use this thing called PHP that she’d never heard of. Eventually, she agreed, but with a warning to us that, “If you bump into issues, because this is not very standard, then it’s your problem.”

We started implementing the project, and about 30 or 45 minutes into it, we started bumping into the kinds of issues that forced us to eyeball the source code to figure out where things were going wrong.

We were looking at parse errors, syntax errors and such, from all different sides, and trying to trim and cut the code and see how it behaved. After a while, we concluded that we weren’t doing anything wrong; there were parser issues that made it reject perfectly valid code.

We sent an email to Rasmus Lerdorf, because it had become clear to us that the language didn’t understand its own basic syntax. He came back to us and said, “Look, guys, I’m not a CS major. I just hack this around. Feel free to propose patches.” We decided to give that a try. We dove into the source code, trying to understand it more deeply, and we were pretty stunned by what we saw.

On one hand it was pretty amazing that Rasmus managed to create something that worked as well as it did, given the lack of architecture. On the other hand, the experience was like opening the hood of one’s car and, instead of an engine, there were mice running to move the wheels. Very impressive if those mice could make the car drive reasonably well, but still not the kind of engine one would choose to power the car on the highway…

Since we preferred the practical elements of computer science to the theoretical ones, we decided to try to implement it from scratch, to see how far we could go. We managed to write it from scratch, and during that time, we not only changed the implementation; we also turned PHP/FI, which was sort of a partially defined pseudo language, into a much more well-defined one. By the way, Rasmus did fix the parse error we bumped into very shortly after we reported it, but the seeds were already sown.

We literally eliminated the possibility for the language not to understand its own syntax, by using the compilation tools the right way. I want to add here that our compilation techniques professor, Michael Rodeh, pushed us to touch base with Rasmus and convince him to move over to our implementation. That in itself is interesting.

Rodeh had a firm view that there are too many development languages in the world, and he didn’t want to be in any way responsible for the creation of yet another one. He hoped we’d convince Rasmus to replace his version of the language with ours, and join us in working on the new implementation.

We weren’t sure that Rasmus would do it, but he was actually quite happy to, and that’s the story of how the modern incarnation of the PHP project was born. Rasmus had been working almost solo on PHP/FI 1 and 2 between ’95 and ’97.

Scott: It’s interesting, because PHP is such a long-lived language, whereas so many other web-programming languages have sort of risen and fallen, although PHP has obviously gone through a lot of evolution, as well.

I worked as a developer for a long time, and like many developers, I often found that I just wanted to use a new programming language or a new framework on the next project, so I could learn more. And so, over the years I have explored all kinds of UNIX shell commands, scripting languages Tcl/Tk, web-development languages, low-level C++ and more.

PHP is really as much here to stay as it’s ever been. Why do you think it has been so successful and long lived?

Zeev: I don’t pretend to know the complete answer to that question, because at the end of the day, a lot of this has to do with dynamics that are too big to analyze.

But, if we put aside a strictly scientific answer, I think that the key ingredient from a technology standpoint is the focus on simplicity. Trying to really solve the web problem in the simplest possible way, and not necessarily always the most ‘elegant’ way.

I think that some people might look down on PHP, because it might not always do things in the elegant way that they expect. Another way to put it is that I think Perl prides itself on the ability to do one thing in many different ways, and to do so in a way that looks very elegant.

At the same time, some people would look at that same piece of code and while they might find it elegant, they might also find it very difficult to comprehend.

With PHP, we have avoided that and focused instead on keeping things simple. We have not tried to create a general-purpose language, but rather a web-specific language. From the beginning, the PHP community started to thrive, and we had a self-nurturing cycle.

In addition to the language itself being simple and easy to use, and very fit for web applications, developers have been very enthusiastic about it and helped others move into PHP. Those were the two key factors for success: a self-nurturing community and PHP’s simplicity. There are many factors, but honestly, they’re all that unique to PHP, such as good support for databases. We had very good support for MySQL very early on.

Probably, the reverse is also true, and MySQL may owe some of its success to PHP in much the same way. It’s difficult to say which project contributed to which other project more.

Scott: I saw something similar happen with Visual Basic, which also set out to be simple. Its initial goals were to be very approachable and learnable, helping people become better programmers and get more done.

I would imagine that PHP probably faced a little bit of tension because of its focus on simplicity and giving people an easy way to get things done. Specifically, it was relatively easy for people who didn’t have computer science degrees to pick it up and be productive with it.

I imagine that PHP really had to evolve to try to be very approachable and learnable by amateur and hobbyist developers, while also meeting the needs of some of the best developers. Is that where something like the Zend Framework comes in? Talk a little bit about how PHP has addressed that tension over the years.

Zeev: PHP 3 became very popular very quickly. We designed it as a powerful language, but with simple, small apps in mind. In practice, though, more and more people were using it for fairly complex applications with growing needs for performance. It quickly became apparent that Andi and I weren’t two geniuses that got it right the first time around.

Almost immediately after PHP 3 was out, Andi and I realized that we could have done things differently. Developers were beginning to use PHP for really complex and demanding scenarios. Basically, the language was playing catch up between functionality and performance. It was capable, but slow, so with PHP 4, we made a lot of progress to solve that problem.

In PHP 5, which was the next step, we really re-invented and re-implemented the whole object-oriented model, almost from scratch. That was perhaps the biggest downwards-compatibility breakage that we’ve ever made in PHP. PHP 5 brought the language to new realms. Beforehand, it was largely ignored by the architects of the world; after the release of PHP 5, some of the mega-vendors, including IBM, Oracle and Microsoft, took notice and have become very much involved with PHP.

The next step in the evolution was the frameworks, specifically Zend Framework, where we went one level above the model and introduced a classier key that provides the building blocks and design-pattern implementations to help developers get productive quickly and create very maintainable applications.

Every step of the way, we kept existing users and existing types of code in the front of our minds. Even though it’s becoming less popular, one can still write a PHP procedural application today. My gut-feeling estimate is that it still accounts for at least half of the lines of PHP code in use.

People often ask me whether I envision the procedural support of PHP ever being deprecated in favor of a purer object-oriented code, and the answer is no. I don’t expect it to ever happen, and not only because I think we would be breaking God knows how many billions of lines of code, but also because I think that for many developers, that’s actually quite a nice way to start in scripting.

Scott: Yeah, exactly.

Zeev: I think we will generally move at the framework level as the place where innovation happens. The language has all the essential building blocks needed to provide frameworks to do most of the innovations.

We will continue to introduce major new language features, as we did with PHP 5, yet that is clearly not where most of the innovation happens today. I think taking a look at frameworks, and maybe extensions that connect PHP to all sorts of data sources and portals, is where the vast majority of innovation is going to happen going forward.

Scott: If you feel that a lot of the innovation is happening outside the core, that is a good thing in a lot of ways. It provides reassurance to the market place so people know they can write code on it and it’s likely to continue to run into the future.

I think you’re exactly right about keeping procedural support. You end up with different factions, and there are always people who will say that if somebody can’t do object-oriented programming, then they have no business writing code.

On the other hand, you’ve got the entrepreneur in a small business who just wants to modify the company web site, and is able to get it done without hiring an expensive consultant. I think that if some people want to do everything with object-oriented code, that’s great. But there is a legitimate community out there using it as a procedural scripting language that makes a lot of sense to them, and enabling them to get their job done is important too.

I guess another testament to the success of PHP is as the “P” in the LAMP stack, even though it’s not just a Linux technology. Probably a lot of PHP code ran on UNIX back in the day and still does.

Microsoft has obviously also taken a pretty strong interest in it, doing real engineering work in Windows 7 to make PHP run better and supporting PHP up on Windows Azure. Are you involved in any of that? And what do you think of PHP on platforms that aren’t Linux?

Zeev: We are very involved with bringing PHP to non-Linux systems. Initially, we developed PHP on Linux, but then moved to Windows when we started working on PHP 4. I think it was Windows 2000 and Microsoft Visual C++ 6. That was in 2000, or maybe 1999, so the relationship between PHP and Windows started a long time ago.

It really materialized relatively recently, in the last three years or so. In 2001, Andi and I were invited to Redmond by Microsoft to see if we could significantly improve the performance and reliability of PHP on Windows.

We were invited by a team called the “Dotcom SWAT Team,” which was formed to convince Dotcoms using Linux to move to Windows Server. They were bumping into a lot of people who were very open to Windows Server but wanted also to use PHP.

We stayed there one week and made some progress, but the next step actually took a few years to complete. I think real progress started when Microsoft and some other big organizations felt that it was really important to support PHP, and to make PHP run well under Windows.

Microsoft has changed in the way it perceives its involvement with open source projects, and has started putting lots of resources into making open source projects run well on Windows.

Scott: How did your involvement with Microsoft evolve after that initial contact?

Zeev: We at Zend worked together with Microsoft in 2006 and 2007 to really improve the stability and performance of PHP on Windows. We introduced the use of an architecture that wasn’t new, but it wasn’t very common, called FastCGI to deploy PHP on Windows.

We changed the PHP build and the way it’s actually running on Windows to run as a single server application with multiple processes, as opposed to as a single process with multiple effects, which is more similar to what you find on Linux. It goes much faster, and more importantly, it has reached production quality in terms of reliability. We have been working with Microsoft ever since, and today Windows is a viable deployment platform for PHP apps.

The Zend product line evolved from a situation where Linux was perhaps our main supporter operating system–along with Solaris and a couple of others–so that today both Windows and Linux, as well as Mac OS X on the development side, are first-class citizens.

We’ve seen quite a lot of uptake of PHP on Windows, and we are very happy about it. Zend has always been very pragmatic, and we wanted to see PHP running on as many systems as possible.

Scott: PHP is a higher-level language, as opposed to something like C, so it insulates the developer in a lot of ways from OS-specific APIs and that kind of stuff. I mean, you don’t have to know a lot about POSIX-compliant C libraries to be a PHP developer.

Another factor is that companies are now starting to look at what percentage of their applications make sense to run in the cloud. That’s kind of an extension to thinking that they want to partner with a hoster, rather than necessarily wanting to run every application in their own data centers, so the hoster can manage and maintain the system, guarantee uptime, and that sort of thing.

To me, cloud is an extension of that, but with some important differences. For instance, cloud basically gives you an API that lets you remotely stop and start instances. Or if it’s more of a Platform-as-a-Service like Google or Azure, it does scaling silently and automatically.

As you consider cloud providers, at one end there is Google App Engine, and it seems like the fact that they didn’t support PHP was an enormous hole, since a lot of things like Drupal, Jumbla, and WordPress are written in PHP.

For a lot of people, that’s the starting point, and obviously with Amazon Web Services, PHP runs just fine, because it is kind of an Infrastructure-as-a-Service that can run whatever they want. What are the opportunities and challenges around cloud development and PHP’s place in the cloud?

Zeev: As you said, the basic stuff already works, because in the most naive way, one can look at cloud as just another way to host applications. Obviously, that works pretty well already, except for Google App Engine, which doesn’t support PHP.

If we look a little deeper and try to see the opportunity here, I think we have done some interesting work around cloud to make PHP even more attractive for developing web applications. We started an initiative called the Simple Cloud API several months ago, partnering with IBM, Microsoft, and several other cloud vendors, with the goal of creating an exceptional layer on top of cloud services.

I’m not talking about the Infrastructure-as-a-Service APIs, but about accessing the cloud services that developers typically have access to for an application that runs inside the cloud, like knowledge tables and things like that. That goal is to create an accession layer on top of those services and what developers could look at as some sort of ODBC for cloud services.

We are half-way to implementing it, and taking a look at simplecloudAPI.org will show where we are in the process. When we finish and push it out, it’s probably going to be part of the Zend Framework. The implementations will be open, and developers will be able to implement it in other frameworks, as well as in other languages.

I think that introducing something like this into PHP, and the Zend Framework, is going to be a first, and it may help spur companies and ISVs not only to use the cloud but actually to develop for the cloud.

It allows developers to create applications in a slightly different way, if they need to create very scalable applications and things like big tables are very, very attractive probably more so than relational databases.

When ISVs and other companies start developing for the cloud and not just deploying on the cloud, I think that PHP, together with the Simple Cloud API, are going to be very effective choices.

Other than that, I think that the more obvious approaches are also going to happen. For example, Zend software could be provided as Amazon or Azure instances that developers would be able to deploy quickly, instead of having to provision a server and install Zend Server or PHP on it. They would be able to just go to the cloud, start up an instance, and boot an application there. Some of this functionality is already available.

We’re also thinking about how to bring forth innovation and change the way people develop applications once they have access to cloud services. And that is, perhaps, one not-so-obvious direction that we’re taking.

Scott: With the Simple Cloud API, is the idea partly to abstract the fact that different cloud providers have different ways of providing, for example, a table-driven distributed data store that’s non-relational? That abstraction layer would make it so programmers wouldn’t really have to code to the differences. They would be kind of coding to a common API.

Is the goal to make Zend PHP code more portable, or is it just to simplify development?

Zeev: It’s actually both. Look at the cloud APIs today. Simplicity is not the number one goal, and sometimes vendors may indeed want to differentiate.

So, our goal is really twofold: both to abstract and to make applications portable. I think both are important before using those services can become more mainstream. Having abstraction, so developers can take their applications easily from one cloud vendor to another, is an important step.

Equally important, making the use of these services easier is definitely also a goal of the Simple Cloud API, hence the name “Simple” in the name.

Scott: Right now, there’s a lot of interest in the cloud as a paradigm shift. It can be used for simple hosting, but it’s a lot more interesting when used around applications that really require a lot of elasticity.

People talk about the fact that there aren’t a lot of standards out there, and different cloud providers do things a different way, whether it’s a VMware vCloud partner, Amazon, Azure, or Google App Engine.

At the language level, or at the library level, it seems like a lot of people are tackling the problem of how to standardize and achieve portability. The angle Zend takes is interesting, because it sounds like a PHP developer wouldn’t really have to learn a lot of non-PHP technologies to get the portability they’re looking for.

Zeev: Most of the standardization efforts that I’ve come across around the cloud have to do with standardizing the Infrastructure-as-a-Service part of it: managing instances, starting and stopping instances, and so on. I haven’t come across a standardization effort for cloud services per se, regarding those services that one gains access to once inside the cloud.

That’s where we decided to focus, and it makes sense for a library to tackle that, for the framework.

Scott: This has been great, but I want to be sensitive to the time. Any closing thoughts from your end?

Zeev: On a personal note, it has been and still is an amazing ride. As you can understand from the origins of how our involvement began, neither Andi nor I had any clue whatsoever that PHP would be as popular as it is today. We would never have expected to see the White House website, Facebook, or Wikipedia standing on top of the technology that we helped get its start.

I’m confident that PHP has a bright future. Its popularity has never been higher, and it’s growing all the time, moving out into the mainstream. Many Java developers are now looking into scripting with PHP as a better way to tackle web applications.

Overall, I think we already are in a good position, and the road ahead looks very positive.

Scott: Thank you very much for taking the time to chat.

Zeev: You bet. Thank you.

Bookmark this:

• Re-inventing the field of business-process automation
• Applying business-process automation principles to software testing
• Creating software-testing routines that are resilient when application code changes
• The nature of innovation in proprietary and open source approaches
• Positive and negative impacts of a down economy on software transitions

Scott Swigart: To start, please take a moment to introduce yourself, your company, and its key products.

Mihir Shukla: I’m the CEO of Automation Anywhere, which is a software automation company. Our flagship product, Automation Anywhere, has been a leader in the space of business and IT process automation. We started about six years ago, with the goal of shaking up the market.

Before we entered the market, products for business and IT process automation were in the cost range of $1 million to $2 million. Even though just about every company needs business and IT process automation, the entry point was too high for most businesses to afford it. Our vision was that we wanted to do business and IT process automation 10 times faster, for one tenth of the cost.

To put that in real-world terms, a project that used to require $1 million should be able to happen with a budget of $100,000. And if it took 10 months to do a business and IT process automation project, it should happen in about a month. That was a pretty lofty goal, so we had to innovate in many areas and change the rules of the game. Today, we have more than 25,000 customers in more than 90 countries.

By shaking up the market in business and IT process automation, we found that thousands of users are using it for test automation purposes. Last year, we decided to more formally enter into that market, and we launched the product Testing Anywhere.

Scott: For a bit of level-setting, how would you define business process automation and IT automation, just to provide a sense of kind of exactly what those two things look like?

Mihir: Every business has many, many business processes, which could include generating reports, or entering patient data into a health care system if you’re a hospital. If you’re a retail company, your business processes would include checking inventory and reordering whatever is in short supply. If you are a dentist, a relevant business process could be to remind people of their upcoming appointments.

And to give an example of an IT process, you may want to install a new operating system patch on various computers. You may want to monitor your data center and find out which hardware is performing below expected levels. You may want to restart various servers routinely.

Those are some examples. We have learned that over 25,000 customers, there are 20,000 unique business processes that businesses rely on.

Scott: And then on the testing automation side, is that automated QA and software testing?

Mihir: Right. That is a very specific example of a process that needs to be automated, but yes, test automation is part of automating a QA process.

Scott: You mentioned that your goal when you set out was to complete implementations for a tenth of the cost, in a tenth of the time. What makes your business process, IT automation, and test automation different?

These are definitely spaces where there’ve been a lot of products in the marketplace for a long time, with really large vendors involved with patch management, systems management, and so on. Talk a little bit about your approach and what makes it different and less expensive and more productive.

Mihir: We decided that if we are to improve it by a factor of ten, we could no longer do what has been done before, or we would be limited to perhaps a 10 or 20 percent improvement. We looked at the space and where the costs involved come from.

We found that the dominant approach to business and IT process automation was very programming centric and involved a lot of consulting requirements. There was not a lot of focus on putting intelligence in the system itself.

Let me give you an example. In business and IT process automation, we created a drag and drop environment and intelligent automation that automatically figures out what it is that you intend to do.

So, for example, consider a business process where I log into my ERP system, type in my dates, generate a report, and email that report to three executives. When I perform that business process, behind the scenes, our software would automatically create the entire automated business process.

It is done in a very intelligent fashion, so it figures out the intent of what you want to do, rather than focusing on your actions themselves. For instance, if you double click on your desktop, we know that you intend to start an application, so we focus on that, rather than the actual fact that you double-clicked.

We developed some technologies that were groundbreaking in terms of how we integrate with various systems for automation. Instead of creating specific adaptors for various systems, we came out with an on-demand integration adaptor for any Windows application. So, suddenly you could integrate with any of the 20 million applications out there.

We also implemented a visualization technology, so the entire automation became visual. At every step of the way, you have a visual picture of what part you are automating.

Scott: How are you extending that to the test automation space?

Mihir: We intend to bring the same spirit of innovation that we did with Automation Anywhere.

Our viewpoint toward the test automation market is that in the last 10 years or so, there hasn’t been any true innovation. Many of the tools out there, even if they have more bells and whistles, more or less work the same way they used to. If anything, it takes longer to learn those tools, and even longer to automate some of the new technologies.

In the test automation space, we brought forward quite a few new innovations. For example, one of the problems we saw in the test automation space is the problem of test portability. You may have a test case that works in your environment, but not elsewhere, such as in a production environment, a partner’s site, or a customer’s site. In cases like that, I cannot take my test and run it on another environment, basically.

We came out with a very intelligent solution, where you can export a test into a standalone executable that has an intelligent Testing Anywhere agent inside it.

Consider a very simple example of a communication between a QA person and a developer. If a QA person finds a problem, they could report it to the developer, and the developer could come back and say, “I cannot reproduce your issue.”

In a situation like that, the conversation could go on back and forth for a while, in a very unproductive fashion.

Scott: That’s certainly a familiar experience to a lot of people in the software industry. How does your approach change that dynamic?

Mihir: With Testing Anywhere, the QA person could take the failing test case, export it as an executable, and attach it along with the bug report to the developer. The developer could run this test case, standalone, without installing Testing Anywhere, and instantly be able to reproduce the problem.

This innovation alone has opened up many possibilities to our customers. We see customers who are using the test to exe capability to do compatibility testing, for example, where you have a test case on one version of the operating system and you want to try it out on a different version to see if the same test case could work.

One of the areas where it’s used significantly is in Agile methodology, where some of our customers have two-week release cycles. That gives you about a week’s time for a feedback loop between a developer and QA to validate the feature, communicate it back to the developer, reproduce it, get it fixed, revalidate it again, and make a release.

They use the test to exe capability for communication between developers and QA. Many of our customers are using it to reproduce problems from a customer’s site or a partner’s site. Problems could be because of user error, or they could be genuine bugs based on certain values.

Issues that were taking an enormous amount of time are being solved now at a fraction of the cost and time.

One of our customers summarized using the test to exe capability well. He said, “15 years ago, when there weren’t mobile phones, life was just fine, but today, if I forget my mobile phone at home, I’ll miss it 15 times during the day.” Once you have it, you cannot imagine going back.

Scott: When I think about the test tools that I’ve used over the last decade or so in software development, many of them seem brittle. Even when you make relatively small changes to an application, you may have to recreate all of the tests, because a button isn’t at the same pixel location, or the output that shows up on the screen isn’t exactly the same, and so it doesn’t key off of it quite right.

Talk a little bit about that notion of making tests more resilient, or the automation more resilient so that the amount of time that you have to go back and recreate them is minimized.

Mihir: The argument for test automation is strongest in Agile methodology, where if you’re going to run through all the tests every two weeks, you’d better automate. The first issue there is that, as you very correctly pointed out, if you change an application, you shouldn’t automatically have to go back and change things. In today’s day and age, you don’t have months and months of product cycles to do that.

Another issue is that even if you have to change it, it shouldn’t take you too long to change it. We know of examples where it takes you 10 minutes to make those changes with Testing Anywhere, and it takes you two days to do it with some of the other software. The difference can be enormous.

The third point is that you want a process that is collaborative, especially in Agile method. Some of the older testing tools are designed for waterfall models or others where you have a year or two to develop and maybe six to eight months to do testing. In that methodology, you are OK, because you’re going through a systematic process. Well, in Agile also there’s a systematic process, but it’s a lot more collaborative.

With Testing Anywhere, collaboration is built in. Whether it’s two QA guys, a QA guy and a developer, a QA guy and a product manager, or a QA guy and outward-facing business users, you can easily collaborate on various aspects.

Scott: What about documentation? How does that fit into the use of Testing Anywhere?

Mihir: You don’t want to make Agile into a cowboy methodology, right? Testing Anywhere creates documentation on the fly that is lightweight but effective, and that helps us create a lot of traction in the market.

I’d like to come back to the issue of what you can do so that if things change you don’t have to come back to it to change your tests again. One of the things that Testing Anywhere does is that, as you create any test case, it doesn’t leave it to the user to figure out and implement what I call the plumbing logic.

The product is intelligent enough to figure out that this is what you intend to do, and it will create an automatic test routine by default. For example, if you double click on a desktop, it automatically figures out that you intend to start an application, and internally, it will just launch that application.

If you take that test case to any other machine, it will continue to work. Whether it’s a Windows application or a web application, it automatically figures out all the controls and it figures out what is the optimal way of matching those controls. It allows you the flexibility to change anything if you like, and the intelligence comes automatically.

Scott: In other words, if you’re double-clicking an application on the desktop, and on a different machine, that icon is in a different location, that doesn’t matter, because the test has figured out the intent. As long as the application is installed, the test is still going to launch the app, even if something’s slightly different from one environment to another.

Mihir: Absolutely. Testing Anywhere does about 100 things like that intelligently and automatically, so when you use the tool, the difference is quite obvious. Relative to using some of the older tools, you would go about 200 percent to 300 percent faster in creating, developing, editing, and managing your test cases. The effect is very measurable.

Scott: To shift the conversation over a bit to the business side, just today, Red Hat said on a community open source site that, with the exception of Apple, all the innovation that’s happening today is in open source.

In reality, though, we come across a lot of proprietary software companies that are doing very interesting things. Talk a little bit about your decision to build intellectual property internally and not develop this as an open source product.

Mihir: As a company, we are founded on the principle of innovation. We believe that innovation is not a property of any particular model, but a result of what you believe in. We’ve seen innovation across all kind of models.

Our goal was to provide a very reliable but at the same time breakthrough technology that has never been seen before, and based on our experience in the industry for the last 15 or 20 years using many tools and technologies, we knew we had ideas that would shake the market up.

It made sense for us to come out with these technologies using a very fast-paced model. We internally use Agile methodology ourselves, and we incorporate many of these new breakthrough features. This made more sense for us than open source.

We also have a Team Anywhere program, which is a program where we invite an elite group of our customers to influence our product direction and provide feedback. In a sense, that creates a collaborative effort that has some things in common with open source.

The collective experience of our 25,000 customers is enormous. These are the people who live the day-to-day life of business process automation or test automation, and they have become a driving force for our innovation as well.

Scott: In other words, whether your software’s being built by the community or the community’s just providing feedback, collecting that input from the actual users of the product is really important. I would imagine, too, that you’ve probably got some stories about people starting to use your products in ways that you never really imagined.

Mihir: Absolutely.

Scott: Talk a little bit about some of those cases.

Mihir: We believe in innovation with the customer, and we have about six or seven different channels through which customers can talk to us. If a customer wants to tell us of a problem, something good, something that could be improved, or something that’s awesome, we always want to hear from them.

Within the product, we have various ways customers can provide us feedback. On our website, we offer a free service where we have a team of experts who can provide customers with assistance. We have learned so much from our users, across the board, in business and IT automation as well as test automation.

Let’s take an example. When we created the test to exe feature, we understood that test portability could open up many avenues, and we had ideas about a few. We saw enormous breadth in the ways customers incorporated it into their communication between QA and development.

At the site of one of the customers we visited recently, when a bug is identified, it’s back with the developer within about five minutes, and in the next five or ten minutes after that, the developer has reproduced it. That was amazing, and it’s just one example.

Scott: When you launched the company six years ago, the global economy was growing rather nicely. A year and a half or two years ago, the global economy pointed its nose at the ground and started to descend pretty rapidly.

What’s your advice to somebody who’s thinking about being a software entrepreneur, in terms of navigating the inevitable economic ups and downs?

Mihir: I think one silver bullet, even if it’s not necessarily a novel thought, is that if you are very close to your customers, you can navigate through any economic ups and downs. Trouble happens when you are too far from the customer and you cannot adjust fast enough to the changing times.

For us, 2009 was pretty good; we had about 50 percent growth. We faced issues just like most people did, but you want to stay close to the customer, understand their pain points, listen to them, and use that as your inspiration for innovation.

When you do that, success is inevitable. When we launched the product Automation Anywhere, I wanted to be the first one to buy the software, just as a matter of pride. The whole company got together, we launched the product, took a coffee break, and gathered around a table. A customer in Australia beat us to it. That’s the story.

Scott: You were doing something in the marketplace that you planned from the beginning was going to be disruptive. You were going not for incremental improvement, but really for an order of magnitude improvement.

Maybe during good times, a lot of companies might have said, “Well, we have testing tools and automation tools. We know them, we’ve bought them, they’re expensive, and it’s not necessarily strategic for us to look at replacing or augmenting those.”

In times of crisis, companies go out and really look at how maybe they could do things differently, and they’re motivated to go look for savings under every rock. Did you find that the cost and time savings of your product positioned you well for the downturn in terms of the interest customers may have shown?

Mihir: You’re absolutely right; disruptive technologies have a huge upside in tough economic times, across all segments, including small businesses, medium businesses, and especially Fortune 500 customers.

Typically Fortune 500 customers have more budget at their disposal. Of the top Fortune 100 customers, 70-plus are our customers. Many of them just came to us and said, “We heard you do things differently.”

We heard that in tough economic times, it was hard to sell the old test automation tools, not just because they’re expensive, but because of the high cost of ownership and how long they take to ramp up, learn, modify, and manage.

It was taking so long that revenues of these companies were affected. I was very surprised at how open they were to accepting a new, disruptive tool in this space.

Scott: We’re not very far into 2010, but I know that some software and IT companies are starting to feel like things are improving. As you look forward to 2010 and beyond, how are you feeling about growth potential?

Mihir: 2010 definitely looks much better than 2009, that’s for sure. We see a significant improvement in the market, and a lot more budgets have opened up. We see a significant movement in both private and government sectors.

Scott: This has been great, and we’ve covered a lot of ground. Do you have any closing thoughts?

Mihir: In my experience with SIs, I found out that, of the total test automation market, about eight percent is currently in automation. They see a significant market potential for that to go up to about 30 or 35 percent, and everyone feels this stagnancy in the tool space in terms of the tools that are out there.

For this new market, they’ll look for new tools that can offer value, and we plan to fit in that space.

Scott: Thanks for taking the time to talk today.

Mihir: Thank you.

Bookmark this:

• Deciding on the open source approach to development and choosing a license
• Simplifying the creation of multi-tiered applications for the cloud
• Fitting WaveMaker into the broader open source and proprietary ecosystem
• Engaging with the community and deciding what goes into the paid edition
• On the importance of community managers, and how to choose a good one

Sean Campbell: To start, why don’t you take a minute and introduce yourself and your company?

Chris Keene: I’m the CEO of WaveMaker. Our company makes an open source cloud development platform, which you can think of as being like force.com without the proprietary lock in.

We introduced WaveMaker about two years ago as an open source Apache project, and there’s also a commercial version of that same product that has additional features, around security primarily.

At the end of last year, we moved that development platform into the cloud and really started to realize the full vision, similar to PowerBuilder or Microsoft Access for the cloud.

Sean: Tell us a little bit about the open source project and the choice to develop using community involvement.

Chris: I should say first that my last company, Persistence Software, was a proprietary company, and we took that public and then sold it, so I kind of know the drill on the proprietary side. When I came to the open source world, I found that everything I thought I knew was wrong. And so the main way we’ve figured out how to do things in open source has been by doing them wrong first and then fixing our mistakes as we went along.

I think the upside is that if you’re willing to fix your mistakes, open source has an incredible adoption curve. In a little over a year, we gotten more than 15,000 registered developers in our community. For comparison, it took the SpringSource community over six years to get roughly three times as many developers.

We’ve had hundreds of thousands of downloads, but what we really care about is how many people have downloaded, installed, come back to sign up for our community, and then started to participate. To build that kind of a user base in a little over a year is just incredible.

Still, along the way, we made a lot of mistakes. We started out, for example, with an AGPL license, and we found that the open source community didn’t really trust that license for software that was going to be embedded into their own applications, which caused us to switch over to the Apache license.

We’ve ended up tweaking and changing almost every element of our business model, because we’re constantly getting feedback from our community about what they’d like to see and what’s most valuable for them.

Sean: With the Apache license, people can embed it in proprietary software. So the GPL license makes things compatible with bundling with a Linux distro, for example, but the Apache license makes it possible to carry open source in a proprietary product without having to make the whole thing open source. Is that the motivation?

Chris: That’s exactly right. Almost by definition, anything that somebody builds with a cloud development platform is going to embed, at a minimum, your libraries. Because of that, people didn’t feel comfortable under the AGPL license.

Part of the reason for the difference in the community’s reaction is that AGPL is fairly new, whereas Apache has been around for a while, and people really understand and trust it. I think if you are an open source company, you need to be willing to follow the lead of your own community; otherwise your community is basically just a marketing exercise.

Sean: That makes sense, and in general, open source communities vote with their feet fairly rapidly. Generally, they have a certain number of choices in any given space, and it’s instructive to consider, for example, that MySQL and Apache took off even though they obviously weren’t the only options available.

It seems like there is a sense of getting things just right in order to have one project versus another one become a center of gravity that attracts a lot of people.

Chris: I think that’s true, even if a feature-by-feature comparison might suggest that it’s a toss-up. One thing I will say for MySQL, though, is that they absolutely got the install right, and it’s still much easier, in my opinion, to install MySQL than many other of the open source databases.

I think that ease of use and approachability really help drive rapid adoption. That’s a key characteristic of MySQL, Ruby on Rails, and frankly of WaveMaker as well. We think that it really helps drive success when people don’t have to invest months of their lives to figure out whether your stuff works or not.

Sean: Before the cloud, people were using hosting providers to build websites. Hosting providers started to get into virtual private servers, so instead of having a dedicated server that was co-located, they could spin up a virtual instance. Amazon took that to a different level, with utility pricing and letting you spin up hundreds of servers with a command line or an API call.

How does WaveMaker fit into that spectrum, from the dedicated, co-located server to the virtual private server with Rackspace or whoever, to running in a cloud where you’ve got options for real elasticity?

Chris: The cloud itself is just a concept from my perspective; it’s really just a data center with an API. If I’ve got an application and I need to figure out where to put it, the question is, “What steps do I need to go through to get my application running someplace where people can get at it with their browsers?”

Of course, in the co-lo world and the hosting world, I have to call people, I have to put contracts in place, I have to pick up the phone and send emails to make something happen. In the cloud space, I just make a set of API calls.

Not only has the cost and complexity of making something available on the Web gone down dramatically, but I can now start to imagine businesses that previously wouldn’t have tried to put things up on the Web wanting to do so.

Cloud computing really lowers the bar for deployment, in terms of both systems administration and economics. But cloud computing on its own doesn’t really do anything to lower the difficulty of creating that cloud application in the first place. That’s really where WaveMaker comes in.

If cloud computing is all about democratization, utility computing, and allowing more people to put applications on the Web, then we also need a new set of development platforms to facilitate that. In fact, we’d argue the need not just for development platforms but actually for application templates that people can take down and customize for their own purposes.

That set of templates has an intent very similar to what SharePoint templates have provided: a simple means of managing projects, time, assets, and issues to allow a whole new class of developers to build and deploy web-based applications.

Sean: A lot of open source development, on core products, involves very low-level C programming, or those kinds of things. Then there’s another level where people are doing Java, .NET, or PHP programming.

Those who are using a higher level language might be using open source libraries, test and development tools, or development environments.

How receptive have you found the open source community to be to the notion of 4GL languages in general? Also, do the people who are actually building stuff with WaveMaker even really know or care that it’s open source, or to them is it just a really productive way to get stuff built?

Chris: I think WaveMaker has pushed the envelope in terms of the type of developers we’re trying to reach out to. The typical stereotype open source developer is an alpha geek who has never gotten past EMACS as their development environment and eats other coders for breakfast.

The reality is that there are very few people, even very sophisticated developers, who have the full range of skills needed to build a web application. For example, a person might need to build a Java application, put an Ajax front end on it, build a database and database schema, and then deploy that into some sort of a hosting environment.

We’ve found that the target user base for WaveMaker consists of developers who may be very skilled but don’t have full teams where they can afford to put four to six developers with different skill sets on a project to get it through.

We tend to work with very sophisticated developers, but oftentimes they’re in the business units. Gartner calls these people “citizen developers” or “non-expert” developers, but it’s really the notion of people trying to get a lot done on their own.

Frankly, I think that’s the same crew that Ruby on Rails has appealed to. One of the interesting things for WaveMaker is that our biggest core set of customers is Macintosh developers, because they really like visual development. They tend to be very sophisticated, and they’re incredibly pragmatic; they just want to get the job done, and they want the best and most productive tool for getting that to happen.

Sean: My understanding is that WaveMaker is that it’s really optimized for the “forms over data” types of applications. In other words, when NBC is trying to figure out how to build its site for streaming the Olympics, WaveMaker is maybe not the best choice. On the other hand, if somebody is building an expense-reporting application, or maybe even a whole CRM system from scratch, that’s more the niche that WaveMaker fits in. Does that ring true?

Chris: Absolutely. WaveMaker really fits into the classic category of enterprise solutions that help business developers build basic business applications. We’re not trying to focus on the multimedia side of the Web, nor are we trying to focus on the complex transaction processing part of core IT.

We’re really about the broad middle: the 80 percent of applications in the business that are building similar functionality over and over again but customizing it slightly, which is where we think that combining a development platform with application templates is an incredibly powerful way to drive productivity broadly throughout the organization.

Sean: So, if it’s running up on Amazon or you’re running it in your own local data center, it’s running on a server that has some kind of an OS, and it’s running on some kind of a Web server. What is the stack under WaveMaker? Is this a Linux/Apache sort of thing? Does it run on Windows?

Chris: The stack underneath WaveMaker is a Spring/Hibernate stack. We made a decision very early on that not only were we going to release our software under open source, but we were also going to eat our own dog food, in the sense that we were going to actually build our stack on top of an open source stack.

That means Spring and Hibernate on the back end, and it means JFON and Dojo on the front end. What’s great about building a platform on that stack is that it’s incredibly flexible. We can run in any Java server, we can work with any database through Hibernate, and with any security mechanism through Asiji. We can to talk to any Web service through JAX WS.

On the front end, we’re able to support accessibility and internationalization requirements through Dojo. So effectively, we’re standing on the shoulders of giants.

It’s very interesting that in the cloud world, and in the cloud development platform world in particular, companies have gone back to inventing their own wheels. For example, with force.com, every single element of that architecture is proprietary.

Not only do they have to invest a tremendous amount of resources to build the same quality of tools, but by definition, they don’t have the breadth of integration points they would get working with an open source stack.

Sean: If you’re dealing with a Java shop, this stuff probably fits right in. For more of a Microsoft .NET shop, it sounds like this would still integrate pretty well, because of the technologies that you’ve talked about. Do you have any examples you can cite about this being used in a heterogeneous environment where interoperability was important?

Chris: Absolutely. WaveMaker as a platform talks to any kind of a Web service, whether it’s SOAP or RESTful or RSS. We have customers who very easily integrate functionality from WaveMaker with functionality from the .NET environment.

That’s a big part of what’s out there, so we have to be able to build on whatever intellectual property the enterprise has already developed, and that includes .NET.

Sean: Can you talk a little more about the corresponding story for shops that do Java, PHP, and that kind of stuff?

Chris: We’ve got a number of examples where companies have been able to effect a real transformation. For example, KANA is a company that produces call center software. They supply call centers to about half of the Fortune 100, and they have effectively rebuilt their entire stack based on DB2, WebSphere, and WaveMaker.

They created an SOA bus where all of the functionality around these very complex multi-channel call centers is wrapped up into a series of web services. WaveMaker calls into those web services and provides the user interface, the ability to customize that user interface, and the ability to make changes to the back-end system.

Sean: If people are using WaveMaker to build their web front-end, how does it scale up and scale out?

Chris: The work we’ve done with KANA has been targeted toward some of the largest enterprises on the planet, and they’re looking at things like 20,000 concurrent users and sub-second response rates, so we feel very confident that our solution scales.

Here, again, there’s a huge benefit from open source. For example, if we found that our back-end wasn’t scaling the way that we wanted it to, there are a lot of Spring developers out there who could help us make it better. We’ve been doing work with KANA to make our Dojo front end perform better, and again, there are many people with expertise in building very high-performance Dojo clients.

So we’re able to leverage not only our own community of 15,000 developers, but the Spring community, the Dojo community, and the Hibernate community to tweak out the performance and basically learn from best practices.

Open source gives us a huge benefit in this area, as opposed to building our own proprietary back-end and then having to go through all the painful learning process of how to get it to perform without the community’s help.

Sean: I’d like to hear more about how you guys engage with the open source community, at a tactical or business level. You have obviously articulated a lot of good, strong benefits for having an open source development model, but sometimes inherent tensions can creep in.

For example, there are issues around having X features in the community edition and Y features in the paid edition. You don’t have a static product—it’s obviously evolving—so how do determine what goes into the full-blown edition versus the community edition? How do you determine the levels of support?

These questions seem to especially get complicated when the community would really like to see something in the community edition but you’re not yet offering it.

Chris: Those are hard questions, and I don’t think you ever get them completely right, even though you need to get them right enough. One thing that probably doesn’t get enough attention is that the open source community is not just developers. There’s also a very interesting level of cooperation and collaboration between open source CEOs.

I’ve had the benefit of being able to spend a lot of time with Marten Mickos of MySQL, Brian Gentile of JasperSoft, Rod Johnson of SpringSource, and a number of other CEOs. Those relationships have convinced me that open source is as much an attitude as it is a business model. If you really embrace the open source approach, what you get is a set of CEOs and companies that are wired to collaborate and cooperate and share best practices and learnings.

We found some rules of thumb for how to separate our open source and enterprise products. I think that the most important one is that the commercial features should have more to do with robustness and scalability of deployment than they should with fundamental things that you can or cannot do within development.

If you look across commercial open source companies, including WaveMaker, SpringSource, MySQL, and others, you’ll find trends in the things that tend to separate the commercial versions from the community versions. They tend to be clustered around management, monitoring, scalability, security, and other things that you really start to be concerned about when you take an application and deploy it into a robust enterprise deployment.

I think the most important thing is that the point of having a community edition versus an enterprise edition is not to make as many people as possible pay for your product. The point, to me, is that you’re trying to make the people who are willing to pay want to.

Therefore, you don’t pick a feature that could trip people up so that they get started with your product and then you somehow catch them because they want to do something else. The process involves sitting down in advance to determine who would never pay for the product anyway. The first step is to figure out a product that will make them happy and make them good marketers and developers for the project.

Next, you need to determine who would be likely to pay for the product, because they’re using commercial security, commercial databases, or commercial Web servers, or because they need higher levels of security and performance. You need to make sure that those people fall into the bucket where, in fact, there is a payment required.

If you think about it that way, it makes sense, and in fact, I would go one step further. You want to engage with that group of people who are never going to pay for your product and encourage them to invest time in your community. That’s the payment that you want from them.

For the group of people who are paying for your product, you want to provide as close to a commercial software experience as possible.

Sean: Is tracking sales and trying to convert those community members over to paid members even a part of your business model?

Chris: Certainly we track sales, but I have to say (and a number of other open source CEOs are starting to feel more this way as well) that people either have money in their pocket when they come into your community or they don’t.

If somebody doesn’t have money in their pocket, you’re not ever going to convert them. It doesn’t matter how compelling your for-pay product is, because they don’t have any money. The more you try to put the screws to them, the more you fritter away the goodwill that you would otherwise have with them.

Similarly, there are people that come into your community with money in their pocket, and if they find that they’re more productive, they’re getting better results, or their boss thinks they’re doing a great job, then they’ll give you some of that money relatively happily.

I think the whole notion is wrong that you should regard every single person in your community as a candidate for giving you their credit card.

Sean: I agree. At the same time, how do you go about encouraging the non-buyers to incent other potential paying customers? I realize there’s an ethos around it, but at the end of the day there are lights to keep on, bills to pay, and things like that.

Chris: From those people, we’d like to get good karma, because they like our stuff and benefit from it. Then we need to find ways in which they can give back that are non-monetary. I’ll admit that we’re not fabulous at this, although certainly the community is one place to do that.

Whenever we help somebody out in the community, we try to back-channel to them to make sure that things went well. If they’re excited, we usually suggest to them that, if they feel like the community helped them out, they should consider helping somebody else out by giving back to the community.

It’s basically karmic marketing. As people in the community get more confident in their skills, just as in any open source community, people get more involved. In addition to contributing code, for example, we’ve had people create communities in various different languages.

We want everybody to pay for the value that they receive, but some of the people pay in money and other people pay with their time. At some level, we’re indifferent to which they choose, because we think the more people who pay with their time, the more people they’re going to influence. Some percentage of the people they influence, as you said, are going to be looking for a commercial application of what we’ve got.

Sean: Because of the size and the acceleration rate you guys have, do you have a community head or liaison who deals with issues like that as their full-time role? If not, do you have some kind of plan to staff up that area?

Chris: I think open source is definitely a business model where you go hard or you go home, so if you’re not willing to pursue all the implications of open source, you really shouldn’t start down the path. And to me, the core of open source is the community, in the sense that if you don’t have a community, you really have nothing.

If I had two only employees in an open-source company, one of those people would be the community manager. That’s how important I think that position is.

In our company, we have a full time person who manages the community, and his background is technical. My view is that if you have a marketing person managing your community, you’ve said something pretty profound to the community, and I think there’s a spectacular mismatch between somebody with a marketing background and the kinds of questions and concerns that the community is needing to get addressed.

As your community grows, you may need to put a layer of management up there, but I think first and foremost, a community is about dialogue, and you need somebody who has the technical depth to be able to answer the questions of the people who come to the community. Otherwise, guess what? They’re not going to come.

So day one, we put a community manager in place. That is absolutely what drove us to 15,000 registered developers in our community in a little over a year. That’s absolutely what got the attention of companies like IBM, Amazon, Rackspace, and all the others we’re working with, and our community is our crown jewel.

Sean: How do you go about finding somebody to be your community evangelist? In a lot of ways, you can’t measure them using standard marketing metrics. They may even be out of the building, virtually or physically, more often than a traditional marketing manager would be. On top of it, by design they’re bringing you feedback that runs, sometimes, upstream of where you want to go, but by definition you have to listen to it.

Your story might be a personal one, for all I know; maybe it’s someone you’ve known for years. But if you could extrapolate, what should other projects look for when they’re about to hire a community manager?

Chris: The first key is that they need whatever technical depth is required to engage with specific questions in their communities. Second, you need somebody with some kind of management skills. Maybe they’re not going to be managing lots of people, but they need to be able to manage their time and their communities.

They need to be able to ask for help when it’s necessary. They need to be able to recognize when somebody’s really angry and be able to handle those situations. At the same time, they need to be able to recognize that somebody is really happy and take advantage of that as well, for instance by taking the situation to marketing to see what they can do with it.

You also need somebody with an incredible customer focus, because day in and day out, they will need to defuse people who have been disappointed in some way. People are bound to use your product for a while and then have it cause problems for them (or give them that perception). Some of those people will inevitably come back to your community yelling and screaming.

I tend to think that the combination of those skills is most likely to be found in a customer-support operation. People who have done technical support for software companies for a while know the job and how to keep their cool, and they tend have a good perspective where they can see trouble brewing, and similarly they can see opportunities.

That, to me, is the ideal. I think you could get that person from a number of different places, but technical customer support is probably the best place to look.

Sean: Well, we’re almost out of time, so are there any closing thoughts from your end?

Chris: I think we’re very early in the trajectory of the open source industry, and it’s been amazing to me to see over just the last two years, the kinds of changes in attitudes and understanding of what it takes to build a real ecosystem.

I think cloud computing is going to take that up another order of magnitude. Ecosystem is the killer app for the cloud, and companies that have learned the lessons of partnering and working with communities are going to be at a huge advantage as the cloud computing revolution takes off.

Sean: That’s a nice place to wrap up. Thanks for taking the time to chat.

Chris: Absolutely. Thank you.

Bookmark this:

• The creation of the Selenium web-testing tool
• The Sauce Labs startup around Selenium to run testing on cloud systems
• Automated web testing with Selenium
• Taking Selenium beyond testing into software documentation and marketing
• The revenue and business models of Sauce Labs
• The maturation of Selenium beyond predecessors and into the emerging cloud ecosystem

Scott Swigart: To start us off, could you introduce yourself and your company?

Jason Huggins:I’m the founder of Sauce Labs. I think my unofficial title is CTO, but the official one on my business card is “Executive Software Chef.” I’m a fan of silly titles, so that’s my silly title. I’m also the creator of the Selenium web-testing tool, and Sauce Labs is built around the things that we wanted to do, as a startup, with Selenium.

A lot of interesting things happened in mid-2004: Django and Rails came out. And a little bit later in 2005, Jesse James Garrett coined the word AJAX; he looked at what Google was doing with Gmail and Maps, and how they were using JavaScript to create a tremendously dynamic website.

In what seems like overnight, JavaScript as a technology to use in web-app development went from being taboo to becoming something that everyone had to do. Right around that time, I was also coming to the conclusion that JavaScript was cool and useful, independently of the AJAX hype wave.

I was the tech lead of a team at ThoughtWorks that was writing an in-house time-expense system. Unlike most corporate entities, we couldn’t dictate by fiat that everyone should use the same browser. We had a bunch of alpha geeks there, and people used whatever they wanted to.

The classic thing you were supposed to do in web applications at the time was to have everything done on the server and present a kind of static HTML page to the client. This JavaScript stuff made the site more lively and more fun, though, so we decided to use that. The problem was that, even as JavaScript was getting better and Google was showing ways to help it work across different browsers, there were still some rough edges there.

As the project continued, the same series of events would keep happening. We would have a feature implemented in JavaScript that would be working in IE but broken in Mozilla. We would file a bug, and our developer would fix it in Mozilla, but forget to test it in IE. So the next week, the bug was still there, but now it would be broken in IE, instead.

That was frustrating, and I decided that we needed to put a stop to it. We decided to come up with a way to write one test and run it across all of the browsers we cared about. That was the genesis of the Selenium tool.

Scott: How did the tool progress from that point to become an open source project?

Jason: I thought I was going to be famous with this time-expenses thing we were building. It turned out, though, that as I was showing people what we had done to test these two browsers, they said they wanted to use it on their own projects.

Those developers wanted access to the code, so we made a formal request to open-source it. One thing led to another, and it kind of took off. We never actually open-sourced the time-expense system, but Selenium definitely took off.

Scott: What was the trajectory from that point? Did the project grow primarily by word of mouth?

Jason: It grew by word of mouth, one project at a time. At one of the last projects I worked on at ThoughtWorks, the developers were frustrated that running tests with Selenium was taking too long–about 30 minutes to run all tests end-to-end. In a classic agile project, you want to run everything often–ideally, at least every time you check a file in and everything done in 10 minutes or less. If the whole process of build and test takes too long, you’ll skip running those tests.

Every project has a different conception of how long is too long, but usually five to 10 minutes is kind of the ideal. On this particular project, total test time was hovering at about 25 minutes. Mutiny was afoot, and I had the idea of getting a couple of extra machines. Using four machines, I wrote some code to split the tests across the four machines, running concurrently, and we got the testing time back to a good level.

A couple of months later, I heard through the grapevine that Google was also using Selenium, and that same contact asked me if I had an updated resume. Of course, that’s the offer you can’t refuse, to be courted by Google, so I applied. It turns out they were building a Selenium farm, and I arrived on the scene right when they were doing that.

It turns out that the Writely team, which is now Google Docs, had the same kind of an epiphany, if you will. There were also using Selenium at the time, and it was taking an hour to run their test. They ran the testing on four machines, and the time went down to 15 minutes.

They started telling other groups at Google, and it became a virtuous cycle where more teams started using this farm, which enabled them to get approval to buy more machines. Having more machines let them divide all of their tests across those machines and make everything go faster. Selenium eventually started being used for Google Apps and Gmail, Maps, Calendar, and lots of apps that I can’t talk about.

Scott: How did Sauce Labs grow out of that experience?

Jason: I was at Google for about a year and a half, but I was looking to get back to Chicago. Meanwhile, I met John Dunham, who is now Sauce Labs CEO, through Elisabeth Hendrickson, an expert in agile testing, and a mutual friend of ours. John wanted to get back into the testing space, and he sought me out just to have lunch with a guy interested in the same field.

I revealed to him I might be looking to do a start up, too. I was the idea guy, but I didn’t have all the business sense to be able to actually run a company. The initial “cocktail napkin” business plan was to prove the viability of throwing hardware at the software testing problem.

We had proved it on a small scale with my little project at ThoughtWorks and also on a big scale at Google. The next question was to determine whether there was an actual market for it, and we started Sauce Labs to answer that question.

It didn’t hurt that cloud computing was the most hyped story of 2009. The type of testing we’re doing at Sauce is the perfect use of cloud technology. We use disposable machines to run a whole bunch of stuff in parallel and run web browsers on all those cloud machines. Customers then drive those machines through our API to execute their tests.

It’s a classic problem where a long, slow process needs to go faster, and we just need a lot of disposable machines to run it against. Sauce Labs wouldn’t be really feasible without cloud technology.

Scott: Do I understand correctly that Selenium provides a plug-in that you use with your browser of choice that lets you record what you do? My understanding is that it emits a script that can then drive the other browsers through those same actions.

It can actually video record what’s on the screen as it’s driving them through those tasks, and it’s got logic to verify that the right thing shows up on the screen. Do I have that right?

Jason: There are several tools in the Selenium toolbox, but the tool you’re thinking of is a Firefox extension called “Selenium IDE”. The video recording is not included in Selenium out-of-the box–that’s something Sauce Labs has added to our online-based service to extend the capability and usefulness of running Selenium tests in the cloud.

Taking a step back, though, Selenium specifically is a functional testing tool (also called “acceptance testing”) that can verify features by interacting with a website exactly as a user would. Think of Selenium as a programmable software robot that knows how to use a web browser, and does so in all browsers, not just in IE or Firefox. We use IE, Firefox, Safari, Opera, and Google Chrome, on Windows, Linux, and Mac OS. We have APIs for Java, C#, Python, Ruby, Perl, and PHP.

The core minimum Selenium API is a collection of simple commands: “open”, “type”, “click”, and maybe “verify”. Open a web page, type text into a field, click a couple of buttons, and then ask the page if some text or another element is present or whether some state has changed. That’s the minimum essence of what Selenium does.

Scott: My sense, too, is that it’s not just simulating the browser. It looks like one of the reasons it seems suited to the cloud is that it’s basically starting up a VM with specific OS and browser builds, driving it through certain things, collecting screen shots, video, and whatever, and then tearing it down.

Jason: That’s actually what makes us unique as a cloud vendor. Everyone else is putting servers in the clouds, but we’re actually booting up the desktop environments and are running what you would otherwise call “client applications” in the cloud on those servers.

One of the more cryptic tag lines we had was, “Web browsers as a service.” That probably doesn’t play in Peoria, but it kind of gives you a sense of what we’re doing. We’re launching these desktops that you can connect to remotely and see that they are starting up IE or Firefox. It really is starting up the browser and you get to see everything a user would see.

Since you mentioned video, I will add that it’s been really important to me to add that functionality, to see directly what happens when a browser running in a cloud crashes, for example.

It was important to me in creating Selenium, since I am very visually oriented, to see what the browser was doing. I want to see whether it looks good or ugly, and I don’t really believe the application’s working until it actually runs in IE and Firefox. And by default, when you run things in the cloud, you can’t see what’s going on, so we put in the effort to add back that visual feedback when customers use our service. Video recording of every test has been a pretty popular feature with our users. They don’t just have some log files to look at. They can actually view the video, share it with their coworkers, and so on.

Scott: I would imagine that as the web becomes more interactive, testing gets an different or expanded focus, as well. For example, with a traditional shopping cart app, you need to make sure data is handled the right way, and that at the end, the “your purchase has been completed” string shows up on the final page.

On the other hand, to test the functionality of dragging a map or something like that, there’s not a string to test for. In some of these cases, the only way you’re actually going to know if it really worked is to look and see whether the map been dragged to the right position or whatever.

That need for video suggests that you are always stuck with a certain number of tests that require kind of some human subjectivity, even if you can automate it to some degree.

Jason: That’s true, although one of the “maybe someday,” crazy ideas I have is to add in some kind of “Mechanical Turk” functionality. For instance, once the video is created, perhaps you can automate answering the question of whether the functionality worked, even if it takes a human to answer the follow-up question of whether it looked ugly.

In other words, even if there are still questions that only a human could answer, it would be easier for them to do that by watching a movie and answering a couple of multiple-choice questions, instead of requiring them to actually perform the entire workflow manually themselves.

I read a blog post recently by Hugh McCleod, where the author suggested that, if you’re doing a start-up for any kind of company, there needs to be some kind of social object: some conversational aspect of what you do or produce.

For us, that’s this video that we can pass around, which can include assertions about whether it is an example of a passing test or failing test. If the test failed, it becomes a social thing that you can pass around to the other developers to say, “Look at the one-minute mark; you can see where the bug appears and the test fails.”

Ideally, you have a test for every feature in a website. That might be sending a message in Gmail, doing a product search on eBay, or buying something on Amazon.

Scott: It seems like that could have implications beyond app testing. What kinds of things have you considered elsewhere in the life cycle?

Jason: I would like to see some of the lines erased between documentation, marketing, and functional testing. My best example is an iPhone commercial, and even Android now these days. All they do is zoom in on the phone, show you opening an app, clicking buttons, seeing how cool it is.

They just show one feature, and how well it’s working, which is actually an ideal use case for Selenium and Sauce Labs. People write these tests so they show off one little feature, then there’s a video for it. Ideally, if you are a software developer, and you’re coding up a project, you add more features over time to the thing you are building.

You could have this continually growing little check-list of the features that you have written, and you can drill down and look at the recorded video of that feature working on latest version of your code.

You can see the history of that feature working, but specifically you can actually see it working over the life of the project. People are doing screencasts now to help sell their software, putting up little YouTube videos on their site.

The old approach was to let people try before they buy, downloading a trial before they bought it. With the unfocused attention of most people these days, now they want to watch the video before they even try it.

I’d like to make it so these videos get recorded all the time with all of your little features. You can use them either as-is or as your prototype for making little feature screencasts you can put up on your website.

Scott: It also potentially solves the problem of keeping the docs in sync, especially with software as a service solutions that might roll out little changes daily, weekly, or every couple of weeks.

Keeping the videos in sync could become fairly automatic, since if it’s been tested it’s also documented, and the only stuff that’s undocumented is the stuff that’s untested.

Jason: Actually, I’d like to see testing move away from being so focused on the tests themselves, and toward looking at the tests as more like screenplays, with the real focus being on the videos they produce. Those would be the actual artifacts that you care about.

As recently as five years ago, project testing was usually kind of a late process activity, done by one or two people in the corner. You called on testers and they used really expensive commercial tools, in isolation from everybody else.

Testing with Selenium follows the growing adoption of agile methodolgies where testing is something that developers do all the time. By analogy, I see a lot of those marketing screencasts being done very late in the process, and not by the developers. Screencasting today is where testing was five years ago.

I believe screencasting could benefit greatly by being pulled into the development process using the video capture functionality from Sauce Labs or similar tools. I think that the key glue there is automation, as well as encouraging testers and developers to think of themselves as screenwriters showing off their applications, rather than being so focused on acquiring the pass or fail data point.

Scott: There’s Selenium, which is an open source project, and then as you said, wrapped around it is Sauce Labs. Talk a bit about what Sauce Labs does. What do you add beyond the open source project? What’s your revenue model?

Jason: We started with the single focus of Selenium and cloud computing. As a request comes in, we either route that request to an existing running machine, or we boot more machines from a cloud vendor. There’s a lot of code needed that’s not in Selenium to manage this cloud environment and scale the infrastructure.

That’s the bulk of what we do, but we also collect log files, record video of each test, and make all that data accessible through our website. We also add support for this notion of multi-tenancy, in the sense that we have to make it work with lots of customers, so we have to set up and tear down cleanly between customer uses. That’s more code we have to write and maintain.

Along the lines of those security considerations, that’s how I found my other co-founder, Steve Hazel. He created a site called codepad.org, and you actually can paste in code from pretty much any language, and it’ll run it on his servers in EC2.

In addition to the actual security code itself, and managing how to properly sandbox running code in the cloud, Steve also provides that mindset of being security conscious and vigilant about how people can use and abuse your site.

Along those lines, we’ve expanded our mindset beyond just providing cloud execution to become the Selenium support company more generally. Because Selenium’s already been out there for five and a half years, a lot of people who have a pretty big installed base may not want to use our cloud service, but they do want a phone number or an email address where they can get support, bug fixes, updates for the latest browsers, and things like that.

We’ve expanded the scope of the company to a classic open source model of being the obvious company that you go to for help when you use this particular tool. Blended with that, we also buy cloud resources at wholesale rates by the hour and sell them by the minute.

Scott: I think you mentioned that the cloud provider behind this is Amazon EC2. It would seem that there are going to be a lot more choices as time goes on. I could envision you using multiple cloud providers simultaneously, transparently to the user, because it’s at a lower abstraction level where the VM is actually running, versus the data that the user’s getting back.

Jason: Right. There’s also an argument to be made that, because we’re telling browsers to go to some website, ideally those browsers should be as close as possible to the active servers that we’re testing.

It’s a simple matter of the speed of light. If the server we’re hitting is in Japan, and our test lab is in Virginia, latency is going to be a problem. Therefore, we’d like to be wherever our customers are, in the sense that if they’re inclined to host their application on EC2, our browsers will be there, too, so the network obviously is really fast.

There are lots of good reasons to be cloud-vendor neutral.

Scott: Obviously, if you’re in the same geography as somebody that helps, but if you’re in the same data center, that’s even better.

Jason: If nothing else, your bandwidth is free when it’s inside the data center.

Scott: On another topic, in getting ready to chat with you, we went out and did a little bit of homework and came across Adobe BrowserLab, which seems to provide some services that are similar to Selenium. Can you talk a little bit about some of things that are unique about Selenium?

Jason: I’m glad you asked that, because it’s where I think the original synapses fired and I was like, “Hey, maybe there’s something here with Selenium as a service.” The word “cloud” didn’t exist a couple of years ago, but the phrase “as a service” was in fairly broad usage.

What was out there was very similar to that Adobe site. There was something called BrowserCam, which sounds shady, but it’s not.

[laughter]

Jason: Basically, BrowserCam’s a screen-shotting service, and there are a bunch of others, like BrowserShots and Litmus.

The limitation there was that all of them were what I might characterize as just single-page snapshot services. They specifically market themselves to designers who just want to see whether a page that they’re developing renders right in all these different browsers. They don’t really target developers who need deeper functionality than that.

Also, at the same time, five years ago, virtual machines were not as commonly used, so if you only had a Windows machine and you wanted to see what a site looked like on Safari on Mac, you would submit your URL and get a screen shot back.

The most obvious limitation is that you can only give it one URL, and that’s it. They can’t test a multi-step workflow behind a login. So if you had, let’s say, a shopping cart, what you might really be interested in is to go from the front page, add something to the shopping cart, click “submit” and get all the way to the end of the purchasing process, then take a screen shot.

None of those screen-shot services could do that, unless you add some kind of automation mechanism to be able to navigate through the site to get to that point and then take a screen shot.

The genesis of what I originally wanted to do in the cloud was to take a classic screen-shot service and combine it with Selenium. Of course, I kind of took it to 11 and said, “Hey, why not, instead of just screen shots, record a movie?” And then you’ve got, theoretically, 30 screen shots per second that you can play with.

Scott: Is the size of the movies prohibitive at all? Does it take screen shots at set points, such as when it knows that a response is finished rendering, or is it really full-motion video of the browser?

Jason: Right now, it’s full-motion, in the sense that we’re running VNC as the protocol, remote desktop. VNC and RDP are the two main industry-standard protocols. VNC is kind of the open-source lowest common denominator, and RDP, which is the default one for Windows, is the next step up.

We just have a VNC server running on all of our test machines, and instead of connecting as a client application, we have a software library that I added some improvements to. The open source of that library is something called Castro. Instead of a user watching the remote screen, you dump the VNC traffic to what ultimately becomes a Flash video file.

Of course, we have done tweaking here and there, although it is inefficient if you do 30 frames per second. At five or even three frames per second, it actually works out well enough, though.

We’ve made some other improvements as well, although there are still more to be made. Specifically, I would love to bookmark the moments in the video where the interesting things happen, synchronizing it with the command you’re giving to the browser.

To your point, I could see where, instead of watching a video to get to those little bookmarked items, you could end up just getting something that’s a little bit more static. Using the raw video, for instance, you could go to each point of interest, pull out the video frame, and produce a sort of photo book or slideshow of just the relevant frames.

Scott: It would be similar in some ways to apps that detect when a sufficient number of pixels change, and so they know that some kind of transition happened.

Jason: Right. Perhaps it could be streamlined down to a slide show of only the key parts of the work flow, making reviewing the results go faster. Since we have recorded all of the video, we have all of the data, and we can then edit it in any number of ways.

Of course, people still come to us just expecting screen shots, which we will do for them. I think the next step is to synchronize the API traffic, the clicking, the typing, and so on, to make it easier to watch that flow of pictures, rather than having to explicitly take a screen shot of it the whole time.

Scott: Well, I think we’ve covered most of the stuff that I had planned to talk about. Any closing thoughts on your end?

Jason: The only thing that comes to mind is that we didn’t really get into the differences between Selenium user groups. There are developers on the one hand, using programming languages like Java, C#, Python, Perl, and Ruby. On the other hand, there are other Selenium users who don’t really do a lot of programming, but they use Selenium IDE specifically.

Sauce originally started out just to target those people who are using Selenium Remote Control. As we become more of a general Selenium company, we’re also embracing those users who are a little bit newer to automated testing, who are using Selenium IDE. We are branching out to do more things for those people as well. For example, we have a version of Selenium IDE, called Sauce IDE, that integrated with our Sauce OnDemand cloud service. So in a few seconds after creating a test, you can quickly start using our cloud service from inside the IDE.

Scott: Thanks for taking the time to chat.

Jason: Thank you.

Bookmark this:

• The evolution of various data models in databases
• Advantages of object-oriented databases in terms of horizontal scaling
• Handling issues when columns are added or deleted between application versions
• Suitability of MongoDB for the cloud
• OS support and usage with MongoDB
• The company (10gen) wrapped around MongoDB
• Community involvement in the development of MongoDB

Scott Swigart: To start, can you introduce yourself, 10gen, and the role of open source behind it?

Eliot Horowitz:Sure. I’m the CTO and co-founder of 10gen, which was really created to build MongoDB, an open-source, document-based database. We started 10gen and started working on MongoDB mostly out of our own frustrations after building scalable web infrastructure for a dozen years.

It really just came to a point where we were constantly fighting the same battles and building one-off solutions to handle functional gaps in relational databases. We started building MongoDB about two years ago, and we launched it earlier this year. People are getting very excited about it.

It’s very much targeted toward the standard web developer’s needs, anywhere from a simple one-guy website, all the way up to a Facebook-level, highly scalable, highly distributed system.

Scott: Building a storage engine is no trivial task, and beyond that, for a long time the bread and butter way of doing it was with relational databases. There was Oracle on one end of the spectrum and MySQL on the other, with a lot of other stuff in between, but people were used to throwing select statements at a database, joining tables together.

Then companies like Google came along, and over time, they’ve started to stored data differently. It’s sharded; it’s a schema-less store. You see things like SimpleDB up on Amazon, and I think Microsoft’s doing something similar and offering a data store. Talk a little bit about where MongoDB fits in the spectrum of databases.

Eliot: I think the first question you have to ask about any database these days is, “What’s the data model?” First, there’s the relational data model, which includes Oracle, MySQL, and all those. Then you have key-value, which has been around for awhile and includes things like MemcacheD and Dynamo from Amazon, where it’s really just “get input” type operations.

Obviously, those have been used for a long time, and they’re great for many cases. When you need simple gets and posts, they’re highly scalable, highly distributable, and very fast. They also have a very simple data model, although on the farthest possible extreme from SQL.

And then you’ve got a bunch of other data models in the middle: graph databases, tabular databases like BigTable, and document databases like Mongo.

There are a few different ways that you can think about document databases. One of the nice things about document databases is that they’re closely mapped to how most developers are writing code, whereas SQL databases were designed for accounting and banking 30 or 40 years ago, prior to the advent of web applications and the rise of object-oriented programming.

Now, when people are writing in PHP, Ruby, or Python, and they’re writing with data objects instead of rich classes (like user profiles). Trying to map that to relational data models is complex and slow. You can see the evidence of that in the number of different object-relational mapping libraries (ORMs) out there.

There are probably 1,000 different ORMs in the world, so clearly, it’s an unsolved problem, and every week you hear about a new one coming out. That whole class of problems exists because there’s a very clunky mapping from objects to relational databases. With document databases, that mapping becomes much simpler.

In many cases, it enables a direct mapping. You can take your object and direct it toward the database. And when you do need to do a mapping, the whole class of problems is massively simplified.

Another aspect of the same issue is that, when you loosen some of the versional constraints, you don’t have joins, because the data model is more suited to rich objects. For example, if you want to store tags for a blog post, you can store them as an embedded array rather than having a separate table that you have to join against. Once you get rid of joins, scaling horizontally becomes much simpler, which is a great benefit.

A bunch of the new databases coming out right now are trying to completely change the database paradigms, in terms of issues like how replication works or how you query the database. One of the things we’re trying to do with MongoDB is to keep it close to what people are used to. So, when moving from a relational database to MongoDB, there aren’t too many surprises.

You still have collections that are basically equivalent to tables, you still have different databases, and you still create indexes. You can still do an ‘explain’ to see what kind of query plan you’re going to get, and you can do ad hoc queries.

That means that if you’re coming from MySQL, it’s a pretty easy transition to MongoDB. The data model changes, but a lot of the things you’re used to are still going to be the same, mostly because we don’t think there’s any need to change them.

Scott: There was a guy a number of years ago who said that object-relational mapping is the Vietnam of our industry. It’s the thing that people just keep trying over and over to do, and it always ends up being excessively complicated. Back in my coding days, I spent as much time writing the mapping as I would have spent just writing the code to hydrate and dehydrate the objects.

Still, there have been object-oriented databases in the past. Talk to me about some of the things that make MongoDB unique or interesting compared to preceding object-oriented databases.

Eliot: The first object databases came out in the early ’90s, and there were a bunch of problems. For one thing, a lot of code was being written in C++ then, where they sped up the types, and the mapping layer really didn’t work all that well.

You could take a C++ object and save it, but not all the people were doing that. And then you had other languages where you didn’t really have objects the same way you do now. Remember that it was still very early in the trajectory of object-oriented programming at that point, so in some ways it was ahead of the curve.

The applications people were writing then were often more suited to relational databases, I think. Now, web scalability is a lot more important, because things can get big pretty quickly. I think people want much richer information from their databases, and they’re trying to use them in very different ways than they used to.

If you’re building a banking system or an e-commerce system, relational databases work pretty well. It’s really when you get into Web 2.0 sites that relational databases really start having troubles. If you look at Twitter, Facebook, or those kinds of websites, the needs have changed.

Really, though, I think the biggest thing is that the programming languages have changed a lot. No one’s writing web applications in C. Most web applications are now being written in PHP, Ruby Python, and some Java, all of which map very nicely to a document database. You can interact with any one of the languages and it all works pretty well.

I also think, going back to what we mentioned before, that the object databases before were actually more closely related to current graph databases than to document databases. The document database is really just taking MySQL, and instead of having a row, you have a document. So I think it’s a much simpler transition and it’s actually much closer to MySQL than a lot of people might think.

Scott: I’m calling it object-oriented, and you’re calling it document-oriented. Can you unpack that for me a little bit?

Eliot: In our case, a document is a BSON document, which you can think of as similar to a JSON document. You can have numbers, strings, dates, embedded arrays, and embedded objects.

If you have a user profile, instead of having first name, last name, then street, city, zip, you can have an address sub-object; you can have an address, and that can have fields. Then you can have multiple addresses, so instead of having address1 and address2 fields, you can have an array of addresses, and it’s already much simpler.

In your code, you can map that very cleanly. You have an address object that maps to this object, and you’ve got an array of them. You could have none, or you can have as many as you want. The database doesn’t really have a clunky interface, and you’re not joining, so there’s no huge performance hit. It’s all stored in the same place on disk.

Scott: Is a document basically a language-agnostic object, for lack of a better term? In other words, it’s not tied to PHP or Ruby or any particular programming language, but it basically has object concepts?

Eliot: Right; it’s basically like JSON, which is used pretty heavily in transporting data across the web now. We use BSON, which is basically like JSON except that it’s binary and has more types. JSON doesn’t have a native date type, and there are more types you need for web infrastructure.

Rather than hacking JSON, we just created a binary version that’s both easier for computers to read and extensible. It has more types, so you can have as many types as you need for real web applications.

Scott: Talk a little bit about scaling. You mentioned that because you don’t have joins and those sorts of things, and the data is all stored in one place, it scales horizontally very well. What does that actually look like, and how does it put the data back together, find which nodes different pieces of data are stored on, and resolve those kind of issues?

Eliot: The simple case is user profiles. Let’s say you have a billion user profiles and you needed 10 machines to store them all. Basically what you would do in this case is to say, “OK, I want to start my users collection,” and you could shard it on anything you want. You could shard it on email address or first name, or you could sort it geographically. You could shard it by country and then by state, depending on what kinds of queries you do.

Let’s say you sharded by using email addresses. Looking for a particular user is very fast. Now let’s say I want to find all people who live in Connecticut. It basically does a merge sort across all of the different shards.

It will go to each one and say, “OK, give me your users from Connecticut and get all of them,” and then it will aggregate them for you on the fly. From a driver/client perspective, it’s just the same if you’re sharded or if you’re not sharded. This is what people have been doing manually for years.

I’ve built many systems in the past where I needed 11 different databases to store something, and I did the sharding myself, which is a lot of manual work. What we’re doing is basically just generalizing what developers have been doing themselves in one-off solutions for years. It’s much simpler, because you don’t have to do joins.

A distributed join is very, very hard, but when your user profile is one object, it makes the problem much simpler.

Scott: It’s always interesting for me to ask where the edges are. What are some scenarios where a relational database may still be a better choice? Are there other tables where, maybe, a big table store is a better choice?

Eliot: I think big table solves the same general problems, although there we have some advantages over them. I think the document model is a little bit better than the tabular model, and there are a lot of subtleties there.

The big reason you use a relational database is if you need multi-object interactions. We don’t support multi-object transactions because of the performance overhead and complexity of doing distributed joins.

If you’re writing a banking system and you need to debit my account, credit your account, and do it completely atomically, we don’t support that. However, you can act atomically on any single object.

For example, you could say, “Increment this field on this object by five.” You can have it so lots of threads can do it, lots of different machines can do it, and it’s all atomic and safe.

Scott: Another issue that’s interesting is dealing with versioning. For example, in the course of versioning an application, you may change your objects, add properties to them, or various other things. With a relational database, it’s a relatively painful process up front, where when you change the schema, you’re changing the schema for the database.

That can mean that you have to populate a whole bunch of new columns with default values or do other things to migrate the database to the new version. With Google App Engine, it doesn’t work that way. You’re never really rolling the whole database forward to a new version, but you have to do more defensive programming

For example, let’s say you pull a user out of the database and that user hasn’t been touched for a version or two of your app. You’re pulling out the data the way it was stored the last time that user was saved, and so you may have to do some stuff in your code to map that older user data to your newer object. How do you guys handle that?

Eliot: You could migrate the entire database, but you don’t have to, which is the big thing. The simplest case is where you want to add a field. Here, the nice thing is that you can just start using the field. If an old object doesn’t have it, it will just return null, and it’s pretty easy and clean.

If you want to really change the format, you could actually have a migration script that migrates all your data, but that would require you to bring your site offline for an hour or so.

That’s another big change over the last ten years: people are no longer are willing to have maintenance windows. I remember working on systems where it was totally acceptable to have a two-hour maintenance window on a Saturday morning.

If Twitter went down for a two hour maintenance window once a month, I don’t think people would be too happy, and I think databases have to handle that. In any case, databases typically require you to do a little bit more work on the client side to make sure that you know what you’re getting out and you know what you’re putting back in.

Scott: On another topic, the cloud is a very important subject these days, and it seems like MongoDB is very well suited for something like Amazon EC2 and infrastructure on demand, where you’re able to install your own software and those kinds of things.

Obviously, the things that are more “platform as a service,” whether it’s ads or Google App Engine, have their own store, and to some degree you get what you get, which is what they provide. Talk a little bit about where you’re seeing usage of MongoDB in the data center versus in the cloud and some of the interesting uses you’re seeing for it.

Eliot: It does work well in the cloud, because it’s horizontally scalable, although it’s certainly not dependent on that usage. We see a lot of people running it in their own data centers, on their own hardware.

If you look at the startups that are using Mongo, a huge percentage of them are running on EC2, and I think that has less to do with MongoDB than with the fact that EC2 is a popular place to host sites.

We’re perfectly happy to run anywhere, but if you look at where people are building websites right now, it’s very heavily EC2 and other cloud-based solutions. You do not see very many startups buying and racking hardware to go on the Internet, which I did six years ago.

Horizontal scalability is a great thing for the cloud, because if you do grow quickly, you can add capacity very quickly. If you’re not horizontally scalable, the cloud doesn’t really help you that much; it’s actually worse, because you can’t buy or rent a $2 million Sun box on EC2.

Scott: If you start out with a small instance and you want to upgrade, and it’s your own hardware and your own data center, you can put more memory and CPU in it. With a cloud provider, you tend to get the type of instance you said you wanted, unless you want to shut it down and migrate the whole thing over, which causes that downtime.

Is this basically a Linux technology, or do you have significant Windows users?

Eliot: We have more Windows users than I would ever have guessed. If you asked me nine months ago if we were going to have anyone running in production with Windows, I would have said, “Maybe one or two.”

It’s a low percentage, but there are definitely many people running on Windows in production. We fully support Windows, OS X, Linux, and Solaris.

Scott: That’s interesting for the Windows developers. Is it sort of like PHP on Windows, or are people actually using it as a back end for .NET apps?

Eliot: PHP and .NET; not so much anything else. There’s a C# driver, and someone is working on a pure .NET driver. My knowledge of that right now is a little weak, but there are a bunch of different .NET things going on. I think there is a link adaptor, also.

Scott: We’ve talked a lot about the database and the project, so can you talk a little bit about 10gen and the company wrapped around it? What do you do around MongoDB, for instance?

Eliot: All the core MongoDB developers and the main developers of all the official drivers are 10gen employees. We basically sponsor the projects, and we are doing most of the development work. We also offer commercial support and training.

There are other interesting revenue opportunities for us, but those are the main things right now.

Scott: Talk a little bit about where it’s going. What are some of the top feature requests, or what do you see as some of the opportunities and interesting directions?

Eliot: If you look at our road map for this year, there’s no one big feature. I think the only big thing we’re doing right now is getting the auto-sharding to be fully production bulletproof. Traditional master-slaves and all that kind of stuff are pretty much bulletproof at this point.

The rest of it is just a lot of features. We are sort of similar to MySQL or PostgreSQL in terms of how you could use us, and people want all the features that they’re used to in MySQL and PostgreSQL. These include things like full-text search, SNMP, and all the assorted add-ons providing special indexing.

Scott: That must be an interesting thing to balance, because for example, MySQL’s original claim to fame was that it was simple, it was fast, and it wasn’t trying to be your enterprise relational database.

Then it just sort of kept growing and growing, because people kept asking for more things, until people looked at it and said, “What happened to my simple, lightweight, fast database?” And then you saw people going off and starting to work on Drizzle and those sorts of things.

How do you balance that request for one more thing versus wanting not to lose sight of what you set out to build and the scenarios that you really wanted to nail?

Eliot: I think it comes down to a few key things. One is making sure that we don’t regress on performance, which is one thing we’re very careful about. We constantly monitor our performance and make sure we’re not doing things that hurt it.

If there was a feature that would hurt our performance, we would think long and hard about implementing it, and we are definitely more interested in making the basics work than we are in adding more features.

We’re also careful about which features we let in, emphasizing common web infrastructure needs. It’s not useful for us to be simple and fast if you also need another database because you can’t do x, y, and z.

It’s very much the 80/20 rule. We want to cover the most general, wide-reaching things that people need, but it really comes down to good software engineering and good product management to make sure that we don’t screw up the basics. You don’t break the simple stuff by adding extra features. Another important aspect is good documentation.

Scott: I don’t want to throw too many software companies under the bus, but there is often a willingness to regress on performance, because they figure that hardware has gotten faster, so the user won’t be affected, even if, in terms of CPU cycles, it’s taking longer to do stuff.

How hard and fast is your policy of comparing the old version and the new one on the same box to make sure it didn’t get any slower?

Eliot: The next version coming out in a couple of months will be significantly faster than the previous version on the same hardware. We’re constantly working on that, so I’d like to think that that’s a pretty hard rule.

Scott: As time goes on, ultimately you want people to be using it, and you want it to meet people’s needs. Can you talk a bit about cases where, even if a change might hurt performance but really helps you nail a usage scenario, implementing it might be the right choice?

Eliot: I think a lot of that really comes back down to software engineering. For example, you can have such a feature but make sure that people can turn it off if they’re not using it, and that it won’t hurt their performance if they do turn it off.

We’re really trying to be very good on that level.

Scott: You mentioned that the people who are writing MongoDB mostly work for 10gen. Talk a little bit about the community story that lives around that.

Eliot: We’re seeing very few contributions to the core server code, mostly because it’s big, it’s flawed, and it’s complicated. There is a huge community, and it is unbelievably helpful in terms of the drivers. We’re getting lots and lots of great patches and contributors to the official drivers, so if you look at the PHP, Perl, Ruby, and File/Print drivers, there are tons of contributions.

The community has also been building a lot of great tools that sit on top of the drivers. One of the things we’re trying to do is keep the drivers pretty low level, very simple, and as fast as possible. If you want to hook into a login service, or if you want to hook up Apache to MongoDB for logging, we’ve seen a lot of great projects like that.

Scott: That’s interesting, in the sense that it’s a relatively common thread across a lot of successful open source projects that the number of people who contribute to the core might be really small, but the number of people who write add-ons, for example, might be very large.

It sounds like the ancillary components, for lack of a better term, related to MongoDB largely consist of those drivers, as well as administration and reporting tools that layer on top and add a lot of value.

Eliot: Right, and for example, there’s also a pretty standard adapter for Apache to log to MySQL. People are going to naturally want that to go to MongoDB, and there are going to be a lot of other different things that people will want.

That’s where the community has been very helpful, because we can’t be experts on a thousand different technologies, so it really makes sense for the experts in those fields to build them. It really works better for everyone.

Scott: Well, we’ve gotten to the end of our time, and I think we’ve covered everything I wanted to. Thanks for taking the time to talk to me today.

Eliot: Thank you. I’ve enjoyed it.

Bookmark this:

• The beginnings of DotNetNuke and engaging with Microsoft on open source
• Open source development paradigms interacting with proprietary software
• Seeking venture capital and monetizing DotNetNuke
• Response by the community to creating a commercial version
• Overcoming the pressures of a down economy
• Future directions: cloud platforms and improved distribution channels

Scott Swigart: Shaun, why don’t you take a second to introduce yourself and DotNetNuke?

Shaun Walker: I’m the original creator of DotNetNuke, which I started back in 2001, based on a reference implementation that Microsoft had released for the .NET Framework 1.0 Beta, an application called the IBuySpy Portal.

It was a sample application, released under a fairly liberal license, that allowed developers to come up to speed on the new .NET Framework. The application had some interesting characteristics, in that there was this notion of extensibility, although it wasn’t a fully baked solution.

I spent about a year on my own toying with that application and enhancing it. Really, what I had in mind at that point was building out a solution for amateur sports teams, so they could manage a website presence for themselves.

After about a year of doing that, I realized I really didn’t have the resources to convert what I had into a commercial venture. I had been active in the Microsoft developer community, specifically the forums on the www.asp.net site, where there was a small group of developers sharing their enhancements around that application.

It was actually Christmas Eve, 2002 that I decided to release my source code as an open source application. That initial release I called the IBuySpy Workshop, because I wanted it to have some connection to the original IBuySpy Portal.

I just announced it in that discussion forum on the ASP.NET site, and immediately, there were a lot of downloads and excitement around it, so I worked feverishly for about three months as people downloaded it and sent feedback. I think I was releasing every couple weeks at that point.

In March, Microsoft reached out to me because they had recognized that there was a bit of a community building around the application I had created, and they didn’t have any plans for enhancing the original IBuySpy Portal or building a community around it on their own.

Scott Guthrie reached out to me and invited me to come down to Redmond. We chatted about the vision for ASP.NET 2.0, because that’s what his team was working on at the time. We also discussed how DotNetNuke could be aligned with that vision and the importance of community, which I found to be quite refreshing, because it was not apparent to me that folks within Microsoft were thinking seriously about the benefits of open source.

We worked out an arrangement where Microsoft actually sponsored me for one year, which allowed me to leave my full-time job and focus my full efforts on building the DotNetNuke product and managing the ecosystem that built up around the project.

After that one year sponsorship, I had to find my own source of revenue to sustain the ongoing development. I relied mostly on advertising and sponsorship in the early years, then took on additional full-time resources to help manage the project.

It was in late 2006 that we actually took the bigger leap and formed DotNetNuke Corporation, which was a more suitable vehicle for governing the ongoing needs of the project.

At that point, I brought aboard as co-founders three of the contributors to the open source project who had been committed and loyal to the project. We worked under arrangement for about a year before we got serious about looking for venture capital. We were successful in getting a Series A round of venture capital in late 2008 from August Capital and Sierra Ventures.

Scott: We’ve talked to people who work on a lot of projects, from smaller ones to the largest ones like the Linux kernel and the Apache Foundation. Even though every open source project obviously evolves a little differently, what you’ve laid out there is almost a textbook story.

You started out scratching your own itch, like you said, with the IBuySpy sample that Microsoft put together, and it turned out that what you did was interesting to a lot of other people. A community formed around it, and so you decided to really invest in it and made it your day job.

The other people who came on board were basically people from the community who had made a name for themselves working on it. If you look at other open source communities, they’re also very reputation-based and reward people who actually get things done. Having an idea is great, but the ideas that tend to win out are those where someone has the idea and then rolls up their sleeves and implements it.

It’s really interesting to hear that basically that same road map was followed here as for the LAMP stack. This is built on top of Windows and the .NET Framework, completely proprietary technologies, but you’ve built an open source project and an open source business following the same trajectory you’d see elsewhere.

Shaun: I think that the open source paradigm transcends platform. It’s not necessarily a Windows versus Linux debate. The whole notion of open source is just built on openness, transparency, the willingness to share, and open protocols. All of that can live in either environment very comfortably.

It turned out that the solution that I’d come up with in those early stages actually related to a very large problem. That’s one of the other fundamentals of starting an open source project: you have to be scratching an itch that a lot of people are feeling.

The problem has to be large enough that it’s interesting for the initial person to solve, and it also has to be such that other people are willing to get involved in it and get excited about it. Without the passion, that organic growth really can’t happen.

DotNetNuke, right from Day One, enjoyed a very passionate developer community, because they were all working toward solving a fairly large problem: both individuals and businesses need website presences.

Some of the early tools that were available for entities to get online were really quite limited in their functionality, and so with the emergence of content management systems and web application frameworks, the sophistication of those online properties has gone up significantly. It’s really been software applications like DotNetNuke that made that happen.

Sean Campbell: I imagine that a lot of people, especially given the last year or so in the economy, are curious about that journey, going through and getting VC funding. What challenges would you reflect upon and say, “Well, if I’d known this before, I would have done something a bit differently during that process.”

Shaun: We spent a considerable amount of time doing fundraising in 2008, and I was surprised to learn the extent to which it’s not easy to raise money.

The fact that we had a very large and vibrant open source community wasn’t nearly enough for a lot of investors to show commitment. Really what they were looking for was some commercial aspect of the project as evidence that people were willing to spend money on this solution and not just download and use it for free.

Prior to this year, we didn’t really have much of a commercial component to the project. We could point to huge download numbers and a huge number of extensions that were available for the platform but we really didn’t have any evidence to point to business traction. That’s really what the venture capitalists are looking for.

They felt like they were taking a significant risk because it was an unproven model. In the end, we did find a couple of investors who were willing to take that risk and it’s worked out very well. We had a great year in 2009.

We offer support and other proprietary features around the platform and we’ve grown from 0 to nearly 400 paying customers in year one. We’ve hit all of our revenue goals and that wouldn’t have been possible without the injection of capital at the end of 2008. That allowed us to add more full-time resources to the project which really started to accelerate its growth.

Sean: How did you make the decisions around which aspects of the project you thought people would be willing to pay for and how did that guide the development of the project as a whole?

Shaun: In the very early stages of the project when we didn’t have a commercial version, we would implement pretty much any feature that came to the surface from the community as being desirable. Over time, we found that a lot of those features that we were adding really only catered to a small subset of the community, largely the more serious business users.

When it came time for us to introduce the commercial versions of the product, it was actually fairly simple to dissect the needs of the different constituents. We could look at a typical community user as really only running a single website off of a single install, with fairly limited content management needs.

On the other hand, a more serious business user might be using our platform to run multiple websites off of a single install, and they might have way more advanced content management and scalability needs, as well as demands around performance and security. So, if you looked at those different vectors, especially performance, security, and advanced content management, it became fairly obvious where different features should reside across the different product lines.

The other thing that’s important to note is that, right from Day One, the DotNetNuke application was architected in a way that’s very modular. So introducing proprietary functionality around the open core was fairly straightforward for us to do because the application had always been designed that way.

We have a very vibrant extensions ecosystem for both what we call skins, which are the look and feel, as well as mini-applications that can be embedded in a DotNetNuke framework, which we call modules. There are more than 6,000 different extensions available today on our marketplace at www.Snowcovered.com.

With that vibrant ecosystem already there and people already used to paying money for extensions, it was a fairly easy transition for people to adjust to the fact that there was going to be a commercial version of the core application as well.

Sean: Did you have any resistance from users who had been getting the application for free for years, when all of a sudden you started telling them that they would have to pay for certain features? Sometimes it’s just a vocal minority, but there’s generally a group like that.

Shaun: That aspect of open source in general speaks to the transparency aspect. Basically, anyone who’s involved in an open source project can get up and stand on their soapbox and vocalize what their concerns are. Unfortunately, their concerns can be quite self-serving, rather than considering the broader community or the broader context.

As we made the transition from being a pure, free, open source product by adding a commercial version, there was certainly a vocal minority in the community that expressed their concerns through our public discussion forums.

As a project owner, you have to be very attentive to staying on top of those and not just ignoring them, but speaking to people’s concerns so that they understand that broader context because often they hadn’t considered some different perspective.

It was surprising, but a lot of people in the community who were vocally opposed to us having a commercial version didn’t consider the fact that we were actually getting requests from many businesses that wished there was commercial support available for the product.

Some people have the perspective that if you use it for free, then support should be free, and they hadn’t considered the broader business implications of that.

Once you start explaining those things, people can start to get a bit more comfortable with it. You’re never going to please everyone all the time, though. There’s always going to be a certain subset of people that are not going to buy into your arguments, and they’re going to continue to be vocal.

That becomes a challenge to deal with on an ongoing basis but I guess we’ve gotten pretty savvy at dealing with those things over the last seven years. It’s no different than any other open source project.

To be honest, that friction is a good thing because it keeps everybody a bit more accountable. Even if certain people have a very extreme point of view, by being forced to listen to that point of view, it keeps you more from straying from what your true goals should be.

Scott: It’s become almost cliché, I guess, to talk about the economy, but you guys were going for VC funding during a pretty tough economic time. It’s always interesting to hear stories about people who have thrived in those circumstances.

Talk about some of the implications of seeking funding during this climate.

Shaun: We signed the term sheet in September of 2008 with August and Sierra, and a couple weeks later, the bottom fell out of the economy. At that point, we had the term sheet in hand and we were going through due diligence. Due diligence took much longer than expected, because we had some clean up to do in terms of making sure everything was in order from a legal perspective to satisfy the investors’ needs.

But the investors were committed at that point to doing the deal, so we worked through due diligence and were successfully funded and then they gave us a runway, which a lot of companies didn’t have, to focus on introducing some new products in 2009, adding a support component, and staffing up the business.

Where a lot of businesses were laying people off, we had the ability to actually hire people. We got to hire some really great people, because a lot of very talented people were out of work.

The other thing that worked in our favor, obviously, is that open source companies traditionally have lower costs than proprietary offerings. So when we look at some of the competitors that exist in the proprietary space, DotNetNuke is a more economical, more competitive solution in that regard.

I think that we picked up a lot of business this year based on the fact that we are a more economical offering for putting up a website.

The combination of those factors really helped us in this down economy, and as we’ve come out of this, hopefully 2010 is going to be a much better year for the global economy. We’re hoping that it helps us gain some additional momentum.

Scott: What do you see as the future of DotNetNuke?

Shaun: In 2010, we have some high-level objectives in terms of things we want to deliver. One area, in particular, is the marketplace where commercial extensions are sold for the platform. Like I said, there are about 6,000 extensions in that marketplace today, but it’s not really a seamless experience.

A customer has to install DotNetNuke and then they have to browse through our marketplace, make a purchase, and download a product. Then they have to upload it to their site.

As a software marketplace, the iPhone’s App Store provides a much more seamless experience, where you can view the different extensions that are available for the iPhone and choose to purchase them immediately. They download seamlessly to the device and they’re installed.

We want to build that type of tight integration between the DotNetNuke platform and the DotNetNuke marketplace in 2010. We believe that will really help consumers and businesses better manage their websites in a more seamless manner with less obstacles.

Another area that we see a tremendous amount of interest is around the cloud. It’s still challenging today if a business wants to put up a website. Although there are over 50 hosting partners today that offer a one-click install of DotNetNuke, if somebody comes directly to our site to download the software, they have to figure out what the next logical step would be to get it up and running. Basically, there are a number of obstacles in the way of a person accomplishing their goal.

This whole notion of cloud takes that complexity out of the equation as it provides a turnkey, instant gratification installation process that enables a customer to get up and running immediately in a suitable environment with all the necessary resources in place.

Scott: What’s your story around the Windows Azure platform? That seems like a logical cloud for you to land on because it is .NET based, and it’s not the most painful thing to migrate an ASP.NET application over to that.

Shaun: We are definitely talking to the Windows Azure team and it’s one of the cloud platforms that we would like DotNetNuke to run on. We are also talking to the Amazon EC2 team and, in fact, we have DotNetNuke running on EC2 already.

Like you said, Windows Azure is a natural choice because it’s native Microsoft technology but, at least in the early releases of Azure, it’s not a no-brainer in terms of getting a .NET application running on Azure. You have to do some re-architecture and, at that point, I don’t know if you’re tied too closely to the Windows Azure platform.

Microsoft has recently come out with some announcements that they are going to be offering some Windows virtual-machine-type deployment scenarios, much like those Amazon provides. That might be an interesting cloud option for DotNetNuke, as well.

You’ve also got to consider that those are the two predominant cloud players but there are a lot of smaller players, as well, like Rackspace and even smaller hosting providers that are offering cloud type offerings that give you a lot more control.

One thing that I’ve noticed with some of the larger players is that, in order to commoditize the service and allow it to be offered at a low price point, they actually take away a lot of features and a lot of flexibility that you would traditionally get in a typical web-hosting environment.

That’s where a lot of the smaller players are able to compete, because they can provide cloud-type services but with greater flexibility. So, we’ve got to look at all of these things when we look at running in a cloud.

Scott: It sounds like you are definitely rapidly pursuing the cloud, in a provider-agnostic way, and you’d like to be an option for as many clouds as reasonably makes sense.

Shaun: Right; we are avoiding lock-in, which aligns with our open source ideals.

Scott: Thanks for taking the time to talk today.

Shaun: My pleasure.

Bookmark this:

• The origins of Freenet and anonymous peer-to-peer architecture
• On the mathematics behind “Six Degrees of Kevin Bacon”
• Observations about optimal strategies for managing an open source community
• The Swarm project’s approach to distributed data
• Implementing fork/join and related concepts in distributed systems
• Relative beneits of open source and proprietary approaches to development
• Incorporating open source into an overall business strategy

Scott Swigart: To start off our conversation, could you please take a moment to introduce yourself and talk a bit about the different efforts you divide your time between?

Ian Clarke: Sure. I have a degree in computer science and artificial intelligence from the University of Edinburgh in Scotland. I started my career as a subcontractor to the European Space Agency.

Prior to graduation, I did some work on emergent systems and anonymity on the Internet. That grew into an open source project for which I act as founder and coordinator called Freenet, which started back in 1999 and has been ongoing ever since.

When I made Freenet publicly available after I graduated in 1999, it ended up getting quite a bit of publicity, and we released our first version in March 2000. In fact, a paper we released in 2000 describing Freenet ended up being the most cited computer science paper of that year.

Freenet was kind of novel, in that it was the first defense that allowed anonymous peer-to-peer architecture. Freenet also was precursor to the concept of the distributed hash table, which is a technique to efficiently store and retrieve data from distributed computers.

I moved to California and started a company, raising $4 million in venture capital, primarily from Intel. We developed a peer-to-peer content distribution technology, which we sold in 2003. All this time, Freenet was my hobby going on in the background.

I then did some consulting, and as part of that, I designed a peer-to-peer video distribution system for Janus Friis and Niklas Zennstrom, who I’m sure you know are the creators of the software you’re currently using.

I handed that off to their team in 2006, and it became a project known as Joost, which you can look at as the next big thing after Skype. I then co-founded a company called Revver, which was one of the early entrants into the short-form video-content sharing space. Revver is unique in that we were advertising against video back into 2005 and sharing the revenue with content creators.

Scott: How did that set the stage for the projects you’re working on now?

Ian: One of the problems I was working on as Chief Scientist at Revver was how to target individual videos with specific ads to maximize revenue. I became very interested in the whole field of collaborative filtering, predictive analytics, and recommender systems.

My current day job, which is a project called SenseArray, grew out of that. One area I learned a lot about with Revver concerns the challenges of scaling up a website, scaling up databases, and worrying about what data goes where and what data is cashware.

I believe that a lot of the architectural problems inherent in building a website and having a bunch of people actually start using it should not be the concern of the software engineers. Those issues should be handled by the framework in a mostly transparent way.

Those thought processes gave rise to a project called Swarm, which I spoke publicly about only a couple of months ago for the first time. Right now, it’s at a very early conceptual prototype stage, because I’m trying to get people interested in helping me work on it. Swarm is extremely ambitious and encompasses some very difficult problems.

Scott: Freenet sounds like the kind of thing that people might have been more interested in 2002 or later, in the sense that it predates the Patriot Act and widespread eavesdropping over global communications. Freenet provides an anonymized way to get out content and communicate with other people.

What do you think made Freenet interesting at the time you launched it, and how did it become more interesting because of changes culturally and geopolitically between 2000 and today?

Ian: When we first launched Freenet, it really generated two kinds of interest which were almost orthogonal to each other. One could have easily occurred without the other, but they both happened at the same time. The first kind was academic interest in what Freenet was doing and how it was doing it.

Freenet did a couple of things that had never really been done before. One of the core problems that Freenet tried to solve was to find information quickly from data that is distributed across a large network of computers. Many of those computers are unreliable, and some could be operated by your enemies.

You cannot rely on any form of centralization in order to do that. You probably remember Napster, which was a peer-to-peer network that maintained a central directory recording where everything was. Of course, that central directory can be shut down, and that really was not an option for Freenet.

At the time, Freenet was designed to be used in countries like China, Saudi Arabia, and Iran, where governments are not constrained by constitutions and laws in the same way that they are in this country.

Freenet was intended to operate in a very hostile environment, which creates some difficult problems. How do you find a piece of information, without consulting some centralized entity that keeps track of where it is? I developed a technique for doing this that relates somewhat to small world theory.

Scott: That’s the same theoretical basis as the Six Degrees of Kevin Bacon game, right? [laughs]

Ian: Exactly. The whole notion that you can connect between any two people through about six other people actually has a lot of mathematics behind it.

It turns out that if you just randomly create a network of things, you cannot so easily get from any one point to any other without omissions. For that to work, it must be a small world network, and it just so happens that human relationships do form a small world network, whereas that’s not true of any arbitrary network of things.

One of the novel things in Freenet was exploiting small world theory to solve the problem of searching and retrieving data in a decentralized, efficient, scalable way. I think that’s part of what generated the academic interest, and some of the stuff we were doing with cryptography was also quite new.

We certainly did not invent any new cryptographic algorithms, but we were using existing algorithms in some novel ways. That generated interest in the academic communities, computer security communities, cryptography communities, and what later became the peer-to-peer communities, although Freenet slightly predates the term “peer-to-peer.”

The other type of interest we were getting was mainstream interest from media like the New York Times and 60 Minutes, which related primarily to the copyright implications of something like Freenet.

Around about that time, of course, Napster was getting a huge amount of publicity, but a lot of people realized that Napster would probably get shut down, and here was Freenet, which could not be shut down. That generated a lot of mainstream media interest that really didn’t have much to do with the academic interest.

In some ways, they conflicted with each other, especially since some academics dismissed Freenet because it was getting so much mainstream publicity, which is kind of a weird effect.

Scott: What did you learn from Freenet in terms of managing an open source community? A lot of people have talked about why communities succeed or fail, but what are some of your lessons learned from that experience?

Ian: I have made a lot of mistakes, and I think I will continue to, so I certainly do not really see myself as being an authority. I think what has worked quite well for us has been being very open in terms of allowing people to join the development effort. I know that in some open source projects, it is incredibly difficult to gain access to source controls, and it is a big deal when you do.

We take the opposite tack. If somebody emails us and says they have an idea about how to do something, we give them access. If they do something wrong, we can revert it, which is the beauty of source control. We do everything we can to lower the barriers to entry for people to contribute to the project.

Another lesson has been that it’s very important to explain to people that they shouldn’t wait for someone else to tell them what to do. I find that a lot of people send messages to our mailing list, saying that they want to be part of the project and asking for something to do. Invariably, nobody will respond to them.

People need to know that nobody is going to laugh at them or shout at them or get mad at them if they just try to improve something. They can just find something that needs doing and then go do it.

Initially, one of our most prolific contributors over the years was Oskar Sandberg, who actually ended up doing a Ph.D. on stuff related to Freenet. He initially joined the project by writing a very simple Perl utility. To be honest with you, I can’t even remember exactly what it did, but it was something extremely simple, and from there he got more and more involved.

It’s really about lowering the barrier to entry, but also making it clear to people that they have to take stuff. They can’t wait for it to be given to them.

Scott: In other words, it’s better if someone comes to the project with their own itch that they want to scratch rather than “Hey, I just want to be part of this put me to work.” We hear from a lot of people that the best developers are often the people who are really using it all the time and so they know, “Hey, this is cool but it would be even cooler if it could do X.”

To return to Swarm for a moment, one of the things you mentioned that got my attention is that this is an incredibly ambitious undertaking. What is it that makes Swarm so ambitious, and how is that likely to impact the management of the community around it?

Ian: To put that more in context, I should first talk a little bit about what Swarm is. The basic premise is to build an environment where computer programs can be written so that they can either run on one computer just like a normal program or be distributed across many computers in a way that is mostly transparent to the programmer. The fundamental concept behind Swarm is to move the computation, not the data.

All computer programs are essentially a collection of code that operates on data, and the data can potentially be distributed across multiple computers. The concept with Swarm is that, if code that is executing on one computer needs to do something to data that is on a remote computer, you should move the computation to where the data is.

The conventional approach would be to move the data to the computer where the code is executing, and the Swarm approach is actually a little bit like the MapReduce concept that you may be familiar with.

Scott: In fact, I was just thinking that I should ask you how this approach is similar or different from Hadoop’s approach.

Ian: They both share the maxim “Move the computation, not the data,” but Swarm is really a lot more general. In order for a problem to be solvable through MapReduce, it needs to be possible to break it down into a map operation and a reduce operation.

Some problems are reducible in that way, especially stuff like indexing web pages, but many problems are not. In particular, anything that requires the equivalent of a database Join between two tables simply can’t be broken down into a MapReduce in any kind of efficient way.

In short, then, MapReduce is the same general concept but only applicable to a specific class of problems, whereas Swarm is a lot more general. Really, you can write any computer program, and with Swarm in particular you can write your code in Scala

I’m not sure how familiar you are with Scala, but I consider it to be the successor to Java. It has a lot of benefits of Java but without a lot of its annoyances and with some neat features that Java may never get, like closures, type inference, and that sort of thing.

You write a computer program in Scala, and it can really do anything. There is no requirement that you can break it down into MapReduce operations. It can just be an arbitrary computer program.

Scott: Can you share a few particulars about the Swarm approach to handling distributed data?

Ian: Let’s say an object on the computer that is executing happens to have a pointer to an object on a remote computer. At that point, Swarm essentially freezes your computer program, serializes it through a mechanism called continuations, and migrates the computation to the remote computer, where it starts up again.

From the perspective of the programmer, you don’t even need to know that this is happening or where your code is executing. Swarm can actually move the execution state of your code around between multiple computers in a way that the programmer doesn’t really have to think about it.

Now obviously, it’s not inherently very efficient to serialize a computer program’s state and migrate it across the network, so the second component of Swarm figures out how to arrange the data to minimize the number of times that the program state has to migrate from one computer to another.

Those are essentially the two components of Swarm. Firstly, allow a computer program’s state to jump around to follow data, so that you can just distribute data across multiple computers and then the computer program just follows it where it needs to go. The second component helps us be smart about where we put the data to maximize efficiency.

In terms of what makes this hard, the continuation mechanism I mentioned is something called portable continuations, which are supported by very few programming languages. Even Haskell cannot support portable continuations, and you know, if you can think of an obscure capability for a programming language, chances are Haskell can do it.

[laughter]

Ian: It just so happens that in Scala 2.8, which is the upcoming version, somebody implemented portable continuations. In many ways, this is the kind of thing that makes Swarm even possible.

In other areas of difficulty, there are all kinds of concurrency issues when code is acting on data that can be distributed across multiple computers. You have to deal with persistence and consistency, and all of that.

Problems of consistency and concurrent access to data are part of a very active research area at the moment, and there are a lot of hard problems there. How do you know when it’s safe to delete data so you can efficiently implement garbage collection across many computers? How do you implement stuff like atomic transactions when your data is distributed across multiple computers?

All the time, you’ve got a separate overseer process that’s watching how your computer program is jumping between computers and trying to move data in order to optimize that. I’ve got a lot on my plate, and I cannot follow all of these problems myself. At the moment, I’m trying to attract people with expertise in areas like handing concurrency and software transactional memory to the Swarm project.

Scott: I am thinking about the approach you describe and comparing it to a more traditional environment, where 100 web servers are talking to a handful of middle-tier servers, which are talking to one huge honking database. You scale that thing up as large as you possibly can, and if you can’t scale it up any larger, you fail.

I’m also thinking about the big table approach, where data is shared across lots of nodes. With something like Google App Engine, you get scalability, but you lose some relational database operations like Joins. You may be limited in terms of not being able to do the equivalent of a select that returns a million rows. You’re limited to a thousand results or something like that.

You talked about Joins and a lot of these concepts, and it sounds like what you are doing with Swarm is envisioning a novel data store. The data is distributed across a lot of nodes, and there’s an optimizer that’s always positioning the data so it will be the most efficient for the code to come to it.

At the same time, though, you’re still supporting what people tend to think of as relational operations, like Joins and those types of things. Am I envisioning it correctly?

Ian: I think you’re absolutely right that you could do things like Join using Swarm, although that would really be at a higher level of instruction. So, for example, somebody could implement a relational database on top of Swarm, but Swarm is not itself a relational database. Really, Swarm’s representation of data is as an object graph, much as data is represented in memory.

Scott: It seems like the concept of moving the execution to the data becomes more and more important as everything now just generates terabytes of data, from super colliders to oil fields.

It’s no longer practical to move the data to the computation, and you talked about your code running in one place for a little while and then having its state packaged up and running somewhere else.

Still, a lot of times what you want is for your code to sort of fan out. How does Swarm deal with the notion of making the logic fan out and go parallel when it can across nodes? And does it take that responsibility off of the developer, or is that something they would have to be conscious of?

Ian: This is something I’m thinking about at the moment. For example, one thing you could do is implement something like a fork/join approach to parallelization. I know there is a fork/join framework that will probably be part of the next major version of Java. You could absolutely implement something like that on top of Swarm, where you explicitly say to it, you can parallelize this if you want.

I wouldn’t say the parallelization is completely transparent to the programmer. It’s not like the system is examining the code and being very clever about figuring out what can we do in parallel and what we need to do serially.

What you can do is implement something like fork/join so the programmer can explicitly give the system permission to parallelize something. Then Swarm will decide where it makes sense to run on one thread or whether to fork it and send different parts of the execution to different other machines and then recombine it all at the end.

In short, the answer to your question is that you can do parallel processing with Swarm, but it probably won’t be completely transparent to the programmer.

Scott: Still, it sounds like it would be implemented from the programming standpoint in more of a declarative way, rather than the with imperative fork/join syntax that you would have in other languages.

Ian: I think that is almost certainly true, although I should say that a lot of this really is very much still on the table. You could implement fork/join as a library on top of Swarm. Swarm gives you primitives such as fork a process or communicate between processes, and you have a lot of flexibility about what you build on top of that.

Scott: Taking the conversation in a slightly different direction, with Freenet, you manage the community, and with Swarm, you’re trying to attract people to tackle some nearly intractable problems. Then you also do proprietary software development.

Talk a little bit about the difference in the way you see things working in the proprietary world versus the open source world and where you see advantages or disadvantages to either approach for certain kinds of problems.

Ian: There is the obvious benefit in the proprietary world that when you employ people, they more or less they have to do what you say, which is certainly not the case in the open source world, where you have to persuade people.

Open source can make it a lot more difficult to put processes in place, because programmers by nature often resent processes unless they strongly believe in them, although even in the commercial world, if you impose something and nobody believes it’s any good, then it’s probably not going work very well either. You can implement structure more easily in the proprietary world though, I think.

The proprietary world also tends to present fairly clear goals. If you’re going in the right direction, people will buy it and keep paying for it, and if you’re going in the wrong direction, they won’t buy it, and you don’t have a business.

With something like Freenet, it is hard to know what exactly you’re trying to be. For example, are we trying to service people living in countries like the U.S., people living in China, or both? The goal of proprietary software is to make money, and hopefully you can translate that back to how you need to prioritize different feature-functionality today.

Scott: I’ve never heard anyone call that out as an advantage, although it makes perfect sense. To put it another way, in proprietary software, there’s very low latency between the moves you make as a software company and the signals the marketplace sends back to you.

On the other hand, a lot of open source projects don’t even know how much of it is out there. It’s very nebulous how much “share” it has, so there’s a little more latency between the time a project starts to head in the wrong direction and the point where it’s really receiving clear information, that it is negatively affecting the use of that project.

Flip it around and talk about some of the advantages of open source that aren’t really present in proprietary software. For example, if I am a Fortune 500 company, I’m not in the business of building a web server, but I need one. It’s a lot more beneficial for me to devote a handful of programmers to Apache so we get the things we want, rather than building one from scratch.

Ian: That is certainly very true. From the perspective of the industry in general, open source permits people to create and collaborate on aspects of infrastructure that don’t need competitive advantages, but just need to work.

Because of that factor, you can have Microsoft and other companies, whether they like it or not, wind up cooperating with each other to the benefit of all. If you look at it from a game theory point of view, proprietary software can be more of a zero-sum game that tends to discourage collaboration, whereas open source tends to encourage or even enforce collaboration if it’s a viable open source license.

I certainly believe that the software industry is much better off with open source software. But on the other side, in many situations in my career, I’ve been building proprietary software, and people have said, “Oh, you should open source it. Wouldn’t that be great?”

It would be great for people in general, but it would remove the incentive to create the software in the first place. Because generally speaking, open source business models tend to be services-based business models, which do not scale as well as product-based business models.

To return to your original question, the advantages of open source versus proprietary are different, depending on whose perspective you take. You can build a business creating open source software, but I think it’s more difficult.

Scott: We talked to a VC, and his comment was “Sometimes we give open source credit for being more different than it is.” There are a lot of companies that go with kind of a “freemium” model, where they have a free version of the product, and then they try to up-sell you to the enterprise version. It’s the equivalent of a loss-leader, which isn’t new in business, or the modern equivalent of the 30-day evaluation.

Do you think it’s true that we sometimes highlight more differences between open source development and proprietary development than there really are?

Ian: It all depends on your strategy. I think there are many different reasons for open source. For example IDEA, the Java IDE, recently released a community version. I certainly hear a lot about companies that build open source community versions and then try to up-sell people to a commercial version.

I have not seen a lot of solid data demonstrating the success or otherwise of that approach. I think there are some situations where a company has a wider strategic interest to create a platform; like for example the Google Android platform, or Google Wave.

But, in those cases, a much longer-term strategic interest informs those high-risk projects. I think a lot of Google’s open source projects do not make money, and will not make money, even in the longer term. But luckily enough, they’ve got a cash cow in the form of advertising that they can use to subsidize all of these things.

Scott: You see a lot of experimenting, but it seems that notable successes are not all that prevalent. It’s also hard to determine whether success is because they went with an open source/enterprise business model, or whether something else contributed to their success.

Just to be sensitive to the time, can you add any closing thoughts?

Ian: I’m a huge user of open source, and I have several hats on my head. I’ve got a software engineering hat, where I just want there to be cool stuff, and anything that helps there be cool stuff is all right by me. I think almost all open source falls into that category. I think open source’s existence benefits the software ecosystem and innovation in general.

Then I’ve got my entrepreneur’s hat on, where I’m thinking in terms of whether being a producer of open source can make me money. The software I write is predictive analytic software, and I typically sell it to CTO-level people.

It’s quite a technical sale, so one of the challenges is to find customers and get them to pay attention for long enough that they actually understand what the software does and whether it can help them.

I have a blog where I will release small, relatively self-contained snippets of code that do stuff that relate, in some way or another, to what my product does. I know that the type of people who are going to find these snippets of code may well be the type of people who work for a company that could use the software I create.

It’s not so much a loss-leader as it is a publicity tool, and that is definitely one way in which producing open source is in the business interests of even a small software company.

Scott: Thanks for taking the time to chat.

Ian: Thank you.

Bookmark this:

• Eucalyptus as a technology-neutral abstraction layer for cloud computing
• Enabling utility pricing models for compute resources
• Toward the ability to mix and match platform services with infrastructure services
• Open source significance and strategy in Eucalyptus
• The evolving role of community in the Eucalyptus project
• Real-world implementations and usage scenarios

Scott Swigart: Could you start off by introducing yourself and Eucalyptus?

Rich Wolski: Sure. I’m the Chief Technology Officer and the college professor whose research group did work on Eucalyptus originally at the University of California, Santa Barbara.

The project began as a computer science research investigation. We were looking at how to combine large-scale HPC programs in fields like chemistry, physics, weather forecasting, and biology with the emergence of utility computing offered by third parties, which is to say, cloud computing.

We started looking at that question in an empirical way, using a driver application of weather forecasting code. The goal was to combine Amazon’s AWS, a number of university data centers, and the NSF supercomputer center site in one large execution platform for this legacy weather code. To do that, we actually wound up having to build an interface layer for that code that could run in university data centers.

To make our programming job easier, we made that interface layer look just like Amazon AWS. We were faced with the problem, from the application perspective, of porting the code to Amazon. That interface layer–that emulation, if you will, for university data centers–is what we released as open source, which became Eucalyptus. That’s how we got ourselves into commercialization.

It’s an open-source platform that’s actually architected to support a bunch of different APIs, with flexibility in terms of virtualization, operating systems, and hardware. It implements cloud computing with a sufficiently high-fidelity imitation of Amazon that the weather forecasting code we were working with couldn’t really tell the difference between when it was running at Amazon and when it was running at the universities.

Scott: When I think of things like the Amazon API, there are two things that spring to mind. The first is the API that you access, generally remotely. From my machine here I could spin off a hundred instances and provision them with Amazon machine images.

I could shut them down, I could take snapshots of images or take snapshots of the block source. All of the admin stuff that you can do through the management console is what I think of as the Amazon API.

If I understand what you just said, you guys replicated that API so that if I’ve got code locally that’s spinning up instances and kicking off jobs on Amazon, it could also kick off jobs through, say, a third-party vCloud provider, as long as they’ve bolted on Eucalyptus.

It’s exposing that same API, so I can federate my clouds and do the same logical operations even though, for example, starting an instance on VMware’s infrastructure is different from doing so on Amazon’s. Do I understand correctly that you have built a sort of abstraction layer so that I don’t really see that as a user?

Rich: Yeah, although I think it’s important to understand that that abstraction layer is more than just the API. Really, what Eucalyptus does is implements the cloud abstractions in an essentially technology-neutral way.

There are a set of abstractions that all successful clouds today support, and we’ve figured out how to implement those abstractions independently of the API that accesses them and independent of the virtualization technology that is used to implement them.

Eucalyptus is in the middle, as a generic cloud infrastructure. It is built so that you can manipulate that cloud infrastructure using a number of different APIs, although the one that has been most successful for us is Amazon. That’s the one that we ship with open source, but we’ve done others.

On the bottom part of it, there are lots of different virtualization technologies that can be used to realize those cloud abstractions, both in open source and in proprietary forms. Eucalyptus is built so that it could use any of those virtualization technologies to implement the abstraction.

Scott: So, in other words, you are positioning Eucalyptus to be the lingua franca of the clouds, in terms of starting instances, stopping instances, and that sort of thing.

Rich: Right. There really are three sets of abstractions that most clouds support. There are VM abstractions for starting, stopping, and manipulating virtual machines. There are abstractions having to do with storage, such as what persists, what does not, how it is accessed, and what the range of access is. Finally, there are abstractions having to do with networking: how your VMs are named, how can they communicate with each other internally and externally, and those types of things.

We have an internal framework that is able to combine different virtualization technologies to implement those abstractions and then export them via different APIs.

Scott: When an instance is running on Amazon, the instance itself doesn’t really persist data. So if you shut that instance down, its “local drive” basically gets blown away, whereas with other providers, that may not necessarily be the case. Or you may set up a portion of your data center so the instances are persistent.

How do you deal with those underlying differences that surface up through whatever kind of cloud or virtualization architecture somebody is using internally?

Rich: That’s a great question, and the answer is that the Amazon API, and in fact all the other APIs we’ve looked at, have really well-developed notions of Quality of Service attribution. I’ll just speak about Amazon concretely, because that’s the one that seems to motivating a lot of others.

In Amazon, there is this notion of VM type, and there is also a notion of availability zone. Availability zones in Amazon are there specifically to give you some idea of independent failure probability. Two different VMs working in different availability zones have independent failure probabilities, so you can calculate the likelihood that you’ll lose both of them simply by calculating the joint probability.

We actually co-opt that a little bit and use the notion of availability zones as a collection of common QoS characteristics. One of those could be “is it failure-independent?” but that is not necessarily the only one.

For example, you could have two different availability zones in a Eucalyptus cloud, where the underlying infrastructure is implemented in two completely different ways. And what you want to export to your users is the notion that each specific availability zone has a particular set of QoS attributes.

Therefore, if VM persistence is an attribute that you want to export in one or more of your availability zones, you can do so. Often, there are capacity-allocation issues associated with that. If you’re going to do VM persistence, you have to have more of a persistent store available, and the Eucalyptus administrator is responsible for doing that capacity matching, so that the size of the availability zone, the things that can go in it, the users that are entitled to request image starts from it, and so on and so forth, are properly controlled.

Scott: Does costing show up in the API at all, in terms of VM, storage, and network resources? In other words, is there a way through the API to know what an instance is costing you, or is that something that you need to know in advance?

Rich: That’s actually something we think about a lot. There’s no problem in putting the cost into the API, but it’s difficult in many cases to compute exactly what that cost should be automatically. If you’re charging against a credit card, you can calculate the occupancy cost or the number of bytes or whatever. There’s some monitoring that’s necessary to do that, but you can do it.

On the other hand, if you’re making calculations based, for instance, on the financial costs of running a VM, it’s far more complex. There are factors based on the cost of the hardware, the operating costs, and a specific life span of that hardware in a specific company. It’s a very difficult problem, because of all the factors that go into the cost benefit associated with owning that machine.

Of course, you can use average costs. Rather than calculating the operating cost on a particular machine, you can assume a reasonable rate and feed that to the system as a parameter. It would be no problem to export that through the API, although I’m not sure you get much utility out of calculations based on a static number like that.

On the other hand, Amazon has just announced spot pricing, which changes the equation. Now the price is going to be fluctuating dynamically, and you don’t want to know so much what something cost you in the past, as much as you want to know what it’s going to cost you in the future, and that’s a whole different ball of wax.

I do think that almost all of the APIs at some point will include some notion of cost, but the utility of that is really going to be predicated on how predictive it is. I don’t think the providers have gotten to a point yet where they can reliably produce those kinds of predictions, so I suspect we’re a little ways off before we see it.

Scott: I was thinking more in terms of the customer-facing price showing up in the API, rather than the internal costs. That seems to be a simpler matter, since as you said, there are a million things that factor into internal costs, many of which aren’t really programmatically discernable, such as what your people are costing you and those sorts of things.

The notion of spot pricing leads me to another thought. Today, an IT manager may have a workload that they think may be well suited to the cloud, because their demand is fairly elastic. Maybe the organization needs 100 nodes for just a week, or two hours, or whatever.

Customers only want to pay for peak capacity when they need it, so they’re manually looking at various cloud providers and trying to find a fit for their needs. They may also be considering an internal cloud that bursts to a public cloud as needed.

Of necessity, they’re looking at all these relationships very individually, in terms of the characteristics of each provider and so on.

I think about when web services were just catching on, as the trendy technology of the moment. There was a lot of talk at the time about web services marketplaces and directory services, where if you wanted a weather service, you’d go to a directory service and find one. The stuff you’re talking about seems to lend itself to the evolution of a kind of cloud marketplace, especially if cloud capacity really is a commodity.

In fact, that’s the direction that people like Amazon seem to be pushing it, where there are spot prices for instances, for CPU time, and that sort of thing. It would allow you to monitor a marketplace of many providers and place workloads dynamically based on changing prices. Am I thinking too sci-fi here, or do you actually see this happening?

Rich: I actually agree with everything you said, and I would enhance your description in the following way.

There are business uses for which commodity markets with fluctuating prices make a lot of sense, assuming that the offerings are completely interchangeable. Typically, customers buy the lowest common denominator, because it’s a volume business.

At the same time, there are real advantages in certainty and long-term contracts, and being able to repurpose things you buy based on unforeseen events. Spot pricing can be either a hindrance or a boon, since if the spot price happens to be poor when you really need to move on something, you might have been better off with a longer term fixed price fee for service.

There was a study at UC Berkeley, I believe, of network bandwidth. They tried all sorts of pricing models for people in their homes, and they found that, in fact, spot pricing for network bandwidth at home is a really bad idea. People liked it the least of the various pricing options that were made available to them.

I think there’s going to be a computational commodities market, and I think the key there is going to be interchangeability. You have to know that if you buy it from one provider or another, or if you run it in-house, you really are getting the equivalent service. And in computing that’s tricky, because it’s a multidimensional quality of service bundle, and you have to match on all dimensions. Multidimensional matching is a hard problem, particularly if things are statistical.

It’s kind of hard to do that equivalence relationship, but you can do it, at least to some approximation. And for the things where you can deal with spot pricing and make those kinds of decisions, I think it’s going to work just fine.

I don’t see that as the only way that public cloud providers are going to offer their services, though, and I also think that independent of whether the market is commoditized or not, the on-premise cloud is going to play a role, acting as a hybrid.

Scott: That makes sense relative to the electricity market, too. Governments, agencies, and companies enter into long-term contracts with each other to get determinism around price, but there’s also a spot market for power.

To turn to a different subject area, Eucalyptus lets you get to the notion of arbitrary infrastructure as a service. While each service provider may have its own unique set of value adds, broad availability facilitates federating workloads across providers.

It seems like platform as a service started to evolve almost completely separately, but they seem to be coming together a little bit more now. Google App Engine was probably the first platform as a service provider that a lot people really noticed.

The first option was to go the infrastructure as a service route with Amazon, and then you were your own administrator. You had to figure out what OS to load, patch your own boxes, and so on, but you could basically do whatever you could with a virtual machine. On the other hand, you could go with Google, which required a bit of a rewrite, but there were all kinds of things you didn’t have to worry about.

Now there seems to be work happening toward being able to layer platform-as-a-service frameworks on top of infrastructure-as-a-service offerings. I can envision future where you can sort of mix and match those.

How does Eucalyptus fit in to that world, where you’ve kind of got arbitrary platforms as a service on top of arbitrary infrastructures as service?

Rich: Let me start off by saying that when Eucalyptus was a research project, we teamed with another research group at UCSB that was interested in doing for Google’s App Engine what we did for Eucalyptus, namely building an open source version of GAE. They layered it on top of Eucalyptus, and it’s called AppScale. It’s still a research project today. You can get AppScale and Eucalyptus, run them in your data center, and have GAE and Amazon all to yourself.

From the beginning, we really thought that people would want to mix and match platform services and infrastructure services. I think it will be interesting to see how applications work at different levels in the stack. One of the key questions with regard to this layering is how much of the inter-relationship between the application and its data can be explicitly represented.

With platform as a service, you wind up uploading into the cloud the data you wish to process on and the code you wish to process it with. In infrastructure as a service, you have to upload those two things and you have to upload the environment in which that code is going to run.

I think you can characterize software as a service in the terms that you upload your data, or you upload no code. With platform as a service, you upload your data and you upload your code, but you upload no environment. With infrastructure as a service, you upload the data, the code, and the environment.

Given that schema, what really differentiates platform as a service from infrastructure as a service in terms of the power-benefit trade off is how much you can tease apart your application code and data from the ecosystem of software in which it lives.

That’s a harder job than it might sound. We all think our code lives in isolation, but it’s been a long time since people have written anything of substance that lives completely by itself.

If you want to use platform as a service, you have to get back to that sort of software engineering purity, where the environment in which it runs, including the libraries, operating system version, security policies, and so on are really kind of digested from the code.

Particularly complex, scalable applications consist of multiple components, some of which are perfectly suited to platform as a service deployment, and some of which are not for whatever reason.

Because of that, we think the key is this layering, which allows platform as a service and using application components that make the most sense. If that’s the premise, we hope you use Eucalyptus, but if not, we still think that layering makes a lot of sense.

Scott: I talked to some guys who were doing Google App Engine development, and one of the things they said that was really sort of eye opening to me was that platform as a service is fantastic if you can live inside of that sandbox. So, with Google, for example, everything is a request and a response, and you have a certain amount of time to process the request; otherwise it just basically kills it. But, you get infinite scale, you don’t have to patch your OS, and those kinds of things.

That worked for 80-90 percent of the project they were working on, but for the other parts, they would have Google App Engine call out to other stuff running over on Amazon Web Services, for example, that would do some of the longer-running kinds of things and generate results.

The picture you just painted allows that whole thing to exist inside of that data center. You can have the infrastructure-as-a-service piece and the platform-as-a-service piece talking to each other, relatively seamlessly. Maybe you even run them on the same servers, and to accommodate service peaks, things move around relatively transparently, and from an internal IT perspective, that’s going to be the norm.

Platform as a service is going to be great, infrastructure as a service is going to be great, and they’re going to have to communicate with each other, because sometimes one of them is going to be a better fit than the other, and large complex apps are frankly going to require both to some degree.

Rich: That’s right, and I think you see in Azure that kind of acknowledgment. I know this sounds strange, but when I look at the Azure architecture, I see those platform-as-a-service types of restrictions built in. Still, you could imagine exporting in Azure more infrastructure-as-a-service type functionality, and then that would have to be coordinated in a reasonable way.

I think one advantage of Eucalyptus is that as you were saying, if you want a Ruby platform, a Java platform, a Python platform, or some other platform of your choosing, all of those things can happily coexist over the same infrastructure as a service framework.

Scott: Let’s flip over and talk a little bit about the open source nature of Eucalyptus and its significance.

Rich: There are a couple of ways to view the presence of Eucalyptus as an open source project. The easy one is the motivation for us to do it as open source, which is that we as researchers have really have spent a lot of time working with and benefiting from other open source efforts.

If you are working in the university environment today, doing empirical system science, you are almost assuredly using somebody’s open source tool kit or platform or software infrastructure to do your work.

And so we felt like it was paramount that if we built something that might have utility for other researchers’ investigative activities, we should make it available as an open source. And frankly speaking, before we even thought of commercializing Eucalyptus, we knew we were going to make it open source. Having now gone down this path and having become a commercial entity, we still feel very strongly about this.

Even from a personal experience perspective, we feel that open source has a lot of value that can be passed on to the community, as long as you can support that notion commercially. From a business perspective, I think there are some fairly good reasons to do it as well.

The first is that we get access to a lot of information very quickly about what Eucalyptus does and does not do. That information is freely given to us as people take the software and try things with it. We can see, almost instantly, which things we have done that have been successful, where we need to work on the quality of the system, where we need to work on its stability, what features might be missing, and those kinds of things. The open source community is very good about providing that information.

A second aspect that was a little less clear to us when we started this, but which is actually turning out to be very valuable, is that Eucalyptus is built to run in pretty much whatever environment you have.

We consider Linux environments primarily, but we really spent a lot of time when we were building it initially thinking about university data centers where we don’t know what version of Linux is there, we don’t know what the vintage of the hardware is, we don’t know well how it’s administered, and those kinds of things. As a result, we’ve architected the system to be flexible in this way.

What we’ve learned from the open source community, though, is just how widely varied those environments can be. We’re constantly hearing from people who say, “This is what my data center looks like, and Eucalyptus doesn’t do X, Y, or Z. Please help me.”

That type of interaction is not really feedback about the code or what Eucalyptus does, but rather it’s feedback about how Eucalyptus can be configured. Through that information, we’ve been able to make Eucalyptus extremely configurable, so that it really does allow itself to be molded into whatever environment you want to put it in. And that, I think, is part of its value proposition in a commercial sense that we didn’t anticipate.

Scott: Just to drill down a little bit, remind me what license Eucalyptus is under. Is it an Apache license, BSD, GPL, or something else?

Rich: It’s actually under two licenses, which has to do with the university and commercialization phases we’ve been through. The university license was a BSD-style license, modified in a way that the University of California, Santa Barbara absolutely had to have in order to allow us to release it from the university.

We had to modify that license when we commercialized, because it’s a UCSB-specific license. At that point, we took a look at it, thought about what the rest of the community was doing, and switched to GPL. It’s actually GPLv3 going forward, and as time goes on, less and less of the university code is present in the actual open source code base. At some point, it will be all GPLv3, but for now it’s both.

Scott: That makes sense. In terms of community involvement, you mentioned that you get a lot of feedback from the community about how it’s working in a certain environment, specific issues, and things like that.

With some open source companies, the majority of the people working on the project work for the company. That was sort of the way MySQL was originally structured.

Rich: Right.

Scott: Others really want the community contributing code. Where do you see yourselves on that spectrum, in terms of where the code comes from?

Rich: We’re actually moving away from one extreme. When we started out and looked at taking community contributions, we found that we were rolling the code forward so fast that people were contributing code that was absolutely against our internal code base. And so they would get upset because they put a lot of energy into building code that we couldn’t take.

What we decided was that, in the first phase of the project, we would really limit code contribution to things that talk to the Eucalyptus API, meaning things on top of Eucalyptus or attached to it or communicating through the API, and then to take patches. We would take bug fixes, as long as they were not obsoleted by a bug fix we had internally or whatever.

What has happened is that we were trying as hard as possible to do a complete implementation of AWS while AWS was adding features. As a result, we decided to finish off the initial AWS API for open source, do a set of bug fix releases to re-factor some things internally for performance and stability reasons, and then adopt a much less aggressive release schedule.

Moving to a status where we’re releasing every six to twelve months instead of every six or eight weeks makes it more feasible for us to entertain contributions to the core. It’s important for us that the QA be done well.

We’ve put a lot of energy into making Eucalyptus extremely concurrent, which is a big differentiating factor between Eucalyptus and some other open source offerings. Eucalyptus is a very, very concurrent system, and what we find is that when people add things to it, in order to get stability, they often put locks in, because it’s easy.

That approach gets their thing to work, but at the cost of reducing the concurrency. We probably will look very long and hard at those contributions to make sure that we preserve the quality of what’s there, but I imagine that in the next year or so we’ll see community contributions to the core as well.

Scott: I remember seeing news articles that mentioned, for example, Ubuntu and Eucalyptus in the same paragraph. What are your relationships to distros or other open source projects?

Rich: We’ve always wanted to make Eucalyptus as widely valuable as possible in the open source community. The Ubuntu partnership has been fantastic in that way, as they have really put a great deal of energy into adding things that they believe their distribution community really wants.

They put a lot of energy, for example, into improving the installation experience in a way that is much more compatible with what the Ubuntu community is accustomed to seeing. And we think that’s great.

By the same token, we are open to working with other projects or companies that have different needs. We do work with the Debian community, we’re actively working to package it for Squeeze as part of their open source offering, and we’ve talked with some of the other distros about whether it makes sense for us to be part of the distro or just to remain compatible.

We have agreed to continue to package Eucalyptus for the Linux distros that people ask us to package it for, and we explore ways that we can have closer partnerships. That’s ongoing, but so far, in the distro space, our closest partnership is with Ubuntu.

There is a lot of synergy between what Eucalyptus does and what other projects do. So again, in this platform, there is this sort of layering capacity. We have ongoing compatibility and synergy discussions with a number of different partners.

There is also an ecosystem building up for tooling. RightScale is a good example of a company that provides powerful tools for using Amazon. You can program your Amazon allocation much more effectively through RightScale than you can by hand.

Because we’re Amazon-compatible, RightScale works with Eucalyptus, so that’s a nice partnership. If you want to use the same kind of dashboard to manage your Amazon and Eucalyptus clouds, you can, and there are a number of companies that provide that kind of tooling, backup services, and other things that complement what we do.

We think of those partnerships as being really important, particularly since we’re open source, because we think that the success of cloud computing today is going to come from an active ecosystem and not the success of any one single technology.

Scott: Talk a little bit about your business model. How do you guys keep the lights on around this open-source project?

Rich: We’re practicing what I think is commonly referred to as an open-core model, where we provide the core of the system as open source, and then there are value added components that can be purchased from us under license. The bright line differentiator for us is whether the feature or component is really tailored towards the notion of commerce or not.

If you want to experiment with cloud computing, if you want to understand it better, or if you want to see what it does and what it doesn’t do, then open source Eucalyptus is for you. If you want to make money with it and start a business using Eucalyptus, then there are things we think you need that we feel like you should, in some measure, pay for, because you’re going to go off and resell the value that comes from them.

So, for us, it’s actually pretty easy to see what goes in open source and what doesn’t. We have professional engagements that surround our proprietary products. We also offer support, under contract, for the open source code, but that’s not really our main business. We’re really a software development company, and we’re leveraging the open source community and the open source project for the proprietary products we ship.

Scott: Obviously, people are using Eucalyptus internally for things that are kind of confidential and not public, but what are some of the really cool stories you can talk about where you’ve seen Eucalyptus put into production?

Rich: The best example of where it’s in production is actually something we can talk about. NASA took Eucalyptus and added Luster, which is a high performance shared file system, and some other things, and they’re running it in production at NASA Ames.

They offer cloud services to other agencies within the government, and we know it’s in production, because they call us and they yell at us about it. We think that’s pretty exciting.

The other stories around Eucalyptus are really more usage scenarios. We hear a lot from the open source community about what people are doing with it. We can sort of talk about that, in the sense that they only tell us so much.

There are interesting use cases that have emerged from Eucalyptus. One is the cloud bursting of the disaster recovery model. People think of that one almost immediately. A lot of people are also looking at the idea of augmenting their on-premises infrastructure with public clouds.

But then there’s also this funny phenomenon, where developers become frustrated with the pace at which they can get resources dedicated to their projects, and they pull out their own credit cards and start developing in a public cloud. Often that code is very valuable, and when that comes to life, the company for whom that developer is working often wishes to pull that stuff back into the company and run it on premises until it can be properly vetted, software engineering can be done on it, and so forth.

So, we see this going both ways. We see businesses that want to burst from their on-premises IT into the public clouds, and then we see development which taking place in one venue, such as on the public clouds, and for whatever reason, the people who own the development want to move it back into the private space, perhaps only to move it forward.

The result is a kind of homogenization of the on-premises IT with public utility providers. We think that represents a really exciting future usage scenario for Eucalyptus.

Scott: It’s a slow moving trend, because it’s difficult for companies to get there, but there’s pressure for internal IT organizations to restructure themselves essentially as service-delivery arms.

We’ve talked to people in several companies who have told us, “I have to compete with everything out there. Any business unit that comes to me is free to buy from Amazon, they’re free to buy from Google, they’re free to buy from whoever they want, and I’ve basically got to make them a deal. I’ve got to deliver a better service at lower cost.”

Rich: We hear this idea that internal IT finds itself competing with commodity providers, but I predict that will prove to be a short-term phenomenon. Whenever a commodity emerges, it becomes inexpensive as well as very common. In the technology business, there’s often an advantage in using something that’s more rare, before your competition and before it becomes a commodity.

Where your business has to innovate, that’s really not necessarily a commodity business. I think what we’re going to see in the future is that on-premises IT will really be the engine of innovation, experimenting and coming up with the best ideas before the competition can do them.

And then as those advances commoditize, becoming the sorts of things where you’re going to compete on price and really try to drive the margins down, there will be pressure to move them from your internal infrastructure, out to the utilities, to make space for the next innovation.

I think long-term, that’s really what IT is going to look like. I don’t think it’s all going to be public, and it’s certainly not all going to be private. I think the business model that rationalizes how to partition resources between public and private is going to be driven by that innovation cycle.

Scott: That’s great. Looking at the time, I think that’s a good place to wrap up the interview. Thanks very much

Rich: Thank you.

Bookmark this:

• Background on Hadoop and its origins
• Implementing support for MapReduce to handle distributed data
• Suitability of Hadoop for cloud computing
• The relationship between Cloudera’s Hadoop version and the Apache version
• Cloudera’s business and monetization model around Hadoop
• Licensing models as they apply to Cloudera and Hadoop

Scott Swigart: Could you introduce yourself and Cloudera?

Amr Awadallah: I’m CTO, VP of engineering, and cofounder of Cloudera. I was previously VP of engineering for data systems at Yahoo!. That’s where I first got exposure to Hadoop and recognized the need to form a company to commercialize it. The company is just 13 months old now.

Scott: Can you fill in some of the blanks for our readers who may not be entirely familiar with Hadoop?

Amr: Sure. With traditional data systems, you have a bunch of filers where you store your data. A compute grid is connected to these filers, which essentially is a bunch of computers that fetch over the data, do computations on it, then store the data back.

That model worked fine, but it started to break when data sizes got too big and moving the data between the storage network and the compute resources became a big stress on the network infrastructure.

There was also a big stress on the filer heads. A filer consists of a bunch of disks, and all these disks are attached to a head, which is really what does all the serving. The filer head itself starts to become a bottleneck when you’re running a job that accesses a large proportion of your data. If you’re just searching a file or two within your data, it’s still OK.

Those factors are what drove the need for Hadoop. We had massive amounts of data to process, and existing systems were not capable of scaling sufficiently. Hadoop itself came out of the open source community, though Yahoo! played a very big role, as well.

Scott: I’m always interested in the actual mechanics of how something as complex as Hadoop comes to be. Can you speak to that specifically?

Amr: Two developers named Doug Cutting and Mike Cafarella were trying to build a web-scale, distributed search engine. They built a system called Nutch about seven years ago.

Obviously, the system needed to be massively scalable to index the whole web, so they were spending a lot of energy in that area. Around that time, Google published two key papers, on Google MapReduce and the Google File System, GFS. It’s the marriage of those two technologies that enables the large scale data processing that we can do today.

GFS allows linking a lot of commodity hardware together to present one large, unified file system. MapReduce has two parts: the programming model and the job execution framework.

The MapReduce job execution framework lets you spread your job over multiple nodes, so that the first level of mapping, where you’re reading your data in and doing the first level of aggregation, is all local. All the communication is purely local between the computation resources and local disks. Therefore, you don’t take a performance hit or stress the network from passing data over it.

In the second phase, which is called the reduce phase, some inter-node communication takes place, which does necessarily stress the network infrastructure to some degree. Usually by that stage, though, the data has already been reduced in size to the subsets that you care about, so the stress on the network infrastructure is dramatically reduced.

When Doug and Mike saw these papers, they had a “Eureka” moment and started to integrate these technologies into their web-scale search technology with Nutch.

Scott: How did the work at Yahoo! coincide with what Doug and Mike were doing with Nutch? I assume that Yahoo! must have adopted an open source approach in place of some of their proprietary algorithms. Is that true, and what went on behind the scenes with those decisions?

Amr: At the time, Yahoo! was also doing a lot of work internally on their infrastructure for serving search. As the web grew exponentially, the cost of serving a search was getting prohibitively high, and Yahoo! needed a more scalable system to cost-effectively build the web index.

They saw the MapReduce and GFS papers from Google as well, and in light of the work being done by then on the open source Hadoop project, they recognized that they had a choice. They could either build proprietary MapReduce and GFS technology from scratch, or they could adopt Hadoop internally and support the open source project.

A very big debate went on inside Yahoo! about the right direction to go. The proponents of the open source approach made the very good argument that supporting Hadoop would take away one of the key advantages that Google had. Namely, only Google had the systems to really take advantage of MapReduce and GFS, which locked in a tremendous advantage for them.

They argued successfully that it was in Yahoo!’s best interest to unlock that technology, so other startups could start adopting it and building ideas around it. The idea was that this approach would shift the power from Google.

In response to those arguments and others like them, Yahoo! hired Doug Cutting on staff and invested very heavily on building a large Hadoop team. In fact, the vast majority of the code base in Hadoop today came from Yahoo!, so they certainly deserve a lot of credit for that.

Scott: How did Hadoop move beyond its foundations in search?

Amr: After Yahoo! started to integrate the technology within its web infrastructure initially, Hadoop started penetrating a lot of other areas of Yahoo! as well. Now it’s being used in Yahoo! mail for spam filtering, as well as on the Yahoo! front page to model which stories to display to which users based on their interests. It’s being used for ad modeling and ad targeting, as well. Facebook was another early adopter and supporter of Hadoop.

Around that time, which was maybe two years ago, it was starting to be clear that Hadoop had implications for more than just the big web properties. The technology is useful to many other domains as well. In fact, this notion of moving computation to run as close as possible to the data is something that everybody should be doing, given how big data is growing these days.

This trend is being reinforced by the fact that commodity servers are becoming very powerful in terms of the number of cores and the number of discs you can have in them. Now you can have eight cores very economically, and within the next year you’re going to have 32 or even 64. You can have anywhere between four terabytes to even 48 terabytes of disk space on one of these standard server nodes.

It has become very clear that the trend is toward large hardware topologies, where systems will need to be very resilient to failure. They will need to know how to shift workloads and data around when things fail, so the system just heals itself.

Scott: One of the things that this model depends on is moving away from a traditional relational database that doesn’t really fragment the data across lots of nodes. Instead, those traditional systems are really based around the idea of a single, monolithic body of data.

That seems to be one of the big paradigm shifts for developers. When they start using something like BigTable where the data is distributed across a lot of nodes, they have to think a little differently about the way they build applications.

Working our way from the bottom up, talk a little bit about thinking differently in how you store your data so that you can use a MapReduce process on top of that.

Amr: First, it’s worth making the important clarifying point that Hadoop is not a database. Hadoop is a data processing system, and in fact, I would even go as far as saying Hadoop is an operating system. The core of an operating system boils down to a file system, the storage of files, and a process scheduling system that runs applications on top of these files.

There are many other components that help with devices, credentials and user access, and so on, but that is the core. Hadoop is exactly the same thing. The core of Hadoop is the Hadoop Distributed File System, which is a file system that’s runs across many nodes. It links together the file systems on many local nodes to make them into one big file system. Hadoop MapReduce is really the job scheduling system that takes care of scheduling jobs on top of all those nodes.

That is the key distinction between Hadoop’s approach and that of database systems. Hadoop, at its heart, does not require any structure to your data. You can just upload files directly from anywhere, like a web server, RFID device, or cell phone mobile device, directly into Hadoop.

They could be images, videos, or just a bunch of bits. They don’t have to have a schema with column types and so on, which gives you tremendous agility and flexibility.

Hadoop has a very nice model that I sometimes refer to as schema on read. Whereas defining your schema as you’re writing the data in limits what you can put in by requiring it to be conformant to the schema that you created, Hadoop allows you to define the schema as you’re reading stuff out.

That gives you a lot of flexibility and agility, since you can add files that have dynamic parts like JSON or new standards coming up like Avro, which is a very good project coming out of the Hadoop project that’s similar to protocol buffers from Google and Thrift from Facebook. Avro makes files have a schema around them as well, but these schemas are semi-structured, rather than conforming to a strict relational model.

That said, it’s also important to point out that structured stuff is a subset of unstructured stuff. The fact that Hadoop at its heart is a file system doesn’t mean that it can’t do database relational stuff. It does actually, in the same way that Windows at its heart is a file system, but you can run SQL Server on top of it to get the relational services, schemas, column types, and so on.

One of the key projects on top of Hadoop is Hive, which actually came out of Facebook. Hive essentially provides a relational database on top of Hadoop that utilizes the underlying file system but has a metastore that keeps the schema of the files.

It knows that a given file is tab delimited or whatever, it knows the column type for these files, and Hive allows you to write SQL against these files. It will look up the schema and then it will write for you the MapReduce jobs so that you don’t have to go and learn MapReduce from scratch.

Now you have the flexibility of going either way. One approach is to get at the core of the MapReduce framework using Java MapReduce, which we sometimes refer to as being like assembly language for Hadoop. It gives you the most flexibility and performance, but it is fairly complex and difficult to learn.

Alternately, you can go in with a high level language like Hive. In this case, you can just use SQL, if that’s what you’re used to, to write your job. Hive itself has lots of optimizations. It understands the underlying MapReduce framework, so it can properly map your problem on top of your data.

Scott: Obviously, search is a logical use of MapReduce, but I’m sure there are many others. For example, in the oil and gas industry, a single oil derrick might produce terabytes of data a day. In other industries, there might be huge amounts of financial data to plow through.

It seems that every industry is faced with figuring out ways to deal with these large bodies of data. Given that, what are some of the more interesting uses or problems you’ve seen thrown at Hadoop?

Amr: The group at Yahoo! that I came from was using Hadoop for data analytics and data warehousing. We had something like 100,000 web servers across the world, and once we collected data from across all these servers, we dumped it into Hadoop, which became the place where we stored all of the data, instead of traditional network storage.

Our reasoning for doing that was a matter of economics, given the quantity of hardware. Hadoop lets us scalably process that data, clean it up, and normalize it so we could pass it along to the systems that need it.

Hadoop is getting very wide adoption in the data warehousing and business intelligence domains. One of the biggest uses within Yahoo! right now is dealing with all of the log information from servers. Analyzing that information allows for better spam filtering, ad targeting, content targeting, A/B testing for new features, et cetera.

It’s not web-specific. For example, everybody does data warehousing, and we see very strong adoption there.

Separate from that, your example of oil companies is a very good one, as is the financial sector. Right now, we do have a couple of very large financial institutions working with us on these exact problems, taking huge amounts of data from domains like credit card processing and building predictive models for fraud that enable better decisions, for example, about whether to block or allow a given transaction.

In the stock market, Hadoop is being used to do simulations that help predict option pricing and related problems. That’s another very healthy market that we’ve seen growth in.

Another interesting market is genomics, specifically for doing gene sequence mapping and protein sequence alignment. In many ways, that is a fuzzy string matching problem, where you have a string of DNA that you need to match to a template string that has certain diseases identified in it.

Any individual task is not that big, but there are so many of them that you need to split the work over many nodes and do the matching in parallel. Government agencies have a number of implementations that involve massive bodies of data as well.

Scott: Let me turn our line of conversation in a slightly different direction. Cloudera obviously has the word “cloud” in it, and implementations like Amazon Web Services have Hadoop implementations.

Hadoop seems to intersect with the cloud fairly well. When I think of the cloud, I think about having an API to start up and shut down nodes, utility computing, and being able to provision and shut down relatively large numbers of nodes remotely, programmatically.

What makes the cloud and Hadoop a good fit for each other?

Amr: I completely agree with you that the cloud is essentially a set of resources that you can scale up or down with a nice API, based on dynamic demand. There are many services that fit that description, of course, and I should add that a cloud is not always external to your organization.

Even though Cloudera has the word “cloud” prominently in our name, we are not building a cloud or a cloud infrastructure like Amazon AWS or Microsoft Azure. Rather, what we’re building is software that enables cloud computing. Hadoop is at the heart of that, although we’re also building other components around Hadoop that strengthen that kind of operating proposition.

Hadoop fits very well with the cloud model, because in many ways, Hadoop is like a virtual machine, albeit one that actually takes the inverse approach to what one generally associates with virtual machines.

That is, a virtual machine hypervisor allows you to partition a single server to look like many small servers. On the other hand, Hadoop is about linking together a bunch of physical servers to look like one big server, which starts looking like a cloud service.

You can have many users using this resource, and the resource can be scaled up or down, depending on the demand it’s being hit with. That’s exactly the sweet spot for Hadoop.

We’re working very closely with Amazon, and their Amazon Elastic MapReduce service gives you the option to use the Cloudera specific distribution of Hadoop, although Amazon has the generic Apache distribution of Hadoop as well. If you want to get a maintenance support contract, you obviously need to use our distribution.

Amazon’s business is about providing the cloud service itself, and not about building Hadoop. Our business is about building Hadoop itself, so that’s the synergy there.

Scott: As you mentioned, there is an Apache version of Hadoop and then there’s the Cloudera version. As different companies wrap themselves around different open source projects, they’re structured in different ways. Talk a little bit about Cloudera and what you add to the public open source version of Hadoop, in terms of additional software, support, or services.

Amr: I should start by saying that Cloudera is an enterprise software company. Open source is an enabler for us, and it’s part of what we do, but our mission is about building enterprise software for large-scale data processing in internal or external clouds.

In terms of how we go about working with the community, Hadoop is an Apache licensed product, and the Apache license is one of the most consumer-friendly licenses out there.

The Apache license allows anybody to take the code and do whatever they want to do with it, whether that means using it internally, redistributing it, even packaging and selling it. That flexibility makes it a very nice license, but of course, it means that you have to add value in order to make money from the code.

The way we make money is a bit similar to Red Hat, although Red Hat does charge for the distribution, whereas we don’t. You can get our distribution directly from our site without having to pay us anything.

There are many differences between our distribution and the Apache distribution. Primarily, our distribution is well-packaged as RPMs, or RPM packages for Red Hat, or Deb packages for Ubuntu or Debian. That makes it much easier to install using standard tools, and in fact, we are talking right now with Microsoft about how we can package it properly to run on top of the Microsoft Windows platform.

The other distinction is that the Apache organization can be a bit slow in terms of launching new Hadoop releases. We release new distributions almost every two weeks, with the latest and greatest features.

Of course, we release them as test distributions, and we tell people to be careful how they use them. As with other fast-moving distributions, the bleeding-edge stuff can have problems as well. Yahoo! also has a distribution, which is essentially the Apache Hadoop code that has been tested on the Yahoo! internal infrastructure. Everything that Yahoo! has in their Hadoop distribution makes it to the Apache Hadoop release, and similarly for any contributions that Cloudera makes to our distribution. We aren’t forking off the code base, it’s just a matter of the Apache Hadoop official release lagging behind a bit.

Scott: What is the ongoing relationship between Yahoo! and Cloudera?

Amr: As I said before, Hadoop owes a lot to Yahoo!, in terms of how they made it into what it is today, and they continue to contribute a lot. Yahoo! has a very big Hadoop team–I’d say almost 100 people right now. They have lots of open positions as well, for new people to join that team.

A lot of the Hadoop patches that we pick up were developed and tested by Yahoo!, and that’s what we include in our distribution. We do our own development and testing as well, in addition to what Yahoo! does. We listen to our customers to decide on which features/improvements to start working on next. Yahoo’s development roadmap is obviously a function of their internal needs.

Scott: How does Cloudera monetize their work with Hadoop?

Amr: Right now, the only reason you would need to pay us is if you need our maintenance service. Think about software where there’s no upfront licensing fee but there is a maintenance fee. The maintenance fee buys you access to our team, and you can open tickets and ask questions.

Companies using Hadoop in a production environment need a partner they can fall back on when they have problems, and we are that partner. We can also provide advice about what network hardware to use, how to properly balance the number of hard disks and cores on your machines, and how to configure Hadoop itself.

Hadoop has something like 200 different system parameters that you need to properly configure depending on the infrastructure you have and the type of job that you’re running. Writing code using Java MapReduce is a bit of an art right now, and we can help with that to make sure your code is properly written with respect to the MapReduce model.

These are the kinds of services that we offer. We also offer quick patches. If you encounter a bug specific to your problem, and you want to make sure you get the patch for it right away so you can continue operating your cluster, we’ll fix the patch for you very quickly. We have a number of Apache Hadoop committers on staff.

I forgot to mention that Doug Cutting, the inventor of Hadoop, left Yahoo! in September and joined our team. Tom White is another key committer we have at Cloudera. He is also the author of the O’Reilly “Hadoop: The Definitive Guide” book, which is the “bible” for Hadoop right now.

The other thing we do is training, in terms of learning this new technology, how to work with it, and how to develop problems within this new scalable way of thinking. We provide training from the very basic, fundamental kind of stuff all the way to writing algorithms specific to the problem that you are trying to solve.

That’s how a lot of open source companies start, where their main revenue stream is maintenance fees; this allows them to make healthy revenues in their early years.

Most open source companies evolve to an open core strategy, where the core platform is free, but functionality around the core is not necessarily free. That is the direction that we will be moving in.

We’ll keep the Hadoop core free, because that’s Apache, and we’ll continue contributing to that. Indeed, more than half our engineering team is focused on making the Hadoop platform better, but at the same time, we’re now investing in building infrastructure around Hadoop to strengthen the platform.

Scott: It’s interesting to consider that dynamic among companies that are wrapped around open source projects like Joomla, WordPress, or MySQL. Every one is different, even with their similarities.

It’s interesting to me to understand how different companies are structured, because it seems like early on, there were lots and lots of different models people were trying around open source. It seems like the market seems to be settling out a little bit.

One common approach is a sort of aquarium model, like Google or MySQL, where it’s being written by people who all work for a company, and they put the source code out there for all to see. There may be a patch from outside the company here or there, but most of the development is being done by people who all work for the same company.

On the other hand, there’s the example of the Red Hat model, where there’s a pretty vibrant community and the product draws code from a lot of different places. They do a lot of value-added stuff around it, such as providing an enterprise-grade version, fast patching, training, and consulting services.

These two seem to be the predominant models that at the moment. Do you see it in a similar way, or do you see other ways companies are structuring around open source?

Amr: I completely agree. The Apache Software Foundation follows the latter model. In fact, they only allow projects to be “official projects” within Apache if the committers in the project are diversified over a number of companies.

If just one company controls the full code base, Apache will not let it become a core project, which I think is a very healthy approach. It does lead to certain logistics and coordination problems, but it allows the software to evolve much more quickly.

It also has sort of a Darwinian advantage, in the sense that when a number of companies are all working on a problem and there is more than one solution, the community can just take the best one. That said, you do lose control a bit, which might slow things down, even though it helps foster innovation, so it is certainly a balancing act.

Scott: Maybe a year ago, we interviewed Justin Erenkrantz of the Apache Foundation, and that was one of the things we discussed–that requirement for a project to have maintainers from at least three different companies to make it out of the incubator and become an official Apache project.

In another area, the Apache license is very different than the GPL license, for example. How does that fit into the strategy of Yahoo! or Cloudera? Do you think the Apache license versus the GPL license affects the evolution of Hadoop?

Amr: It does play a very big role. As I said earlier, the GPL license has a viral aspect in terms of distribution. Whenever you redistribute the code, you are giving up some rights back to the owners of the code. On the other hand, the Apache license doesn’t have that viral nature to it, which makes it a much friendlier license, in terms of adoption.

We have to work harder to make our distribution, training, and maintenance services really stand out, and to provide additional value on top of the vanilla Apache Hadoop codebase. After all, customers can just go and get the code directly from Apache, without involving us at all.

Scott: There are obviously differing viewpoints on licensing, with some people who really like the GPL for what it does, and some people who really like the Apache license for what it doesn’t do, or what it doesn’t make you do. People argue about which is really the most free.

Amr: That’s a good point, and of course, I have my own opinions about it. Our CEO, Mike Olson, led the open source revolution from the beginning. Mike was the CEO for Sleepycat, which commercialized BerkeleyDB, one of the first open source companies.

Scott: We are running out of time, so I don’t want to miss the chance to ask you whether there’s anything you’d like to bring up that we haven’t discussed yet.

Amr: Hadoop has been around for a few years, but it’s still very young, relatively speaking. There is still a lot of maturing for it to do, and that’s what we are spending most of our energy on right now. We, with the Apache Hadoop community, are doing some great stuff in areas like performance, usability, availability, security, auditing, interoperability, and so on.

We are also building new data system products that augment the Hadoop platform. For example, a big problem that many companies have today is how to reliably collect data from across hundreds or thousands of servers, and bring it into a central data repository. That’s a problem that we are very excited about.

Scott: Cool. We’ll keep paying attention to what comes next. Thanks for taking the time to talk today.

Amr: Thank you.

Bookmark this:

• The role of the Software Freedom Law Center in the world of free software
• Fostering voluntary compliance with the GPL
• Handling violations without putting people on the defensive
• How communities protect themselves with self-regulation
• Negotiating the urge for proprietary “secret sauce” in the marketplace
• Niche markets where open source has not had much impact

Scott Swigart: Could you take a minute to introduce yourself and tell us a little bit about what you do, as it relates to open source and free software?

James Vasile: Sure. Primarily, I work for the Software Freedom Law Center, doing legal work for people that make and support free software.

My client list includes the GPLed content management systems, like Joomla and Drupal. I go to work for WordPress as well, but I also end up doing work with organizations like GNOME, Samba, and KDE.

Just about any large free software project is going to come through the Law Center at some point. We try to give them whatever help we can, and that help varies quite a bit from client to client.

The legal challenges that free software projects face are sometimes unique, and sometimes mundane. The unique challenges are often about making sure that these projects can continue to operate as free software projects, in a world that is occasionally hostile to freedom and open source software.

More mundane legal issues also arise, because it turns out that running a medium or large free software project is an awful lot like running a small business. The main difference is that nobody gets paid. But you have a lot of the same legal issues trying to get out there and distribute your software all over the world and manage the wealth of resources that it takes to make world class software.

The other side of my work is that I help free software projects with managerial stuff, trying to help them make decisions that put them on the best footing for managing their project, creating new strategies, and getting the word out.

It turns out that we at the Law Center talk to more people in the free software world than almost anyone else. That gives us a pretty good point of view to tell them how a given path has or has not worked well in the past.

We offer that experience to people not as a legal service, but just as an ear to listen, and we offer advice where we can. In that capacity, I sit on the board of Open Source Matters and on the advisory board of GNOME. We give a lot of off-the-cuff advice to software projects.

Scott: We’ve talked to a lot of people about the different ways of building open source software, whether it’s corporate built or community built. We have also had a lot of conversations about choosing between different licenses, such as deciding between a GPL license versus an Apache license, or something like that.

One aspect of how I view your organization is as an entity involved with enforcing the GPL. Is that an accurate characterization? And what are some of the issues that people bring to you, and how do you help resolve those issues?

James: We do a lot of the GPL enforcement, at least in America, but that’s definitely not the majority of what we do. Even though we do a certain amount of enforcement, most notably for BusyBox, and Samba, I wouldn’t say that we think of ourselves as primarily GPL enforcers.

More often, we’re trying to help people to find reasons to comply. I think the GPLed CMS world is a really good example of that, because like most of the rest of the GPLed world and the free software world, enforcement is a non-starter. I mean, the vast majority of licenses are respected, because people voluntarily respect them.

Most people know that if they violate a license, the chances of them actually getting rapped on the knuckles for it, or even called out on it, are pretty low. People comply to avoid the risk of the embarrassment at having to retract some code, but more importantly, there are social norms at play that help them decide this is the right thing to do. There are structures that incentivize compliance that have nothing to do with lawsuits or lawyers.

For example, we just had a really good discussion about this in the WordPress community. It was recently decided that WordPress themes have to comply with the GPL.

Themes are extensions to WordPress that include everything from images, to CSS files, to JavaScript, to PHP code. Some of these components are reached by the GPL, which is why themes, to some degree, have to comply with the GPL.

I won’t say that this was a new development in the WordPress community, but there were a lot of theme distributors and theme publishers in the WordPress community that were not compliant with the GPL. This is a shift in strategy for WordPress, in the sense that they’ve decided they want to encourage people to comply.

Obviously, this can be controversial when people are making a living selling themes, but most of the people who decided to come under compliance were fully aware that WordPress was never going to launch a bunch of lawsuits trying to get people to comply. Rather, the reason to comply was to get along well with the rest of the community, rather than thumbing their nose at the project that is producing their livelihood.

More to the point, there are reputational benefits to complying with the GPL in some of these communities. If you work with the project, as opposed to against it, the project is happy to link to you, for example, include you in their extensions directory, their theme directory, and that kind of thing.

There are some practical benefits to complying, but for the most part, it has nothing to do with a legal analysis. Our work in GPL enforcement is not really about trying to whack people upside the head. Sometimes we do that, but for the most part, that’s the option of last resort.

Scott: A lot of times, projects grow up kind of isolated, and then they might reach critical mass, like WordPress or Drupal. On the way to becoming practically household words in the developer and IT community, I imagine that they go a learning process as they decide what kind of a cultural identity they will have.

I could see it being a fairly seismic shift in something like the WordPress community for them to decide that enforcement is important to them. That seems to fit with the notion that the goal is not to file a bunch of lawsuits, but rather to create the conditions where complying with the GPL is the logical thing to do.

As you’ve worked with different projects, have you observed a set of conditions that need to be present for compliance with the GPL to become self-enforced, in a matter of speaking?

James: Its actually extremely varied. Every project has its own culture associated with it and its own values, even. The things that different projects think are important is hugely variable, and so there is no mental checklist.

It really just comes down to looking at a project and the structures and values they have in place, as well as their history. Based on that, one can make a case to people that the license they’re using should be respected for the good of the community.

We point out the ways in which the community benefits as a whole and the way the individuals benefit within that community. Eventually people start talking to each other about how much they care about licensing.

A really good example of that is in the Joomla world, because Joomla started out as a project called Mambo, and Mambo was released by software company in Australia under the GPL. A bunch of people flocked to the product and started making changes to it. The company said, “Thank you very much for the changes, but now we’re going to close the code and release it under proprietary licenses, and basically take this code private.”

It was the GPL that protected them from having this happen, protected their investment in the code, and kept the code free. Even then, though, many people within the Mambo community did not think the GPL was that important, and they did not have a community norm of respecting the GPL.

They sort of treated the code base as quasi-open for most people and then occasionally closed when they needed it to be closed. Certainly, they weren’t requiring third-party extension developers to comply with the GPL.

The shift from that world to the current world where the license is a big part of every conversation about what to do with the project has been one of pointing out to them how much the protections of the GPL mattered in the past. That becomes a reason for maintaining those protections in the future.

But obviously with a project like WordPress, which has never had a crisis of who can access the code base and who can make derivative works of it, that sort of argument isn’t going to be available. So every project, having its own history and culture, is going to have its own reasons for caring about and wanting to respect the GPL.

Scott: I would imagine that it can be an interesting challenge if someone starts out small with a great theme, starts welding more functionality into it, and eventually starts to sell a significant amount. In a case like that, how do you go about making that initial communication?

I picture it being somewhat like music downloads. In that case, of course, the industry decided they needed to bring it under control, so they picked a bunch of people and said, “We’re going to go after you,” and then they created a culture. You are fortunate that you don’t need to take an aggressive approach like that, but how do you educate people who may not even really suspect that they are violating a license?

James: The initial contact can be very difficult, because we need to convey to people that they’re probably violating a license. That rings alarm bells for anybody, especially someone running a small business.

We want to communicate that to them, but in a non-threatening manner. The challenge is to reach them without making them sort of shut down, call their lawyers, and never talk to you again. We want to engage them to talk about ways they can come in from out of the cold and come into compliance.

Having that conversation with people and letting them know, “I’m not here to knock you about, I’m not here to extract money from you, and I’m not here to change anything you’re doing except for this one tiny thing of you agreeing to freely license your product, ” is very difficult, because people have a great lot of fear about what is going to happen if they comply.

Scott: Sure. The small business owner probably hasn’t ever talked to a lawyer except when they originally set up their articles of incorporation, if they even have done that. How do you typically play it out from there?

James: Sometimes we do it well, and I’m the first to admit that sometimes that interaction doesn’t necessarily go well. Usually by the time a case gets to me, the project has already contacted this person, so the discussion has happened outside of my view and certainly outside of my control.

Even though we never know what that is going to look like, obviously there are best practices here. It typically works best to send an initial polite email saying, “It looks like you might be violating. We should have a discussion about that. We’re not here to ask you for money, we’re not here to give you bad publicity, and we’re not here to do anything but discuss ways that we can convince you to come into compliance.”

I think that phrasing of “We would like to have the opportunity to convince you,” is probably the most important aspect, because it can help people realize that force isn’t really what’s on the table. After all, Word Press isn’t going to waste its precious resources suing a bunch of theme designers.

Scott: What’s it like for you educate the projects themselves on how to engage? I imagine that a project owner who has invested their heart and soul in building an Open Source project might be less than diplomatic in approaching someone they feel has violated the license. How do you help them be constructive?

Conversely, how do you educate something like a three-man design team that it’s in their best interest to rewire how they’re going to market? How do you educate them about the fact that doing so will ultimately benefit them?

James: When projects make the first contact, the reason it can be a little bit dicey is that sometimes the people involved truly feel wronged. When they contact the people who seem to be violating the license, some of that unhappiness often shows through as impatience and negativity.

That’s probably not the most auspicious beginning, especially since the theme designer probably doesn’t agree with that position. From the very first sentence of the email, they tend to have diverging viewpoints, and the conversation can go awry.

Projects get better at it just by doing it more often, either in tandem with my office or just by thinking about it a lot more. They can generally benefit from consulting with us before they make the effort and running the emails past us for the first couple times they do it.

Honestly, though, once they’ve done it a couple of times, anyone who regularly does compliance work gets into a rhythm about what the techniques are for getting people to actually talk to you as opposed to shutting you out.

It really is just a question of experience and reminding yourself that the person on the other end of that email has a very different point of you than you do. That’s how we address the initial contact, but what we actually say to people to convince them to comply in the theme context has a lot to with introducing them to theme designers that are complying and making money.

It turns out that the way people find themes is not very GPL aware. In the same way that designers don’t care much about the GPL, customers don’t care much about the GPL. When people grab a theme off the Internet, whether it’s GPLed or not, it’s probably available from both its authoritative official source and some FTP server or bit torrent somewhere.

And the GPL is not going to change that fact. In the same way that Joomla is not going to run around wasting money suing theme designers, theme designers aren’t going to run around wasting money suing FTP sites.

That means that the legal rights they are giving up by GPLing their themes are rights that they were probably never going to exercise in the first place. It turns out that the protections that theme designers hold onto are not worth a whole lot, and giving them up does not cost them anything. GPL compliance is a zero cost proposition in 99.9% of cases.

In the theme and extension world, that makes the decision a lot easier, and it helps people realize that they can comply with the community standards without it costing them a dime. Most people truly want to comply, and they don’t want to be in violation of the standards of the community in which they operate.

Sean Campbell: In view of the fact that most people want to follow the rules, what do you think the projects themselves can do to facilitate that? What should they be putting on their own sites to help educate people before they download source and get started building with it?

Do you think they should be doing more educating?

James: I think that, in the same way that GPL compliance is less about the license and more about the community, open source is less about the openness of the source and more about the community.

The openness of the source code creates great communities, and I think when you’re talking to potential clients, customers, downloaders, and users, the thing to lead with is not the fact that they can look at the source code, or that there is a license that they probably don’t care much about anyway.

Projects should explain to them all the benefits of the community, and why they would want to download software that has a community behind it as opposed to just a company behind it.

They should also talk about what it means to join that community. They should explain that extending the software or building on top of it happens in this community context, where there are certain norms.

I think that’s probably the best approach, because that gives people what they really want to know, which focuses on the social context, rather than legal obligations. It also helps them understand what is so great about the world of free software, which is that we are a collection of people who provide support in this mode for the software that we build.

Scott: You said something very interesting about the idea that following the GPL is essentially a zero-cost proposition. Most people getting paid to write code are working for companies that may not have a strong participatory connection to open source, and that has been even more true looking back some number of years.

In companies large and small, I think there is the perception that releasing their code under the GPL could compromise their business position. They are often concerned that they would be ceding control of their product.

That position, of course, isn’t supported by something like MySQL. No one is going to take over MySQL, because the company building it has all of this expertise around that code base, as well as enhancing it, supporting it, maintaining it, and providing services around it. And when you get down to consumer-level products, that’s even more true. My mother isn’t going to want to go search for the source code. She’ll pay $15 for a theme because it will just work.

In light of that, it was fascinating to me the way you phrased it, that 99% of the time there is no cost to using the GPL. The fear people have that if they go the GPL route, they’re out of business because they’re forced to basically give products away instead of selling them isn’t really well-founded in most cases.

That’s a message that I don’t see out there a lot. What success stories can you point to, where people have gone the GPL route and it didn’t cost them anything, and in fact may have given them benefits they would otherwise not have had?

James: I think you give some good examples. Nobody is going to buy MySQL from anybody except MySQL, at least not currently. The people within MySQL over at Sun have concentrated all the expertise in deploying MySQL and using it. It’s possible that competitors could spring up, but on top of having the expertise in deploying it, the people at MySQL and Sun have the apparatus to support it long term.

It turns out that the people who actually make the software have a huge competitive advantage over everybody else in monetizing that software. No reputable company is going to take the cut-rate deal and compromise on the quality when what they really want is the best support they can get for a reasonable price.

As the people who make the software, you are the best positioned to do other things related to it, and so you become the best source when it comes to servicing that software.

That’s the big reason that it’s a zero-cost proposition. At the end of the day, no matter who might want to go into competition with you using the GPL rights you give them, you are going to out-compete them every day of the week.

The only business they would take from you is business that you wouldn’t get anyway, because you would be pricing too high for that customer. Anyone who could afford to pay your rates would pay your rates, because they want the best that they can get, and you are the best.

Therefore, by definition, anyone who is making GPL-ed software and selling it is in the best possible market position to monetize that software, no matter what the business model ends up being. That’s definitely true both at the high end and at the consumer end for things like games, which are a highly proprietary market where very few companies allow free redistribution of their games.

It turns out that even though you don’t allow free redistribution of your games, there is free redistribution of your games. There is rampant pirating of gaming software for both console games and PC games, and despite that the PC game manufacturers, the game manufacturers continue to sell lots and lots of copies of their games.

The reason for that is that at the low end, at the retail end of things, just like you were saying, your mother doesn’t necessarily want to troll through FTP and BitTorrent sites to try to find a theme or to download the source code for the theme and try to figure out how to install it.

People want one-click convenience and easy, supported deployment of software, and companies that GPL their software are able to do that just as well as companies that don’t. In fact, sometimes they can do it better.

Whatever the mechanism by which you are making money and being “the source” for this software, none of that changes when you go GPL. We used to say that business models might have to be adjusted for people that sell GPL software or distribute GPL software, but it turns out that that’s not really the case. In most cases, you make money off your proprietary software in the exact same ways you make money off your GPL software, and we’ve seen that time and time again.

I think you’re right that we need to start putting up more examples of that and start showing people the examples of companies that have been successfully GPLing software and people that are successfully servicing GPL software that they make.

That’s probably the biggest thing we could do to alleviate the fear that people have, which is that once they GPL it, the door is open and they can never go back, and all their profits will disappear.

Scott: What is the corresponding message in the most extreme cases, where somebody feels like they have an algorithm that really is their competitive differentiator. I’m thinking of something like a gaming engine, a compression algorithm, or a market-analysis algorithm where they know they’re doing things that nobody else is doing.

I guess, what’s your message to those companies that are convinced–perhaps correctly–that they’ve got a relatively small amount of code that gives them a huge competitive differentiation?

James: First, in most cases they’re probably wrong, because most of the time, people who cook their own algorithms are not going to beat the combined wisdom of everyone else trying to solve the same problem cooperatively.

That said, it might be true that in some cases, a private algorithm could beat whatever the current state of the art is. Still, they’re probably not going to beat it for long, and moreover if what you’ve done is truly novel, there are other mechanisms for protecting the patents and whatnot that come into play.

Most of those people are probably mistaken about their unique competitive advantage, and a lot of what’s left would probably benefit from having everybody work to produce a better model or algorithm, to combine the advantages of many different services.

If you have five different companies, each with their own private competitive advantage, imagine what they could do if they got together and gave everybody the benefits of all their advantages in one package.

I think that there are still a lot of reasons to go the free software route, even at that point. At the same time, your expertise matters, and your awesome algorithm is best used by you. Even if you free it and let other people improve upon it, you are in the best position to use those improvements. You still have the competitive advantage there and it’s quite a large one.

Imagine if Google were to release the source code behind Gmail. Nobody else on the planet would be able to deploy Gmail, because none of them have access to the technology that Google has, meaning the entire stack that is so uniquely developed for them and their unique purposes.

I’m sure Gmail connects to tons and tons of applications and pieces of infrastructure that don’t exist anywhere but within Google, so even if Google were to hand me the source code to Gmail tomorrow, it would be almost no good to me at all.

Scott: We see that sort of effect with the company I believe is called Aquea that’s wrapped around Drupal. Likewise, Cloud Air is wrapped around Hudu. They would have to do really bad by the project for a long time before somebody could fork it and get the community of users to move over to them.

It sort of happened with Netscape and Firefox. You can see examples, and you mentioned Mambo becoming Joomla.

James: Even in the case of Mambo and Joomla, when the company tried to close source Mambo, that wasn’t even enough to cause a fork. It was a couple years later that the fork happened, after the community tried very hard for a long time to work with this company. And the company tried hard to work with the community as well, and finally it wasn’t working to anyone’s satisfaction, so there was a fork.

Experience shows us that every time people are in a situation where forking might be good, they choose not to do it, because it’s tremendously costly to try to split a community in half and figure out who’s going to do what.

Scott: You mentioned games as being an area that’s very closed and proprietary. Search is another of those areas, as are financial analysis, oil and gas, seismic analysis, and others.

Are you seeing any areas where there is starting to become a real credible GPL-licensed option that there is a lot of community effort around that may be a game changer a year or two or three from now in some of these areas that have traditionally been very dominated by closed source?

James: I wish I could say that I did see a lot of these imminent game changes attracting large communities, but unfortunately I haven’t. The thing about free software currently is that it usually starts with somebody scratching an itch or somebody trying to solve a fairly widespread problem.

In these niche markets where there isn’t a lot of free software adherence already, there aren’t a lot of coders sitting around directly going to benefit from a problem being solved in that area, and it breaks down pretty quickly. One place we see that quite clearly is in the small business application market.

Think about medical billing, for example. Medical offices spend tens of thousands of dollars on software that is by any measure extremely rudimentary, and all it does is help them manage appointments and then submit invoices to insurance companies.

That software is ripe for being knocked off by a free software project, and yet I’m not aware of anybody that has cobbled together more than one or two programmers committed to working on it, giving it any significant amount of attention or time.

It turns out that the free software community has been relatively bad at supporting these niche areas. There are business models that would allow easy entry into these areas and make it possible for free software developers to make the software and make a good living supporting it, and yet still nobody is taking that opportunity.

That failure is one of the things that I’m curious about, and one that I’m seeking solutions to over time.

Scott: This has been a great conversation. One of the things we usually do at the end is to ask whether there’s anything interesting that we didn’t ask about.

James: I think the most interesting thing about the work that I do is really the community-building aspect of it. I touched on that earlier when I said that it’s not about the legal structures or the legal norms, but rather about the social and community norms.

That’s true across the board, and one of the things that I see on project after project is people confusing the value of what they do as license-based as opposed to community-based.

You see people with religious allegiances to certain licenses or even people denigrating other licenses or worrying about who’s using what license in other projects and things like that. What they miss is that the value of the code they’re writing isn’t just based on what license it’s under.

Scott: Thank you very much for taking the time to chat with us.

James: Thank you.

Bookmark this:

• Joomla and its relationship to website owners
• The difficulty of creating success metrics for free software
• The relationship between the core Joomla environment and extensions
• Cloud computing and platform independence
• Transitioning the web to new platforms and the semantic web
• The future of rich applications in a web browser

Scott Swigart: To start, could you introduce yourself and tell us a bit about your relationship to Joomla?

Louis Landry: Sure. I first used the predecessor to Joomla, Mambo, in 2001 or 2002. I was a computer science student at the time, and I was building a website for my friend. I sort of fell in love with the interface, and honestly, I was shocked to see some of the things that they had pulled off on the web.

Over time, I continued to build more and more sites and get more and more involved. At the time that Joomla became Joomla, I was in contact with several of the key members who started a fork to do some development patches and things into the core.

After I was asked to join the core team, I eventually served as project manager, communications team leader, foundation working group coordinator, development coordinator, and pretty much every aspect of the project. I am now development coordinator, working on future research and development, infrastructure, and things like that.

Scott: Talk a little bit about Joomla itself, the strengths of its architecture, and so on.

Louis: Joomla’s an open source content management system, designed to help you build and maintain your web presence. It doesn’t take the same approach as some of the older, larger-scaled content managers like SharePoint do.

It’s not designed to be a full-on enterprise thing. That doesn’t mean it can’t be used in enterprises, but it’s more of a presentation and content organization system than some of the more document-centric systems. Our main target market has been websites for small businesses and other small organizations.

The last reasonable numbers we saw showed maybe one or two million websites out there running Joomla. Most of those are individuals or small organizations putting up sites to share a voice related to some things they care about.

Joomla itself is really a platform with something like 4,000 extensions. It’s much more of a product based system than other applications like Drupal, where all of the extensions tend to be much more tightly integrated with the core system. With Joomla, you find a lot more third-party developers building product-based things. It’s much more packaged up, and you just sort of upload a Zip file and everything installs itself.

Scott: How is the Joomla organization structured? How do you make money, or do you?

Louis: The non-profit organization that holds assets for the Joomla project is called Open Source Matters. It’s incorporated in New York, and it holds the trademarks, copyrights, and those sorts of things. Most of their income comes from Google Ads, I believe.

It’s not really a money-making corporation. I am an owner in a company called JXtended, which sells third-party components, extensions to Joomla, and that sort of pays my bills, along with consulting work that I do around Joomla. That tends to be the trend with people around the Joomla project, if they’re making money here.

Scott: That makes sense. I would imagine that a lot of people are looking for certain functionality, and they may not even know that Joomla’s the solution. Then a consultant can come in and relatively quickly have a site up that meets their needs and just so happens to be based on Joomla.

Louis: That’s actually the case more often than not. People will hire a firm to put together a website for them, and they don’t really realize that Joomla is the platform on which it’s being built. A very large proportion of the people on our support forums and engaging in the Joomla project are small web design shops that are pumping out websites for different clients.

We’ve had more than one case of a client seeing the Joomla offline screen and thinking that we somehow hacked their site. Something went wrong with their Mind Skill server, and when the little screen comes up and says, “This site’s offline,” and it’s got the Joomla logo, they’ll email us and say, “Why did you hack my site?”

[laughter]

Then we have to explain to them, “No, really that’s not what happened.” So it’s funny, but it’s happened quite a few times. That’s indicative of the fact that a lot of people using Joomla don’t really think of it in those terms, since the site was put together for them, and Joomla is transparent to them.

Scott: That’s an interesting aspect of that space. There are obviously a lot of people who need a web presence but are not technical.

You’ve also mentioned the really rich ecosystem, for lack of a better term, around Joomla: 4,000 plug ins, consultants, developers, web design firms, and so on. One to two million sites must rank as a highly successful open source project.

Louis: It’s a pretty sizeable chunk. I think since July, we’ve had three million downloads of Joomla off of our sites. I’d have to double-check the numbers, but I think in the last two years, it’s in the range of 14 or 15 million. It’s been pretty astonishing to see the downloads crank up.

That’s also just the English version. There are plenty of versions that are distributed by our language partners or translation partners all over the world in something like 30 or 35 languages.

Scott: That’s an interesting difference between success metrics with proprietary software as opposed to open source. With proprietary solutions, you can point to an exact number of licenses sold, whereas with open source projects, you often have to rely on the number of downloads. That makes it somewhat more difficult to get an accurate picture of adoption.

Louis: That’s true. It’s difficult to gauge the ratio of deployed sites versus the number of download packages, so it’s really hard for us to know what the actual market share is.

There were some numbers at one point from a company that I believe was called Netcraft, and some people in our community have built crawlers to try to figure out market penetration. It’s been interesting to see these various numbers, and they vary widely.

Scott: It kind of goes against the ethos of open source, but open source projects could easily embed something in them that by default makes the thing phone home, when it’s up and running.

Louis: There’s been a lot of talk about that over the years. A few open source projects have tried it, but those efforts tend to backfire, because a lot of users don’t like it.

We’ve talked about the concept of anonymous usage statistics and those sorts of things several times, but it’s just never happened. I am actually not sure it would be a good idea, simply on the basis of the fact that I have no idea what sort of traffic that would generate for the server collecting the data.

Scott: Right; a little Google App Engine app could just shove the statistics into a big table data store.

Louis: Yeah, let Google suffer. [laughs]

It’s something that we’ve talked about, like having some sort of optional switch upon install, but even there you run into trouble, because one of the biggest ways Joomla is installed is via cPanel and Fantastico, which are server scripts to auto install web applications.

They don’t even run though the Joomla installer, so we wouldn’t see it. That’s another example of how our download statistics would be completely irrelevant, because that doesn’t get downloaded through us–it gets deployed through a completely separate system.

Scott: On the topic of the cPanel and Fantastico installers, talk a little bit about how Joomla updates. There’s always a tension between a really vibrant ecosystem of plug ins and the necessity of keeping the main project moving forward.

Louis: That’s been a real area of balance for us as well. Anyone who follows Joomla knows that we don’t release very often, as far as feature incremental releases, and that’s not always by design. Version 1.5 came out, I think, two years after 1.0, and 1.6 is set to come out probably toward the end of this year, which is about two years or so, as well.

It moves rather slowly, but the wealth of functionality created through the third-party extensions allows us to do that. And because it’s almost entirely volunteer driven, and we’ve been going through growing pains since this is still a relatively new project, we’re still trying to find our footing in a lot of different areas.

We don’t really have an update mechanism in the core quite yet, although that is something that is being actively worked on for Version 1.6. As far as extensions are concerned, there isn’t a real updating mechanism for that either.

Historically, it’s been either a case of uninstall and install the new version, or else install a new version on top of it. With PHP, it’s generally as simple as FTP-ing files up, if you want to do it manually. It has not been painless, but it has also not been a huge problem for us.

Scott: I imagine that the 4,000 or so plug ins are in various different states at any given time, from having been abandoned to having multiple people working on them full time, and everything in between. It’s kind of like when you upgrade WordPress, you just kind of cross you fingers and push the button, and hope all your plug ins work afterwards.

Louis: [laughs] That’s the thing with the slower release cycle. We put maintenance releases out to fix bugs and things like that pretty regularly, but those updates don’t change things in such a way that any of the third-party stuff would break, really.

There have been a couple of cases where we’ve had to change things, but we got out in front of it pretty well, and it didn’t really cause a whole lot of issues. Still, it’s true that when you’ve got a large third-party developer community building external extensions, there’s an awful lot of inertia.

If we go in and change a system in the core that’s going to have a cascade effect on how everybody uses the API, or the systems, or something like that, that’s a lot of ramp up. What we saw with our huge changes in 1.0 to 1.5 was that it took a while for the third-party development community to catch up.

In a lot of cases, they did abandon things, and other people picked it up. That’s part of the beauty of the open source community, right? Somebody says “Oh, well I’m not going to bother with this,” and somebody else comes behind them and says “Oh, I’ll do it.”

Scott: Or, in some cases, there are just a lot of options. When something gets abandoned, there may be four other projects that do something pretty similar.

Louis: That’s certainly true. For example, in just the category on our extensions directory for adding comments to articles, there are pages of different options. Some are commercial, some are not, they have different features, and so on. The richness of choice is really great.

Scott: That brings up an interesting point. I’ve talked to people in corporate environments who are considering using open source software about their methodology for evaluating the safety of a certain project.

They will look at all those stats to see if it’s being actively maintained. They will see if the contributions are just coming from a few people, or from a lot of people. They often have a specific set of criteria they use to judge whether they want to make a bet on it.

Do you have any advice for people as they look at those several pages of options for similar functionality? Obviously, the one that shows up at the top of the list isn’t always the safest choice.

Louis: That’s a great note. I find that this is one of the areas where the commercial side of open source really flourishes.

If I were going out to build somebody’s website, and I was presented with a commercial choice and a non-commercial one where they both met my needs, I would probably choose the commercial one and pay for it. That would give me higher confidence that somebody would be backing it.

That’s not to say that I don’t appreciate non-commercial open source software, because obviously I do. Still, in a mission-critical situation, that’s likely the path I would take. That said, I also tend to look at how active the people are who are running the project.

If I’m looking at those same two options, and the commercial people are nowhere to be found but the people running the non-commercial project are very active, then that tells me a lot. I feel better about their commitment to the platform, as well as their knowledge of how everything works.

I know that they’re more able to integrate certain things. They’d probably be better suited to finding answers to problems, if I ran across them, and that sort of thing.

I think there’s a balance to be made there. I really like to be able to know that whoever I’m getting my software from is active in the open-source community.

Scott: What’s the Windows story for Joomla? Is it common to run it on Windows? Is it one of those situations where some people do development on Windows but they pretty much always deploy on Linux?

Louis: I think that since the majority of Internet servers are Linux, it’s probably safe to say that the majority of Joomla installs are on Linux. And since the majority of end users use Windows, it’s probably safe to say that most people that test it and build up sites and all those sort of things are doing it on Windows.

There are certainly people using Joomla on Windows servers. We officially support Apache, and we don’t support IIS. We know it generally works on IIS, but there have been some issues. We don’t have access to those servers and the software on a reliable basis to catch all of the problems that come up.

We’re actually in talks with Microsoft now to figure out ways in which we can advance our Windows support and make sure that we have the tools that we need so that everybody can more officially assert that Joomla works in a Windows environment.

Scott: What’s in that for you? Do you feel like that will open up a significant new population to becoming Joomla users?

Louis: I’m sure it does, even though on a personal level, I don’t really care that much because I don’t use Windows. I’m not one of those hate Microsoft guys, but I just use Mac OS and Ubuntu.

Still, our mission is to provide a flexible platform for digital publishing online. And what matters there is that I would like it to be available for as many platforms, OSs, and databases as possible.

Whatever we can do to make it easier for people to deploy Joomla, within reason, is something that we will want to pursue.

Scott: Cloud computing is obviously very fashionable at the moment. What interest have you seen around running Joomla on a cloud provider? Is there any work or any thinking about enhancing Joomla to take advantage of some of the attributes of the cloud, like being able to dynamically scale up and scale down and that kind of stuff?

Louis: There are a lot of people in the Joomla sphere looking into playing with the cloud, and there are a few companies I know of in the Joomla community who are actively building things on the cloud and trying to provide integrations there.

The EC2 thing at Amazon is interesting to me, and I’m also very interested in the Eucalyptus project that Ubuntu is starting with, I think, this next version of Ubuntu server. I’ve been talking with some friends in the FreeBSD project about cloud solutions and what a path might be, as well as a bunch of other places.

I think one of the things that I will focus my research on for the next generation of Joomla will be ways we can restructure Joomla from an architectural standpoint to better live in that space and be able to really reap the benefits of the cloud.

Scott: You mentioned Eucalyptus. I’ve heard that name, but I don’t know much about it. Can you talk about it?

Louis: I don’t know an awful lot of the technical details behind it, but Eucalyptus is an open source project that Ubuntu is adopting as sort of their cloud platform. It’s got a bunch of functions, such as the ability to do node requisitioning and various things, and I believe that it’s the software that would tie the servers in the cloud together.

Scott: What I’ve heard, if it’s the same thing, is that it also exposes an API that’s compatible with Amazon’s API. So even though you’re not running on Amazon’s infrastructure, I could write code that could spin up and down instances over in Amazon, but I could point that same code at something Eucalyptus-based and it would do effectively the same thing, even though the whole underlying virtual infrastructure is completely different.

Louis: That’s certainly an aspect of it, and that’s one of the reasons why it was really interesting to me. I also am really excited about the stuff I’m hearing coming out of the Ubuntu server crowd.

Scott: Talk a little bit about that, because it’s not something I’m aware of. In the hosting space, I think of CentOS as being really popular.

Louis: It certainly is. CentOS and Red Hat are probably the standard bearers. Ubuntu’s appeal really stems from being what I chose for my desktop OS. I like the way the file system and the configuration stuff are structured, I like the toolkits, and I like a lot of the things that Ubuntu is doing across that space.

The fact that they’re starting to focus on the server is interesting to me, and I really like that they’re thinking about cloud and moving into that direction, because it looks like it’s one of the focuses that that whole community is taking with the server distribution.

Scott: One aspect of that space that really stands out to me, and that I can’t really figure out, is that one of the earliest cloud providers was Google App Engine, and when it came out, there was a lot of excitement around it. And then people were kind of turned off because they didn’t want to be limited to Python, and then they added support for Java.

It seems like the most obvious thing for Google to support would be PHP, and the second they came out with Java support, some guy ran off and started a project that I can’t think of the name of at the moment, but it lets you run PHP on top of Java.

So this guy said, ” I’m going to port WordPress to run on top of PHP, on top of Java, on top of Google App Engine, using their BigTable, as kind of a science experiment.” He ran off and did it, and it works, but obviously, it’s probably not the stack you really want to build on with something real.

When I look at that, I think that if they supported PHP relatively quickly, a lot of PHP projects–WordPress and maybe Joomla and Drupal–would probably be supporting some version that could use a BigTable data store and run on Google App Engine.

Louis: I’m actually not certain that that’s Google’s vision of what App Engine is about. They’re certainly not a PHP shop in any way. I’ve got several friends that work at Google, and I’ve never heard of them using PHP in anything, unlike Yahoo!, which is a “PHP shop.” They do a lot of PHP stuff.

Google is a Python shop. Yahoo! hired Rasmus, Google hired the Python guys, and that’s the direction they took. I think when they launched it, as is typical for Google, it was closed. You had to be invited to be able to get into the App Engine experience.

I don’t think they were certain about their ability to scale it out if adoption got too high, and the ubiquity of PHP and Drupal and Joomla and WordPress might actually have been a negative for them. I’m not certain that signing on to host all those apps would really be to their benefit.

I would almost think that part of their rationale there was wanting to raise awareness around Python and maybe get some clever people working on Python to see who those people are. Then they could collaborate with them, or hire them, or at least get a better feel for what’s going on in the Python community and expand upon it.

Scott: To turn a bit toward the future, you mentioned that cloud is an area that you guys are thinking about. How does Joomla fit in there? How does it play on this new computing paradigm? What else do you feel are the big opportunities on the roadmap for some deep thinking and important work?

Louis: The obvious thought is the semantic web, which everybody is thinking about. I feel that we in the open source CMS community can play an important role in terms of providing new ways by which information can be organized.

I think the semantic web is a place where a relatively small amount of work could have a very large impact. Still, that focus is difficult, in the sense that I’m really more focused within the Joomla community on the next generation, not necessary iterative versions that are coming out.

I’m trying to rethink the architecture. Joomla as it exists now is based on the same principal concepts as Mambo in 2000, namely being centered around software designed to run on a $6 a month hosting account on some offshore server.

The concept around one site, one instance is still a very serious bread and butter piece of the market for us, but I also want to be thinking about ways in which you can scale out multiple instances running the same software. That ability will position us to fall much more inline with cloud and enterprise deployment schemes.

I want to think about how we can make data more portable, so that you can transport things across instances or just have protocols to push in and pull out data, as well as incorporating the ideas of the semantic web.

I also think a lot about this concept of the rich Internet, like Adobe Air and various other technologies like the high-powered JavaScript engines that are starting to evolve thanks to Google.

It would be really incredible to bring out that rich Internet environment and rich interfaces to make it easier to build a more interactive experience. Those are the things that I have been thinking about.

Scott: This has been a great interview. Do you have any closing thoughts from your end?

Louis: I think we’re really just growing accustomed to this whole new phase of the web, and I’m looking forward to seeing where it evolves. I don’t think that in five years it’s going to look at all how it looks today, and I don’t think the experiences are going to be the same.

I’m really excited to be a part of how this whole thing evolves, including what Microsoft and Google are working on with their Web OS initiatives. It could really change the ways people think about their computers and connectivity. I find all of these things converging and I can’t wait to see what happens.

Scott: I remember one guy saying basically that, until you can run Quake in a web browser, Google is not done. [laughs]

Louis: Actually, you can. Google released an experimental thing called Native Client. It’s not for public consumption yet, but you can compile Windows apps or whatever into this native client code, and then it will run in a browser with a plug in.

So it’s kind of like a virtual machine. It’s interesting what they are doing with that, and I’m very curious to see things like the plug in that they have built enable OpenGL in the browser via JavaScript. I’m curious to see how they integrate that into future stuff.

Their example is quick. You can click in the browser…

Scott: It sounds almost like an ActiveX plug in.

Louis: I think it probably is very similar to that concept, and hopefully they can find a way to make it more secure than that.

Scott: That’s one approach: running a whole different something in the browser, which was kind of the Flash and the SilverLight approach.

Louis: We would much more like to see it work in JavaScript, though.

Scott: It seems like there is also an awful lot of work going that route. These are just things that are mainstream use cases for HTML and JavaScript, and they’re very fast. It’s just a whole new set of scenarios.

Louis: The ability to access the machine-level function-called things like that within the browser would be just intense, but also intensively dangerous. I’m very curious about how we are going to resolve that, because at some point it is going to have to get resolved.

You’re going to have to be able to access the file system and things like that, and I just don’t know how we are going to do it.

Scott: That seems like a good place to wrap up. Thanks for taking the time to talk today.

Louis: Thank you.

Bookmark this:

• How development on the Linux kernel has changed in the past several years
• Drawing in the larger community with the Linux Driver Project
• The breadth of uses for and contributors to the kernel
• The future trajectory of Linux, including functional balance with hardware
• Overcoming the challenges of maintaining universal hardware device support
• The range of change and growth in the kernel

Scott Swigart: Can you please introduce yourself and your association with Linux?

Greg Kroah-Hartman: Sure. I’m a Novell fellow, and I’m working on the Linux kernel with the SUSE Labs division at Novell.

I’ve been doing Linux kernel work for more than 10 years now. I’m a maintainer of several different subsystems, including USB, the driver core, and some other minor things. I’m also responsible for releasing the stable Linux kernels. I used to be a maintainer of a lot of other driver subsystems and such over the years, but I’ve thankfully handed them off to other people.

Scott: Let me start with a really big-picture question, since you’ve been involved in Linux for a long time. Obviously, there’s still lots of stuff happening on the mailing list, but fundamentally, the way code makes it into the kernel hasn’t really changed, and yet the scale of development is much larger than it was, say, five years ago.

How have you seen Linux kernel development change to handle the scale of the effort that’s going on today?

Greg: I’ve been tracking this for over five years now, and we’ve gotten a lot better. There are a lot more people doing the work.

To give an example, for the 2.5 to 2.6 kernel development series, which took about two years, the top 30 people did 80 percent of the work. Now, the top 30 people do 30 percent of the work. The sheer number of developers has also increased. We were running a couple hundred developers, and now we’re running a couple thousand.

Scott: How does that get managed? Are there more layers in the hierarchy?

Greg: We’ve invented some tools to help us out a lot. It used to be that Linus was the only one who could commit anything, or who would accept anything. It was all done through email.

And then BitKeeper was developed, and a number of us started using that. It was very nice, because we had a much faster feedback loop with Linus. I could have him pull 50 patches, see that he pulled them, and then I could hit him with another 50 patches, if I wanted to, within a day. Before, we had a lag of a couple of weeks.

Then, when we couldn’t use BitKeeper anymore, Linus wrote Git, and then that just increased it even more, because everybody can use Git. Our development model increased its capacity, and part of that was that we started trusting more people. Now we have subsystem maintainers and such.

As I said, I maintain the subsystems such as USB, and I have people who I trust enough that if they send me a patch, I’ll take it, no questions asked. Because the most important thing is I know that they will still be around in case there’s a problem with it. [laughs]

And then I send stuff off to Linus. So, Linus trusts 10 to 15 people, and I trust 10 to 15 people. And I’m one of the subsystem maintainers. So, it’s a big, giant web of trust helping this go on.

We’ve developed some procedures that help guide how and when we do merges and releases. And we’ve a very regular release schedule: every three months, we do a release. We have a two-week merge window, and all those patches in that merge window have to have been tested. We’ve gotten the development process down really well over the past four years.

We’re also increasing the rate of change in our development. The same amount of work one of the top 10 developers did last year wouldn’t have even made it into the top 20 this year. Our individual developers have got the work flow down, so we can actually contribute more, to an extent that’s amazing.

Scott: I understand that one of the things you’re involved with is the Linux Driver Project. It seems like there was a certain tension between Linux and certain hardware vendors, in the sense that, for a long time, hardware vendors had been really used to not shipping driver code.

They were used to shipping for Windows, and they didn’t have to be involved with drivers. They just provided an installer. And then, as Linux started to become a lot more popular in the server space and make some inroads in the desktop space, they wanted access to that market.

There was kind of a culture shift in a lot of companies to realize how to provide things like drivers in a way that matched with what the Linux community expected. Take the example of some hypothetical Acme hardware vendor that wants to submit a driver but has never done it before. What does the process look like?

Greg: First, people need to realize that our driver model is different than other operating systems, because all our drivers ship with the kernel. The license requires our drivers to be open, so everything is in the main kernel tree.

Because of our huge rate of change, they pretty much have to be in the kernel tree. Otherwise, keeping a driver outside the kernel is technically a very difficult thing to do, because our internal kernel APIs change very, very rapidly.

In other operating systems, APIs change more slowly, since their development process is comparatively very slow. So, it makes sense, from a technical standpoint and just to save money, to get your code into the kernel. It is maintained that way, and when changes happen within the kernel, your driver code will be fixed for you. If I have to change an API, I change it everywhere.

That approach also lets us see commonality. If see that a driver’s doing the same thing as another driver, I might merge them together in the shared core. That way, everybody benefits. The new driver gets smaller and easier to maintain.

That’s happened a lot. We’ve merged a lot of drivers from a lot of different companies, and everybody’s happy about that. The companies are happy, the users are happy with the smaller code base, and it’s easier to maintain.

The Linux Driver Project started out because of the perception that Linux doesn’t support many devices. It turns out that Linux supports all devices out there. There’s really nothing manufactured today that Linux doesn’t support, in a major consumer market. There are some one-offs, like some small video-capture devices I know of that we don’t have support for, but people are actually working on those.

The initial goal of the Linux Driver Project was to remove all barriers that could possibly be there, which were mostly managerial. To attain that goal, we said that we will write and maintain any driver for free for any company.

It turned out that not very many companies really needed that. A few niche markets needed it, and we’re writing some drivers for them, but most companies had existing drivers floating around inside their company. Their challenge was typically that they needed to get it into the kernel tree, and they didn’t know how.

So, the Linux Driver Project, over the past couple years, has just been a big educational thing. I work with the companies and show them how to get the code in. I maintain the code, massage it and get it cleaned up, and then merge it into the kernel tree. That’s been the majority of the work we’ve been doing over the past two years.

We’ve also now codified a few ways that we can play in the kernel. We can accept code into the kernel that doesn’t meet our normal standards into a staging area. Everybody can clean it up there, and then the code graduates into the real stuff. That’s worked out really well.

Scott: That makes good sense, because someone coming in from the outside isn’t necessarily going to know everything from coding standards to the best practices for doing things the right way.

And so what you’re saying is that you’ve got a place to get that code in, where people who know the right way to do things can look for things that can be factored out of drivers, drivers that can be merged, and that kind of stuff.

It doesn’t force every hardware vendor or everybody who needs to get a driver into the kernel to be a Linux driver development expert. Do I have that right?

Greg: Yeah, although we do have lots of documentation now. We have free books on how to write Linux drivers, and we have documentation of things like our coding style and how to submit code. If people don’t know where it is, we’ll point them in the right direction. In case anyone’s curious, it all starts in a file called HOWTO in the kernel, so they can start there.

Even though we do have those education tools, though, we will also work with companies that want it. We have lots of people who have done this work before and who will do it for you.

It’s a great place for people wanting to get involved in the kernel to start out, because they can just run a script to find some errors and go fix them. We have people that have started off just wanting to get involved who have ended up maintaining whole drivers and subsystems over time.

Still, in the end, my goal is actually to work with engineers from those companies and have them maintain and own it, so they become full-fledged members of the kernel community. That’s the only way we’re going to grow, and it’s working. The number of companies involved in the kernel has grown year over year.

Companies want to get the most value out of Linux, so I counsel them that they should drive the development of their driver and of Linux as a whole in the direction that they think makes the most sense. If they rely on Linux and feel that Linux is going to be part of their business, I think they should become involved so they can help change it for the better.

Scott: That makes sense. In a different area, I saw somewhere that around half of the Linux code that’s being contributed nowadays is driver code. Do I have that right?

Greg: That’s the percentage. Like I said, everything’s in the kernel itself. We’re at something like six or seven million lines of code, and over 50 percent of those lines of code are drivers.

I think 30 percent is architecture-specific stuff, for things like processors and networking. The core kernel is like five percent of the overall code. Those numbers have stayed pretty much the same over the past four or five years.

We change something like 5,000 lines a day, which is just scary. Fifty percent of that change will be in the drivers, and five percent will be in the core kernel. In other words, the kernel is being modified everywhere at that rate of change.

Scott: Do I understand correctly that the core kernel is memory management, process scheduling, and those kinds of fundamental things that an OS has to do?

Greg: Exactly. Basic system calls, memory management, scheduling, and inter-process communications.

And the only reason we’re changing is because people want that change. It’s not like we’re sitting there and going, “Hey! Let’s rewrite the scheduler again!”

[laughter]

We’re doing this stuff because we have to in order to survive, because people want it and because people need it. We do it for fun, but we don’t gratuitously change things. A big part of what drives that change is that what Linux is being used for is evolving. We’re the only operating system in something like 85 percent of the world’s top 500 supercomputers today, and we’re also in the number-one-selling phone for the past year, by quantity.

It’s the same exact kernel, and the same exact code base, which is pretty amazing. Nobody’s ever created something like this before. And we’re doing it in a way that doesn’t follow the traditional software design methodologies, which is fun.

I go and talk at a lot of colleges, and they’re changing their education system based on how the kernel has changed development models for large-scale systems.

Scott: Like you mentioned, the kernel runs on some of the smallest devices, as well as some of the largest. What makes that work is a lot of stuff that is or isn’t included, whether you’ve got fairly bare-metal, embedded systems, or whether you’ve got desktops and music players, or whether you’ve got systems that are optimized for number crunching.

The kernel is an interesting thing. When you talk to people who aren’t really in the Linux space, they often don’t make a big distinction between the Linux kernel and a Linux distro. There’s a huge distinction though, right? The kernel is a fairly small piece of a distro.

Greg: Sure, but consider Android, which threw away everything from Linux except the kernel, and they built something totally new on top of it. That’s a great proof point that the Linux kernel itself has to be really, really flexible to let people do something like that. It still meets the needs of a very big market, which is pretty funny to watch. From an engineering standpoint it was a pretty neat hack.

Scott: Give us some sense of the kinds of people who are contributing to the Linux kernel. Obviously, there are hardware vendors. The distros out there are contributing–Red Hat and Novell, for instance, are big contributors. Give me a sense of the categories of contributors working on the kernel.

Greg: You’ve got a pretty good short list there. Some distros contribute; some do not, and the same is true of hardware companies. Intel has come on like crazy this past couple of years, and they’re now one of the major contributors to Linux. Consulting groups and educational institutions play a big role as well.

There are also a lot of companies that just modify code for one-off little things, making support better in their hardware. Linux runs on yahts for a lot of automatic pilot steering things, and we’ve had some contributions from there.

More than five or ten percent of the US’s power is generated on turbines that are controlled by systems based on Linux, and those guys contribute some changes they need. It’s also interesting that 20-25 percent of all of our contributions by quantity are done by people that have no affiliation with anything.

Scott: The proverbial guy in his basement or dorm room or whatever.

Greg: Yeah. They needed something fixed, so they did it and moved on. The quantity there is still large, but on the other hand, 75 to 80 percent is done by people who are getting paid to do it.

Scott: What do you feel like is driving change? You mentioned Intel, and Linux seems kind of tailor made for a chip company, because it provides a great opportunity to put new things in a processor or a chip set. Contributing to Linux lets them build support for those kinds of things and to see how they are behaving in a real production operating system.

I can kind of see that new hardware architectures and capabilities drive a certain amount of change in the Linux kernel. I can just see the creation of new peripherals driving a certain amount of change. What other types of change drive the Linux kernel forward?

Greg: Well, those two are major. New hardware accounts for the majority of our changes. We get more processors with more and more cores. We had to do a lot of work to make, 8-, 16-, and 32-way machines run really well, and now we’re running 8000 processor machines really well. The scalability is there, and I actually know of people who have booted larger ones, but we’re not allowed to talk about it.

USB 3.0 was sponsored by Intel, and it was shown on Linux first–the first implementation of any operating system. Intel gets their hardware working well fast, and working with the Linux community lets them get it out to developers and other people who are making devices fast. The fact that they can do it for Linux much faster than they can for other operating systems is a huge driving factor for what we need to do for things like 10 Gigabit Ethernet or 40 Gigabit Ethernet.

Some of those speeds have required major changes to the way we do networking and the way we handle processes or data flowing through the kernel. We also have to accommodate workload issues, like real-time workloads, which required big changes to the scheduler and the way we do locking and the way we do other things merged in.

Note that those are all requirements driven by people who are using Linux. The real-time guys want to use operating systems like Linux for their stuff. It also drives Wall Street and the Chicago Mercantile Exchange and the Tokyo Exchange and the London Stock Market.

Scott: What do you see on the horizon, in terms of hardware use cases that aren’t quite mainstream or available yet but that are going to drive some additional kind of significant change all the way down to the level of the kernel?

Greg: 40 Gigabit Ethernet is going to need some changes, and the networking guys are actually working on that already. There are also going to be more and more cores, better real time operation, and USB 3.0. The USB 3.0 devices aren’t out yet, so once we start seeing more of those, we’ll start seeing optimizing things there. SSDs are great; they’re growing fast.

People seem to forget that changes they want to the kernel that it seems like no one else will want very often end up being valuable to a lot of people. For example, maybe the embedded group makes some power management changes, and it turns out that the server guys really want that, too. Then the server guys want better scalability, and it turns out that the embedded guys love those scalability changes, because they’re running dual-core ARM systems now.

Part of what’s helped us make things work is that it turns out that a lot of changes benefit everybody. We’ve become such a flexible operating system by requiring changes to work for everything, and it’s made everything work even better.

There are lots of power management changes in the pipeline to be reworked that help address a lot of the issues the embedded guys have come up with that servers traditionally don’t have. In servers, you can’t shut off that many discrete components, whereas in embedded, you can pretty much shut off every single discrete component in your system, depending how much you want to do with it.

Scott: I remember when we talked to the OLPC guys, one of their requirements was to be able to go to sleep, but not lose the network connection.

Greg: Mesh networking, right?

Scott: Yes. And they also needed super-aggressive power management, because they wanted to get the most battery life possible, because power is such a big issue in developing countries. If I remember correctly, they were actually putting the processor in a sleep state between keystrokes.

Greg: That’s right; they were, but it turns out that the hardware guys can do it better. If you look at more modern processors, the hardware is putting itself to sleep faster. In modern PCs and these new netbooks, the OS really doesn’t have control of the power management stuff anymore, because the hardware can do it better.

It turns out it’s better to run as fast as possible to get your work done, then go to sleep, than to run your CPU at medium speed. The hardware guys realize this, and they know they can handle their workloads better than the operating system can.

Scott: I don’t want to mischaracterize it, and I am intentionally overstating this to some extent, but when you talk to people at Intel, they always say, “Well, that should be in silicon.” It’s kind of like there’s this drive to put more and more and more in silicon, but at the same time, obviously software is growing at a geometric rate.

Talk a little bit about that balance between what ought to be in hardware and what ought to be in software, as well as the notion that something that ought to be in one place today should maybe be in the other a year or two from now.

Greg: The hardware guys are always like, “Oh, I can solve that in hardware. Let me do that. You software guys aren’t ever going to get around to doing this.” And part of that is true.

If you look at the software adoption rates of some operating systems, it’s very, very slow. So, the fact that they make a change to the hardware, and they can save power today in the new processor, means that overall, the amount of power consumption in the world goes down.

People run old versions of Linux and other operating systems very frequently, so there’s sometimes an advantage to solving a problem in both places. They can add it to the hardware, and then they also add it to the software.

In the kernel, we know we typically can’t change user space applications, so we have to make changes in order to accommodate how they want to do things, even if we feel that they’ve done it the wrong way. So, there’s also a tension there.

Some companies actually talk to the engineers really, really well. Intel and IBM are very good at this. They sit down, and they talk to the kernel people once a year. “Hey, this is what we’re thinking of doing. How should we do this? How are we going to implement this?” And they get feedback from us directly. And that’s turned out to be a very, very valuable feedback loop.

It often works out very well in terms of how chips are designed, and how the operating system will take advantage of that. Hopefully, we’re all talking to all the right people, and everybody’s interacting well.

Scott: Let me spin back to something you said very early on, which is that basically that all hardware is supported in the Linux kernel, even though there seems to be the perception that not all the hardware works.

Certainly, one of the things that gets brought up is that if I buy a new device, it’ll come with a driver disk that probably won’t have Linux drivers on it. Talk a little bit about where do you feel like this disconnect comes from between the people working on the drivers in the Linux kernel and those incorrect user perceptions, even from people who are fairly experienced with Linux.

Greg: That perception has existed for many, many years. Back in 2006, I gave a big keynote showing how it was all wrong. Everything is supported, but, yes, that complaint comes up a lot. I joke with Jim Zemlin of the Linux Foundation about this issue all the time. If he publicly says, “We’re still working on drivers,” I always say, “No, we’re done.”

[laughter]

So, there’s a disconnect there. Part of the issue is that we support more devices than any other operating system ever has. That’s a fact that’s been verified by other companies. The problem is that people only care about the devices that they have. Therefore, if your device doesn’t work for some reason, you don’t care how many thousands of other devices out there work.

When I started the Linux driver project, I’d been hearing this from all the major companies that were shipping Linux and cared about Linux. So, I went around to them individually and said, “OK, what do you need me to do? What needs to be worked on?”

Every single major hardware company, said, “Hey, you’re right. Everything works on Linux. We’re fine.” That’s shown by these companies that ship Linux on their machines. Dell, HP, and all the big hardware companies and laptop manufacturers now ship Linux, and we’re working with them.

Some of the disconnect is if you buy a machine that doesn’t have Linux on it, getting it to work sometimes requires weird BIOS tricks to get all the function keys working on a laptop, and things like that. New hardware or hardware when you’re updating a system should work. If not, then we messed up.

The graphics guys have issues that they’ve been addressing to help make that work much better. Intel hardware works wonderfully now on Linux. Their developers are very good. ATI and NVIDIA both ship closed source drivers that work on all their devices. Those all work and have very good support systems. The drivers aren’t in the kernel, but they’re out there.

Anything storage related or networking related has been working with Linux forever, and all new devices support Linux too. They have to, because that’s what those markets are going for.

Then you get to the tiny consumer devices, which is where I like having fun, because I do USB. I work really hard to get everything supported, and we don’t know of anything these days that isn’t. I was in Tokyo the other day for the Kernel Summit and walking around Akihabara and trying to find devices that we don’t support. We had all the kernel developers there and we couldn’t find anything.

I know there’s a disconnect, but if you look on a mailing list for the Windows guys, they have a lot of driver problems, where things aren’t working. Mac OS X does not support very many devices at all.

If issues arise with exotic configurations, I’ll be glad to work through them all. If something doesn’t work, tell me so I know to fix it.

Scott: Is my perception correct that when hardware companies get into Linux, they sometimes start out with a bare-bones driver? It seems that often, the piece of hardware will work, but it won’t necessarily handle the full spectrum of services like power management with the device. Then they’ll provide more capabilities, gradually contributing code to use the device to its full capabilities.

Am I at all accurate in that perception, or is it more often the case that the driver they contribute offers the same level of functionality as on any other OS?

Greg: You’re right. Some things don’t work, although we’ve made it easier for companies to write a driver that will do everything, because we put core functionality like power management into the core. Power management is now handled by the core of the kernel, so you don’t have to add that. You just have to add a few hooks in your driver and then you’re done.

The average driver for Linux is about one third the size of an equivalent driver for another operating system, so you have less code to write and maintain. Still, to your point, there are some new devices that come out whose value add is that they do X and Y differently.

Sometimes we don’t support that yet, because the manufacturer didn’t tell us how to do it or we have to figure out how to fit it into the Linux framework to support it properly. Those little webcams are a very good example. New devices come out that can support features like additional bit rates, frame sizes, or screen sizes, and we need to go and add support to the core of the kernel to support those.

So, your statement is very fair, and that contributes to the disgruntlement you’ll see on my face at times.

[laughter]

If a brand new device that says it can do X, Y, and Z is only doing X on Linux, we immediately start to try to figure out how to get it to do the other things. Sometimes, we just have to work with the manufacturers to do that.

A lot of companies are getting better at working with us to come out with drivers at the same time the hardware comes out. Sometimes we lag by six months just by virtue of that fact that Linux isn’t the first system they care about, so it takes us a while to get it going. Also, if you’re using an older distro, you may need to switch versions.

There are all sorts of configuration issues there, which is why Jim Zemlin of the Linux Foundation continues to say we need to work on driver development, and I agree with him there. There’s always going to be work, so I think I have a lifetime job here.

Scott: Correct me if I’m wrong, but drivers never really leave the kernel, right? Is it more or less true that that’s one of the reasons that Linux supports old hardware so well? That once it’s in the kernel, it’s probably going to be in the kernel for a very long time, so I can grab a six-year-old laptop or six-year-old server, throw Linux at it, and have pretty good confidence it’s going to work?

Greg: Yes, and actually that’s turned out to be a problem. Because our kernel size is growing so much, we want to make changes, and we’re running into places where we have drivers for hardware that we know hasn’t been made in the past ten years. It can be complicated to excise those pieces.

That came up at the kernel summit last week, and actually, I am going to be working on that. We’re going to move drivers that we think are broken if we are pretty sure that nobody out there uses them anymore. We’ll move those into the staging tree, keep them there for about a year, and if nobody complains, then we’ll remove them.

On the other hand, if anybody shows up and says they need one of those drivers, we can use our source code tool to restore and test it. If there is at least one user of an old device in the world, I will gladly maintain the support.

Scott: We’ve covered a lot of great stuff, and now I want to turn to a lazy interviewer trick and ask you what really interesting things I haven’t asked about. In other words, I’d like for you to supply both the insightful question and the brilliant answer.

[laughter]

Greg: Well, just to touch back on that rate of change that I mentioned before, I just looked it up, and we add 11,000 lines, remove 5500 lines, and modify 2200 lines every single day.

People ask whether we can you keep that up, and I have to tell you that every single year, I say there’s no way we can go any faster than this. And then we do. We keep growing, and I don’t see that slowing down at all anywhere.

I mean, the giant server guys love us, the embedded guys love us, and there are entire processor families that only run Linux, so they rely on us. The fact that we’re out there everywhere in the world these days is actually pretty scary from an engineering standpoint. And even at that rate of change, we maintain a stable kernel.

It’s something that no one company can keep up with. It would actually be impossible at this point to create an operating system to compete against us. You can’t sustain that rate of change on your own, which makes it interesting to consider what might come after Linux.

For my part, I think the only thing that’s going to come after Linux is Linux itself, because we keep changing so much. I don’t see how companies can really compete with that.

Scott: Like you said, a lot of the rate of change in Linux as a whole is due to changes in the drivers. It’s fairly unique to Linux that it carries the drivers in the kernel, so as the ecosystem grows, you’ve got more and more people contributing code to Linux.

That’s not the case with other operating systems.

Greg: Right. And while that rate of change is consistent across the whole thing, it’s also easier to write drivers. As I said before, your driver for Linux is one third the size of your driver for Windows, so even at this rate of change, writing a driver for Linux is less work than it is for other operating systems.

In Linux, we’ve re-written our USB stack three or four times. Windows has done the same thing, but they had to keep their old USB stack and a lot of their old codes in order to work for those old drivers. So, their maintenance burden goes up over time while ours doesn’t.

Also, as people change jobs, they generally stay with the Linux kernel community. I’ve been doing USB work for over 10 years now for Linux, and that’s a big body of knowledge. Within other companies, engineers usually move around, and that body of knowledge doesn’t necessarily stick around.

For driver maintainers and driver authors within Linux, it does stick around. The networking guys have been doing that work for 15 years with Linux, and they know this stuff in and out. They can tell you what you need to change in your driver to make it work really well.

They’re unique, and they’ll help out any company.

Scott: Thanks; we’re out of time, and that’s a good place to close.

Greg: All right. Thanks a lot; this was fun.

Bookmark this:

• Defining platform as a service and the creation of Engine Yard
• Advancing cloud computing beyond buzzwords
• Opening cloud computing as a simple, provider-agnostic framework
• Operating system and programming language flexibility
• Providing flexible scalability of data stores
• Paradigm shifts created by cloud computing, including admin roles and rapid start-ups

Scott Swigart: Please take a moment to introduce yourself and your company.

Ezra Zygmuntowicz: I’m one of the founders of Engine Yard. We started life as a hosting company, and we’ve evolved into an open source company with a platform as a service cloud offering that we run on other people’s infrastructure.

I’ve been heavily involved in Ruby and in the Ruby on Rails open source community ever since Ruby on Rails first came out in 2004. Back then, I was webmaster for a newspaper–the Yakima Herald Republic in Yakima, Washington. I was looking for some kind of new language to redo their whole website, because it was kind of an old mess.

I was reading up on Python and Ruby, and when Ruby on Rails came out, I took it and ran with it, building one of the first commercial sites in Ruby on Rails. That site is still live and running there, and that’s how I got into the deployment and hosting sides of things.

Back in the early days, nobody knew anything about deploying Ruby on Rails applications. There just wasn’t much information available on it, and there weren’t many big sites doing it. I was the only tech person at the newspaper, and they bought me a couple servers and said, “You need to put this new Ruby on Rails site online and take the 250,000 hits a day we get at the old site right off the bat.”

That pushed me to start researching how to deploy and scale Ruby on Rails applications on an open source stack. In the community, I became one of the de facto deployment experts, and I wrote a book, Deploying Rails Applications for the Pragmatic Programmers.

The business partners that I started Engine Yard with had a consulting company called “Quality Humans” that was a Rails consulting shop. They had a couple of big clients, and they were trying to figure out how to deploy them. They took them to Rackspace, but Rackspace didn’t know anything about hosting Ruby back then, and Tom and Lance saw a business opportunity for building a high quality hosting platform for Rails applications.

They went looking for somebody to help them with that. They found me in the community, and we got together in early 2006 and started planning this platform for running Ruby applications.

We scrounged up our first $100,000 to get a few servers and our first data center in the summer of 2006. We started taking customers in September 2006, and it’s been a wild ride since then. We’ve grown from a tiny startup to 85 employees in 12 locations around the world, running quite a few very high-traffic Ruby on Rails applications. That’s where we are today.

Scott: How does open source fit into your business?

Along the way we’ve had to build all kinds of tools, and our whole stack is built on open source. We were active in a bunch of open source projects, and we have a number of projects ourselves. We have a project called “Rubinius,” which is a new virtual machine for running Ruby code that we’ve been developing for awhile.

We just hired the guys who run the JRuby project, which is a Ruby virtual machine built on the Java virtual machine so it runs on the JVM. They liked the idea of coming to work for a company focused on Ruby on Rails, so they ended up coming to work for us.

On top of that, we had an open source web framework called “Merb” that we wrote to create a sort of high performance version of Ruby on Rails, and that had quite a bit of success. It became more and more of a compelling product, and we realized that we were stealing some thunder from Ruby on Rails.

We had started our company to support Ruby on Rails, so we got together with the Rails core guys and we decided to merge the two projects. So, Merb and Rails have merged to become Rails 3, so it’s kind of a cool open source story.

The long view is that we kind of forked off to make some performance improvements and provide some different architecting approaches. That was successful, and now we’ve merged that back into the main line of Ruby on Rails to make it better. That’s the little story of how we got where we are today.

Scott: That’s an interesting history, and it I think it’s pretty unique to open source. It proceeds from the idea that a group of people can take a code base off into the wilderness, take it in a different direction to support a different use case or optimize it, and finally bring it back and fold it in.

We see a lot of open source projects go through iterations like that. For instance, between Apache 1 and Apache 2, there were lots of changes, and PHP has gone through similar evolutions. As you mentioned with Ruby, sometimes things go their separate ways, and they may stay separate, or they may converge back together.

Ezra: That’s a pretty cool open source story there, because it shows how open source is a new way of working on software. I think it’s a better way than closed source, because you can get so many more eyes on the problem.

Anybody can look at the code, and if they’ve got an itch, they can scratch it themselves and improve it to do what they want it to do. We were interested in being able to run these Ruby applications in the most efficient and resource-conscious way we could.

When we started building Merb out, it kind of took on a life of its own, eventually starting to look almost like a rival to the Rails project. Both projects basically have the same goals: to build a high-level, full stack that’s easy to get started with and that provides a very rapid application development platform.

It almost felt like we were fractioning the community a little bit by having Merb as an alternative to Ruby on Rails, and like we might be confusing people a little bit. There was starting to be a little bit of rivalry between the two camps, and I just didn’t think that was useful for the Ruby community at large to have this kind of chasm.

We got together with the Rail core guys and started recognizing that our views on most things weren’t so different. We started looking for ways that we could patch the two projects together and come out stronger on the other end.

Since then, we have been focusing all our work on Rails, and bringing the good parts and the performance focus we had in Merb into the new release of Rails, which is going to come out in a beta release sometime fairly soon. Because they’re fairly similar projects, there wasn’t any reason for them to be separate.

It was cool that everybody was able to get along and see that we’re all basically working toward the same goal, and we’ll be stronger if we combine efforts.

Scott: I guess Rails is sort of a platform as a service on top of cloud infrastructure. And since cloud’s the hot topic, let’s go ahead and stick the shovel in there. To start, how do you define that?

[laughter]

Ezra: Nobody knows what a cloud is these days. It’s kind of become a term for anything that runs on the Internet.

Scott: There are some people who are very serious about having a very strict definition, while others are kind of like, “Well, if we just put cloud on our marketing material, we’ll sell more. So let’s call what we’ve been doing for 10 years ‘cloud.’”

Ezra: Absolutely. To me, there are a couple of different kinds of clouds. First are the infrastructure as a service clouds, with the Amazon Web Services platform as the canonical example. That’s what I mainly think of when I hear the term “cloud,” because it’s what I feel the true definition of “cloud” is–an on-demand compute service where resources are disposable and scalable.

I would call something a cloud if it’s API addressable first and foremost. It has to have an API so it can be manipulated programmatically. It has to be able to scale up and scale down on demand, so if you need more server capacity, you just bring a few more servers up. If your load has gone down, you drop a few and you go back down. The business model is kind of metered, like the electrical grid, where you just plug in and pay by the hour.

Then there’s all kinds of other stuff that people are calling cloud. There’s higher level stuff, like platform as a service, which runs on top of infrastructure as a service, but it’s higher level and deals more with application services, rather than the low-level infrastructure.

Scott: What’s your own history with cloud computing?

Ezra: When we started Engine Yard back in 2006, we needed a platform to be able to build this Ruby platform as a service thing. This was before EC2 was out, and before people were calling things cloud, and before there was Amazon Web Services.

We built something in our own data center that looks a lot like EC2, based on Xen virtualization and Gentoo Linux. We still have our own customized version of Gentoo that we optimize for Ruby.

We built a private cloud ourselves, although that’s kind of a loaded term as well. It was a virtualized cluster platform that ran in our own data centers. We built our platform as a service on top of that. That’s higher level and deals with the application.

As time has gone on, the Amazon platform has gotten very compelling, and other clouds have popped up, so we have realized that the value that Engine Yard adds is not necessarily at the infrastructure as a service level. We don’t really give ourselves a big differentiation by racking hardware and running data centers. We can’t compete with somebody like Amazon on that scale, as far as racking and stacking servers.

We realized that our strong point is what happens after you have a VM online with network and compute and storage. Then you need to put an operating system on there and bring that up all the way up to the application. Then you need to monitor, scale and recover your application from failures. That’s more where we fit in, and that’s where our value is–in platform as a service.

Starting last year, we started taking everything that we’ve built, all our glue code, and all our open source stuff that we call our stack, and we started porting it to run on top of Amazon Web Services, rather than our own cloud. We’re kind of abstracting that, so that we can farm that out to Amazon or any of these other providers that have vCloud or other stuff popping up.

It’s not a super interesting business, racking and stacking servers, but the code and the platform that runs on top of that is a lot more interesting to us. So as a company, we’ve transitioned from the model where everything happens in our own data centers to a platform that runs out on the cloud in other people’s servers.

Scott: We’ve talked to a lot of people who are Amazon Web Services customers, and they’ve talked about use cases. Generally, it seems to be fairly good at existing applications.

For example, we talked to some guys who do web resources for the Indy 500. They’ve got an enormously elastic need. When there’s an event, they’re going to have a tremendous amount of traffic, but when there’s not an event, there’s very little traffic. And so they’ve got this need to spin up 100 servers, have them be up for a week or so, tear them down to a handful, and then spin them back up again.

We talked to other people who do diagnostic kind of stuff, and a lot of these show up as Amazon case studies, so if somebody wanted to go look, they could go get details on these. At any rate, this company runs their own internal grid for these analytical models, and when they’d get a particularly large data set from a customer, again, they have a very elastic need.

It worked pretty well for them to take their existing apps and just run them as is. They don’t need to maintain enough hardware for their peak load, and their internal servers now are just whatever they need for ongoing day to day operations. All the elasticity happens out in the cloud.

So that’s one kind of cloud, and then there are other kinds of clouds coming on line. Google’s been out there for awhile with App Engine. There’s also Windows Azure, where you’re coding to a certain kind of API, within a certain development framework, and you don’t even see machines. You just put your code up, and you don’t know or care whether it’s running on one box or 10 boxes or 100 boxes. It’s very request-response based.

And there you’re making a bet. With Amazon, you’re not making a bet on Windows or Linux or any of that stuff; you’re not even really making a long-term bet on Amazon as a cloud provider, unless you program to something like SQS. With somebody like Google or Microsoft, you’re making a bet not only on whether you want them as a cloud provider, but also on whether you really want to stick with that framework. If you’re using Google App Engine, then the only place it runs is on their framework. And the only place that framework is available is on their infrastructure.

Looking at that world, you might think that maybe there’s an opportunity for something that’s sort of in between. On one hand, maybe you could choose an infrastructure as a service provider, as a somewhat independent choice, and on the other hand, maybe you could choose the platform as a service framework, also independently. You could sort of mix and match the two.

Is that the kind of scenario that you are enabling?

Ezra: I think you hit the nail on the head about what we’re trying to do. I look at it like this. You’ve got raw AWS as the canonical example of the infrastructure as a service right now. But they hand you very low level building blocks. You get compute, you get network, you get some memory, get some storage.

You can choose Linux or Windows or whatever, but they hand you these big building blocks, and you still have to assemble those into something that can run your application, and scale, and be reliable at the application level, and so on. You still require quite a bit of system administration, and it’s not something that Joe Schmoe can just use out of the box.

Then there are the higher level things like Google App Engine, where Google has made a lot of decisions for you. As long as you can fit within their box, you don’t have to worry about servers anymore. You just twist the dials and scale up as needed and pay for what you use.

That’s really compelling to people, but it requires them to buy into the lock-in of this Google App Engine thing. You can’t run elsewhere, and you’re going to have to be able to color within their lines. There’s a definite limitation on what you can and cannot do there. You are betting the farm on that being the right way for you to architect things.

We’ve built this platform as a service that takes the low-level building blocks of something like Amazon or another infrastructure as a service provider, and we have done the research on how to assemble the best practices. We have figured out how to load balance a scalable application farm, a database with replication, and all this kind of stuff for you, and we’ve figured out how to work with elastic block storage, and snap-shotting, and all that kind of stuff.

We’ve taken that and wrapped it up into a nice little push button platform as a service, where it’s pretty much just as easy to get going as with App Engine or something. You tell us about your application code and click a few buttons, and we’ll build a whole cluster for you.

But it still looks like your old school Web 2.0 deployment. You’ve got your load balanced applications and your MySQL or Postgres databases. You get SSH access with root login on your boxes, so you can still login and see what’s going on, and it feels familiar to people.

Our platform as a service is kind of in the middle of that. You can still not really know what you’re doing and come and just click a few buttons, and get this full cluster and everything up and running, and then add capacity and remove capacity as your traffic goes up and down.

But there are no limitations as far as what you can run there. It’s just Linux. You have root access, so anything you have been running on Linux servers will still run on this platform and none of this color within the lines stuff, where you can’t do a certain thing on a platform because they don’t allow it.

Scott: Will it actually spin up and spin down nodes as demand increases and decreases?

Ezra: We haven’t released that feature yet, but it’s something that we’re working on right now, and we hope to release it fairly shortly. Right now we can basically do that, but you would get alerts, and then you would have to make a manual decision to click a button to add or remove a node. We are working on adding the whole auto-scaling thing.

Auto scaling is really hard to do in a least common denominator type of way that works for everybody, because it can be so application specific. But we can kind of get away with it because we’re doing this platform that is all Ruby applications. They all have very similar needs and profiles in terms of how they work.

Scott: I don’t know your guys’ architecture, but one of the scenarios somebody mentioned to us was, “It’s not memory or CPU that I’m concerned with; I care about the queue length and the simple queuing service. And based on more queue items, I want to spin up more nodes.”

Ezra: Right. If you just build an auto scaling system that adds nodes when your load average goes above X, it’s never going to work that well. You’re either adding too many nodes all the time, or not keeping up, or whatever.

Scott: Or not adding them to the right tier.

Ezra: Exactly. That’s not super helpful, so what we’re trying to do is build something that takes load into account as one of the factors.

Like you said, maybe they’ve got a queue they’re listening to. If the disk goes above X, maybe they want to spin up two more nodes. Or maybe their application is processing X requests a second, and if they hit a certain threshold, they want to scale up or down in a specific way.

We’re trying to build in some general heuristics that are least common denominator, but still allow it to be a little more intelligent about what it does.

Scott: Amazon’s an infrastructure as a service, and VMware’s approach doesn’t look like they’re going to stand up five massive data centers in five places on the planet. It looks like they have more of a strategy of going after existing hosters and getting them to be vCloud partners. By being a vCloud partner, they’re going to have an API exposed to them that lets them programmatically control their instances, kind of like what they can do with AWS. They’re going to have a self-service provisioning portal, an admin console, and things like that.

Do you see the Rails stuff that you guys are working on down the line supporting this heterogeneous ecosystem of infrastructure as a service providers?

Ezra: That’s definitely something we’re targeting. We’ve been talking with VMware a lot lately about the vCloud stuff, and we had an early demo of our platform as a service working against vCloud that we demoed during the keynote at VMworld in Paris six months ago.

We’ve got the AWS thing down, and it’s working great. We’ve got lots of customers on there and customer rate is growing great. And right now we went GA with our platform at the end of September, so this first version is fairly feature complete.

Porting our system to work against vCloud would give us a bunch of bang for our buck, because that’s not going to tie us to just one infrastructure as a service provider. If we code to the vCloud API, there’s going to be 10 or 20 of these vCloud approved service providers that we can point our application at, and go to town. Then we know how to instantiate a cluster that would run your applications on AWS or on Terremark’s vCloud, or on Hosting.com’s vCloud, or any of these other places.

Scott: And that’s not the piece you are focused on, right? You’re focused on your framework, but like I said, you could envision a future where there’s a Rails cloud framework and a pile of hosts that people can choose from, and maybe some other Java cloud framework or PHP cloud framework or .NET cloud framework, or whatever, and they could mix and match.

In that scenario, if a given department prefers a certain language, there’s a cloud-based, auto scaling dynamic framework that supports that. That also provides a choice of development models and a choice of places to host it that can also be mixed and matched.

Today, I could run ASP.NET on Rackspace and run PHP on Hosting.com, but those are not really cloud frameworks, and those providers aren’t really in the path. The hosting providers haven’t really provided an elastic resource, and it’s been fairly static kinds of units that people have to buy in.

It seems like there’s a gravitational attraction toward the programming frameworks becoming better at cloud development and the traditional infrastructure providers becoming better at being cloud infrastructure providers.

Ezra: It makes a lot of sense. For example, with Google App Engine, you really have to code to their platform to run your app on there. Everything you write in your app has to take into account what they’ve done there, and how to use a different data store and everything.

But with our platform it’s just your typical Web 2.0 stack with your applications here, your memcache here, and your database here. It looks familiar to people, and they can run their existing applications on it but still take advantage of the scalability in the cloud.

If they get their application running against our platform on AWS, as soon as we add other back ends, they’re going to be able to click a button to deploy it out onto a different cloud without any changes in their code or anything.

We’ll take care of the differences in architecture underneath and still expose the same interface to the application developers. They can just worry about writing their code and let us take care of figuring out the best architecture for each different cloud.

Scott: One other facet is that if people want to run this framework internally, they can benefit from using the same framework internally and externally.

Is the stuff you guys are working on mainly a Linux thing, or could you see people running Rails 3 on Solaris? Does it work on Windows? What degree of choice do people have there?

Ezra: Rails does work on Windows and Solaris, but Ruby in general is kind of sub-par on Windows compared to Linux, and nobody really deploys production Ruby stuff on Windows unless they absolutely have to integrate with some .NET or other Windows-specific stuff.

Scott: As you look at the other choices that people have in that Linux ecosystem, a lot of times it’s Java, PHP, and a smattering of other things. And then there’s Rails, which has gained a lot of popularity in a relatively short period of time.

Popular conception seems to portray Java on one end as the strongly typed, object oriented, structured enterprise programming language and PHP on the other side that lends itself to learning in a weekend. Coming from a Microsoft background, I think of it like VB for the Web. There’s a very low barrier to entry, but the people with computer science degrees point at it as not real programming.

As a development framework in general, and then thinking about cloud development scenarios of the future, what do you think makes Rails a good fit?

Ezra: I think that Rails is really good at building a prototype and getting something out and usable to your customers very quickly. It makes some of these decisions for you in starting a new project.

If you’re starting a project in Java, you’ve got to sit down and figure out what application server you’re going to be using, what choice of frameworks, and which ORM, and which XML config files. You’ve got to piece that all together, and then spend a few days configuring it, so there’s friction to starting something new and just trying it out.

Rails has a really nice out of the box experience, where it makes all these decisions for you that are fairly sane. You can tweak them later if you need to, but it allows you to just jump the gun and get something started very quickly. And then there’s a whole slew of plug ins for doing social networking things, friend of a friend, or sending emails, or all this stuff just out of the box you can plug in with very little code.

One of the big advantages of Rails I see is that it allows you to take an idea and have something to show people in a week or two with a couple guys developing on it. That might take three or four months to get the same prototype in a Java system. It’s just very rapid development.

You can rapidly prototype a website, get it in front of your customer, and then iterate based on real customer feedback, rather than having these really long iterations where you get a bunch of specifications from a customer, and then you come back to them three months later.

Then the customer doesn’t like it, so they give you a bunch more specifications. Then you come back three months later, and maybe you’re kind of ships sailing past each other in the night, not really getting the feedback that you need.

With Rails, you could be showing stuff to your customer every day, or every couple of days, and then iterating based on their real time feedback. You can just be so much more productive with it, and I think that’s where its big advantage is.

As your app grows up and becomes more complex, Rails can support that too. And it has tunables and other options so you can get in there to make it really scale. There are good stories for the stack and the way to scale it now.

One of the other big advantages is that Rails enforces a directory structure, the way the app’s laid out, and the testing frameworks and so forth. That means you can hire a Rails developer, and they’re going to be able to find his way around pretty easily. There’s not going to be a month or two of learning how you customize J2EE or whatever. So there’s this shared knowledge between all Rails apps that allows you to get programmers up to speed much quicker.

Scott: When you compare it to PHP on the other end, is the difference the model of your controller paradigm, or what? Because people think of PHP as being a very rapid way to stand something up and prototype it.

Ezra: You can also rapidly prototype stuff in PHP, but when you do and then it gets successful, you’re probably going to have to rewrite it entirely because it’s probably a spaghetti mess. With Rails, you could rapidly prototype something, and then if that prototype is successful, you’re built on a very solid foundation where you can keep scaling that project and keep it maintainable as you add developers and stuff.

Scott: With things like Google and Google App Engine and Bigtable, it seems that the popular approach to scale a website is by using a traditional kind of relational database back end. It could be MySQL, PostgreSQL, Oracle, SQL Server, take your pick.

But if you really want to scale infinitely from the beginning, you’ve got to build on top of one of these distributed, shared databases, whether that’s SimpleDB or Bigtable or whatever. Talk to me about the options that you have at the data layer.

Ezra: There’s this whole no SQL movement going on, where people are exploring other databases, because they’ve outgrown MySQL or PostgreSQL. The reality is that I think there’s a little bit too much hype around that stuff right now.

Most people will never outgrow MySQL or PostgreSQL, even though we have a lot of people who come to us saying that they’re going to be the next Google or Twitter, so they have to be able to scale huge.

They might lose a lot of time by trying different options, whereas they’d be better off just starting with a SQL database and getting something out there fast. Then, as they hit the ceiling, they can move portions of that out into some kind of key value store or something else.

We have support for a couple different key value stores on our cloud. You can use Redis, or Toyko Cabinet and Tokyo Tyrant. We have out of the box support for those two. We also like MongoDB a lot. And also MySQL with as many replicas as you want.

The progression usually starts with the thing on MySQL server and a couple app servers, and as I scale those up, I might add a slave or two to the MySQL server and do read-only queries against the slaves and scale down my reads.

As I start to swamp that master database with my write capacity, I might move certain features out into a memcache or a Tokyo Tyrant tier where I don’t need as sophisticated data access patterns and ad hoc queries; I just need raw speed and scalability of storing a ton of data.

If you have some app that really just screams out, “I don’t need a database, I just need a key value store,” you might start with one of those from the beginning. Still, you’re going to want to keep your customer and your transactions–anything to do with money–in a SQL database with the proper constraints and stuff.

But there is quite a bit of stuff, especially in web pages. You can store rendered HTML template fragments in a memcache or a Tokyo Tyrant, because they might be pretty expensive to pull all the data and build a template, and then you might cache the result of that.

The idea is basically to scale MySQL up as far as you can go and then start taking features and pushing them off into a key value store to take a load off of the database, and to have a system where you’ve got a relational database augmented by key value stores, and you’re using them each for what they’re really good at.

Scott: That ties into another thought I had. As some sites have gotten really popular, they have crashed under their load, so they pushed all their images and static data into S3. The page would just have a bunch of image tags, and it would pull the images from Amazon S3. They would off-load a ton of bandwidth from their servers, and the user wouldn’t really see any difference.

There were a bunch of case studies around that, which told the story of how Amazon S3 saved some sites when they got popular, because they were able to off-load most their bandwidth, without really off-loading any of their processing.

Like you said, Google really forces you to color within their lines, with a limited choice of programming languages, no relational data store, and not really great support for a background process. It’s still sort of under the constraints of a request and a response.

People have taken the part that won’t fit in those lines and stuck that over on Amazon, and they just call out to it from the Google App Engine code, or those kinds of things. People I’ve talked to who have programmed on Google App Engine have said that data store is the paradigm shift.

That’s the hardest thing to wrap your head around if you’re used to programming against a relational database, which everybody is. Paradigm shifts are hard. It took 20 years to go from procedural programming to object oriented programming.

Ezra: I think those things are definitely in the future for a lot of apps and a lot of programmers, but like you said, they are paradigm shifts, and a lot of people just aren’t ready to make that shift yet. Cloud computing is pretty young still, and everybody is still trying to figure out how it’s going to work, what they’re going to use it for.

There’s another good five years here where people need to move their legacy applications to the cloud, or they want a cloud environment that looks similar to their on-premises stuff, but they might not be ready to make the jump to the full-on Google App Engine style.

I think something like what we’ve built will be more compelling for the next several years. It takes advantage of the elasticity of cloud, and the programming capability of the APIs and stuff, but it still kind of looks like your traditional infrastructure.

Scott: It might be compelling for a lot more than the next five years. At the end of the day, free is really nice, and it is going to attract a lot of customers. But, the corollary is that those providers are not necessarily going to make a lot of money off of those customers, because they were attracted by something that’s free.

You can kind of democratize and commoditize a certain kind of lower end of the hosting space, but I look at you guys, and I think, OK, there’s a framework that I could run. I could run it internally. I could in the future run it on a variety of external partners, whether it’s Google, or vCloud or whoever. It’s one of the frameworks I could choose.

Tell me if this is part of your pitch, but I could see companies that are looking at this saying, “Well yes, but you’re a small company, you’re a start up, we don’t really want to bet on you.” Is this a place where open source helps, since you’re able to tell them that they’re not really betting on you but rather on the longevity of an open source project?

Ezra: I think that one of the strengths of our software is that it is built entirely of open source components. We have a lot of proprietary stuff that ties it all together, but at the end of the day, the stuff running your app is Linux, nginx, Rails, MySQL, memcache, and a bunch of other stuff that’s all open source.

If we went out of business or you didn’t want to host with those anymore, you’re not locked in. You can pick up your app and go elsewhere, and be able to put something together that looks close enough where you’re not going to have to rewrite your code. That helps, because you don’t get locked in when you use our platform like you would with App Engine or something.

Scott: We’re getting near the end of our allotted time, so is there something interesting I didn’t ask about?

Ezra: I think we’ve covered the main points. The whole thing is a paradigm shift, since there are still a lot of people that I call “the server huggers,” who really like racking and stacking servers. They like seeing all those blinking lights, and they like having direct control over the servers and being able to choose the hardware down to every last card. Those guys are having a bit of a harder time with this transition.

I think the whole transition to cloud computing is really going to come to a head and force a merger between typical operations guys and developers. Since the infrastructure is programmable via API, developers are enabled with the power to control the infrastructure without having to call the systems guy every time they need a new server.

I think that your traditional operations guys are going to have to learn to become developers, and I think your traditional developers are going to have to learn more about operations, so that they can play within this new environment that can be totally programmable. I think that’s going to be one of the big paradigm shifts here.

Scott: It’s also interesting that you mentioned that in order to start out, you had to scrape together $100,000 and put together a small data center. Today, the distance between an idea and a product in the market is getting smaller.

That’s due in large part to the work that companies like you, Amazon, and Google are doing. The start up money that somebody needs to take an idea and run with it is far less, and if that notion doesn’t work, fail fast and try something else.

It seems that these technologies are just letting people try things out and spend a lot less time doing the market opportunity analysis and writing the business plan, and instead just trying stuff.

Ezra: The whole switch from Capex to Opex is a real game changer, and you’re right that it is largely cloud-enabled. If a couple of guys at a university have an idea that needs a couple hundred servers to run, in the old days, they would have to have a lot of money to buy a hundred servers and rack them up and get them available.

With the cloud, you can boot a hundred servers and play with them for a few days and shut it off by just spending 100 bucks or something. Being able to that easily try out an idea and see if it’s feasible or not is a game changer.

That flexibility is really compounded by things like Rails, which are enabling really rapid development of these sites. It gives a lot more power to individuals, and smaller teams these days can build pretty complicated products very fast and then scale them very easily.

That’s huge, I think. That’s one of the biggest things that this whole cloud trend is changing. All these open source technologies, along with the infrastructure on demand really does allow people to just try big ideas and then fail fast. Like you said, if it’s not going to work, they’re not going to be out a lot of money.

Scott: It also enables departments within a 150,000 person company to be really entrepreneurial and really experiment with things. The boss could practically try out an idea with his credit card, and the budget could go on an expense report.

Departments can go outside of traditional IT because they’ve got a budget and for things that are below a certain threshold, they don’t need approval. They can have a hosted CRM system or mail accounts or all kinds of things, including their own servers in the cloud that their IT department doesn’t even know about. They’re completely off the radar.

On one hand, IT cringes at that, but on the other hand, companies want risk taking, especially if there isn’t even really that much risk with this kind of entrepreneurial spirit, because it’s just hard to predict what’s going to work.

Ezra: There’s a big push for agile development. Test-driven development, Scrum, Agile, and all this kind of stuff has really changed the way people write software. They do rapid iterations and they test and they get stuff done very fast.

I see this whole space we’re in as enabling agility and deployment as well, because it used to be that maybe your development team is rapidly iterating and coming up with new features and all this stuff, but then there’s this big brick wall to get it deployed. They had to hand it to the ops guys and order servers and wait.

It was a big ordeal to get stuff deployed, but this new approach totally wipes that out and allows you not only to rapidly develop your product but also continuously deploy it and rapidly have it online and visible. It tears down that last wall of having your product go from idea to actually being out alive in the world and doing something useful.

Scott: I was just reading “Information Week” and there was a CIO saying, “One of the reasons I’m going with a hosted solution is my people never have to perform an upgrade. They never have to upgrade the CRM system. They never have to upgrade the mail system. Those used to be big projects that took lots of planning and lots of experimentation, and I will never have to devote staff to doing an upgrade. And at the same time, we feel safer making customizations because there’s a little bit more of a feeling that that fast provider isn’t going to break everybody’s customizations, because they’ve built the sandbox that we can customize within.”

And so they’ve got better visibility into what people are doing and not doing because it isn’t the case anymore that just anybody could be doing anything. They built the sandbox up fairly incrementally in terms of customers wanting to be able to do X, so we let them do X. Well, now they know they can do that, so if they’re upgrading their infrastructure, they’re able to take that into account.

I guess we are out of time. Thanks for talking with me today.

Ezra: Thank you.

Bookmark this: