Interviewee: Daniel Jacobson

In this interview we talk with Daniel of NPR. In specific, we talk about:

• Providing open access to content at NPR
• Extending content to multiple platforms
• The role of open source in NPR’s public mission
• Relevance of open source ethos to no-for-profit endeavors
• Hurdles to making their technology public
• Significance of commercial acquisition of open source projects
• Choosing technologies and technology philosophies
• The elusive nature of content control in a connected world

Sean Campbell: Can you tell us a little bit about your background and your organization?

Daniel Jacobson: I started out as a developer, initially in Microsoft technology, and from there I migrated into technologies like Java, PHP, Oracle, and MySQL. I was a developer for quite a few years, but I eventually transitioned into more of a managerial and leadership role at NPR.

That transition started with the advent of our content management system, which I led the building of and deployed back in 2002. It is still the centerpiece of everything we do at NPR, in digital media. Now I’m leading all of the application development efforts in digital media at NPR.

NPR obviously still is a traditional media organization. But digital media is now a major part of who we are, making us a multi-platform media organization. The digital media arm, of which I am a part, is basically responsible for distribution for all platforms other than radio. The CMS manages all of these distribution channels for digital content, including NPR.org, the API, podcast, RSS, and mobile platforms.

Sean: Unpack for us a little bit what you offer to the public and organizations through the API. I’ve experimented with it a bit, and it’s amazing how fast you can carve something together and throw it in something like Google Reader to get a fairly fast updating stream.

Give us a 30,000 foot view of what’s there and what people can do with it.

Daniel: Almost anything that you can get on NPR.org, you can get through the API, to the extent that we have the rights to redistribute it. In terms of slicing it to get interesting feeds of content, we have a whole slew of categorization schemes and keywords that we apply to each story.

That lets you go into the API and make a feed, identifying content by categories like topics, the program it came from, or the personality who presented it, like Nina Totenberg or Noah Adams.

You can also build mashups like an “All Things Considered” feed of content where Noah Adams talked about the Appalachian Trail. You can then parse that custom feed in one of our many output formats.

Currently we offer eight output formats. NPRML is essentially an XML format that matches as closely as possible the data in the database. It’s the richest format that we have. We also have RSS, media RSS, Atom, and podcast outputs. We actually built a layer on top of the API called “Mix Your Own Podcast” so you can just throw terms in there and create a custom podcast on the fly.

We also have HTML and JavaScript widgets, and we also output through JSON. In addition to that, you can paginate through the results, so if you get 20 results in the first page, you can page through the next 20. You can also do date range searches.

Our goal is that any way you would want to search through the API, we provide a mechanism to do it.

Sean: Can give us an example of a particularly interesting way or successful way this functionality has been used?

Daniel: That’s a timely question, because I’m preparing exactly that information for OSCON right now. [laughs]

Our target when we built the API included four distinct audiences. One of them was obviously the public, so we could open up the world and see the great things they would build. We also had our member stations in mind, as well as business partners and ourselves.

Later this month, we will be doing a series of major launches, and people will be able to see how we’re implementing the API on our live site. Describing the ways NPR is taking advantage of it may be good context for how other people can feel comfortable in building things and knowing that whatever they build is going to persist. This is the case because NPR’s products will be based on the same API.

The API is going to be the centerpiece of our site; it is the technical infrastructure, with every single page is derived from the API itself.

We have also created another layer on top of it that enables more front line people, including editors and designers, to add custom feeds to anything on the site. We have built tools in our CMS that have enabled them to construct API feeds on the fly, drop them onto the page, and apply a style sheet to present the content in the way that they want. It will also let them configure how many stories, what elements to show,– basically enabling the editors to have control of the related content for the story that they are working on.

Sean: How are the ways those other audiences use the API different from the ways you use it internally?

Daniel: We really see the API as a way to get content to member stations so that they can reach their audience better. Quite a few stations out there use it in novel ways, including Minnesota Public Radio. They just did a new site re-launch and they are making extensive use of the API. Some other station re-design sites are coming up, and they have told us they are making extensive use of the API as well. North Country Public Radio is another example. They have told us that somewhere around 50% of their pages have API-derived content in them.

Partnerships are another target audience. The API gives us a very easy, fast mechanism to get content up on their platforms. Instead of working with a partner to try to get our two systems to talk to each other, we can simply point them to the API, and they can pull the content out and build it into their application.

As NPR explores other platforms, like mobile, iPhone sites and apps, Android, and others, it’s very easy for us to get our content up there. Beyond that, obviously the public is a big target audience for us.

One of our favorite apps is called “NPR Addict.” It’s on the iPhone, created by a public API user named Brad Flubacher. His app takes our feeds and streams the audio on the iPhone. Back in December, we launched a station finder API, so he is taking advantage of that as well to find member stations. There is actually another station finder app on the iPhone Store as well.

Sean: Other than the iPhone, what other platforms do you see a lot of interest around for NPR content-related apps?

Daniel: We have seen a range of code wrappers being built, including some Perl and Ruby and other implementations. One of the more interesting implementations was an audio player someone built for the KDE environment. This is a very interesting area to me, because we’re probably not going to be building apps for KDE, but somebody went out there and filled that void and reached out to an audience that otherwise wouldn’t get this kind of customization.

Some other mashups done by public users include Twitter mashups (like NPRBackstory and All Tweets Considered), Reverbiage, which displays NPR stories on a map based on their locations, a Simile Timeline mashup and some widgets focused on NPR Music. We have also built a Yahoo! widget and a Google Gadget, both of which place NPR content on those portal sites.

Scott Swigart: Tell us a little about the intersection between what you guys are doing and open source in general. I’d be interested in knowing about the open source you are using for the technology you are building, as well as hearing about the open source development audiences you are hoping to make life easy for as they connect to your API and consume content.

Daniel: We in digital media have a philosophy that to the extent possible, we want to use open source technology. We can’t always do that for a variety of reasons, but that’s our first instinct.

We’re an Apache shop, and we use Debian operating systems for all our web servers. We’re running PHP for outward facing websites, and our CMS, which we built in-house, runs in Tomcat and JSP.

The one major area where we are not using open technologies is our current database, which is Oracle. That said, we have actually built the API essentially as a data repository layer that sits on top of Oracle. It’s really just an XML layer that stores all the content.

We did this for a variety of reasons, starting with performance, and it also gives us a lot of modularity to transform content pretty quickly on the fly and get it into any of those formats that I mentioned earlier.

But in addition to that, we wanted to create an abstraction layer so that we are not dependent on any given database technology. That has real benefit to us in case we have to pull the database out of the system for whatever reason, and we actually saw an opportunity here to migrate off of Oracle. Sometime within the next year or so, we will probably migrate from Oracle to MySQL.

That would be the last major step of moving entirely to open source technology. We do use some other smaller proprietary tools, but at the core, we are using open source.

Scott: What about contributing back to the open source community?

Daniel: The API was really our first venture into that, aside from RSS, which I don’t really consider to be an open source move.

The API is really the core thing that we can contribute to this world as part of NPR’s public service mission, in the sense that it allows us to give our content to the users and let them use it in meaningful ways that adhere to our terms of use.

Those terms of use are very important, because obviously, we don’t want people making improper use of the data, and if they are going to use it commercially they are going to have to talk to us and set up agreements.

But assuming they are using it within the terms of use, we really want to make the content available out there, so that people can do all those interesting things that I mentioned and hopefully more.

I think that in the next several months, you will start seeing more aggressive open source moves from NPR. We are going to start trying to contribute code and other tools to the community so people can do interesting things with them.

We’re still researching exactly how to interact most effectively with the community, opening our technology up to them so they can do whatever they want with it, while we understand our role in that (if there is one).

At OSCON, we are conducting a media mashup with the New York Times. We seeded it with several mashups that we have created and shared the code on github. We are trying to facilitate the conversation about code and apps and working with other media organizations.

Scott: We interviewed Mark Fronds at the New York Times, and he seemed to be fairly pragmatic about a certain ideological intersection between journalism and open source that led him to feel like open source is a good foundation to build on top of.

Of course, unlike NPR, the New York Times is a for-profit entity. He liked the idea of building proprietary intellectual property on top of open source, using things like Apache licensing that are a little bit freer in terms of being able to build IP that you don’t necessarily have to contribute back.

NPR is a little bit different ideologically, in terms of making the content freely available. You’re probably not as much out to build a business and a revenue model. Ideologically, how do think that plays into how open source fits into NPR’s mission?

Daniel: The fact that we’re a non-profit organization is the key point. Our public service mission is to inform and educate, and we continually seek out those ways that will give us the best opportunity to fulfill that mission.

Of course, we still have to generate revenue, but we have to factor in more than just the bottom line, and part of that is how well we are serving our stations and reaching the public. Opening up our content through an API gives a lot of latitude to serve our audiences, without restrictions that commercial media outlets like the New York Times, Reuters, or the BBC have to contend with.

For example, I think ours is the only major media site that does not have any rate limiting on our API. We opened up all of our content, and you can download all of our MP3s through Mix Your Own Podcast, to the extent that we have the rights to do all this, which is most of the content.

We basically just jumped head-first into this pool, because we saw that this was the best opportunity to get people interested, engaged, and writing very interesting applications or widgets based on our content.

Ultimately, that’s going to reach a greater audience for us, because if people are, for example, writing this KDE app on a Linux platform, we’re not going to be there on our own. So now this tool allows us to reach a new audience, and it potentially exposes new people to NPR.

NPR’s position is also unique among media outlets in the sense that we’re not truly giving up the content in the way that “New York Times” would if they offered full text. Their prime asset is their text, whereas one of our prime assets is audio. Of course, we’re generating more and more text, and images, and things like that, but the New York Times is a text organization, so if they’re feeding out their full text content, it’s gone.

It’s still important for us to maintain the brand and make sure that the attribution follows it. If people are really seeking out the audio, though, which is a large portion of the audience, that audio is still being distributed from our servers, and it’s always a revenue opportunity.

For example, for MP3 downloads, or for access to the M3U or WMA links, or any of that kind of streaming material, we have the opportunity to do a pre-roll on top of any of that. We’re not doing so in most cases, but we have this kind of revenue advantage, I think, because our asset is more bound to our servers, and it’s harder for the public to disseminate.

Scott: You’re obviously building technology that will stay inside of NPR. Could you envision releasing the internal code in a way that would let other media organizations replicate what you have done to expose their content?

Daniel: The big hurdle is the time, money, and resources to abstract the code so that it is portable and can be used in any kind of platform, or at least some subset of platforms.

That’s a challenge, and it’s especially a challenge because we are a non-profit organization and have fewer resources compared to other media organizations like the New York Times, USA Today, or the Washington Post.

We have five permanent in-house developers, and we sometimes bring in contractors. That’s a pretty small team to be running an operation that yields so much, so the first hurdle’s going to be resources.

The philosophical hurdle doesn’t really exist. We would love to open source as much as we can, and even if it’s not truly open source, we would like to do things like enable our member stations, who actually have a very similar plight to us, to create and distribute audio-rich content.

We would love to create tools that would enable them to do more, and to make it portable so that they can just pick it up and plug it in. We talk about that pretty routinely, and ideologically, we are all for it. Resource-wise, though, it’s a challenge.

Open sourcing the CMS would be a huge endeavor. I can imagine an app that sits on top of the XML layer that is more generic so that it can read in any XML and then convert it and yield output formats, whether it’s Atom, or RSS, whatever else. I’d love to do that.

We also publicly document everything that we can. For example, I post pretty regularly to our Inside NPR.org blog with details of our technical implementations, strategic decisions behind what we did with the API, and the challenges we’ve hit with it.

We’re pretty open with our knowledge, and we’re trying to get better about being open with tools. I think that the big picture goal (which I think is a real long way out) if possible, is open sourcing our systems.

Sean: Regarding your plan to migrate from Oracle to MySQL, what is your perspective on the significance of MySQL being acquired by a commercial company?

Daniel: That acquisition did give us some pause, but there are several reasons why we are still considering MySQL. Open source is certainly attractive, and as I mentioned, part of our core philosophy is to use open source platforms where possible. The fact that it is open source is not the primary reason, but it’s a nice reason.

For example, if we determined that Oracle made the most sense, because it’s better suited to the enterprise, because we need a support contract, or whatever, we would adjust our approach accordingly.

For us, however, it boils down to cost and scalability. If we want to scale our Oracle system and create redundancy, and allow us to swap in and out machines for maintenance or set up a RAC system, that’s very expensive. For a nonprofit organization with limited resources, that’s a challenge, and every time we add new CPUs, our price goes up.

With MySQL, the cost of adding new CPUs is limited to the cost of the CPUs and resources we need to implement them. The clustering environment is also a lot easier to implement, whereas RAC takes very specialized skills, so there’s a consulting cost on top of it.

MySQL is a much easier clustering environment, and it gives us that scalability for the future. We can have that redundancy, and we can always throw more hardware at a problem without worrying about those financial issues. The added benefit of it being open source, with a community that will continue to build on it, is icing on the cake.

Our view is that it is very unlikely that MySQL will be shut down as an open source project. If it is, the major alternative would be PostgreSQL, which we actually have already talked about, but I think Oracle would be foolish to shut down this open source community. That would create a major backlash that would probably also lead to forking of the code anyway.

Sean: There are light leaks around the community, and whatever Oracle would decide to do, someone would try to continue it in some form or another.

On a completely different topic, though, what are you guys running in terms of the operating system? And what are the factors that made you look more at MySQL than PostgreSQL?

Daniel: We use Debian as an OS. In terms of leaning more toward MySQL than PostgreSQL, there are two factors. The first is human resources, in the sense that our people know MySQL much more so than PostgreSQL.

There’s a little bit of a learning curve and expense that goes with PostgreSQL implementation that MySQL wouldn’t. That includes setting up the server, integration of the clustering, and building database schemas that are optimized for MySQL versus PostgreSQL.

The second factor is just that anecdotally, from some really simple tests that we’ve executed and some research that we’ve seen, MySQL tends to be faster. I don’t know what the current studies show on the latest versions of each, but speed was one of the factors in making this decision.

Scott: There seems to be another movement out there, the NoSQL movement towards Big Payable and stuff that really distributes and sort of clouds out I guess. Is that something that you guys have any interest in?

Daniel: I’ve been reading a little bit about that lately, and I think that the XML Repository is some form of an implementation of that. So I think philosophically, I agree: thick relational systems that have all kinds of joins carry a cost, and it’s usually in terms of performance.

I like the philosophy of the NoSQL approach, and I think that creating this really flat XML layer is a step in that direction. In some ways, there is still a benefit, however, to a relational database model, and I think it depends on what the system is.

Basically our approach is that we keep content management relational and highly normalized, which is very effective for managing data for a small set of users.

We try to abstract that out, flatten it, and get it outside of the database in this XML layer, for performance and for all the reasons that the NoSQL movement is even getting a voice. I think there is a purpose for both and that’s of our approach to our architecture.

Scott: Maybe it’s the nature of developers seeing things in very black and white terms, but there always seems to be a methodology du jour. You see the same kind of thing with Ruby on Rails as it comes along and PHP.

There tends to be a trend toward making something really simple, and then complexity gets added over time, until something really simple comes along to replace it. Everyone says, “Hurray—we’ve done away with all that complexity,” and then a week later, people start to complain about its limitations, so it starts to become more complex in its own right.

Daniel: I don’t think a smart technologist is someone who knows all the little nuances of a given technology or even a series of technologies. A smart technologist is someone who knows when to use which technology for which situation.

Sometimes it’s the perfect scenario to use a relational database model and really chunk out your data in normalized ways. Sometimes it makes sense to make it flat. People who understand architecture are the people that can actually make those right decisions.

Regarding simplicity versus complexity, RSS is a good example, and I think it has a place in the marketplace and in the history of the evolution of the Internet. It’s just not enough anymore in this distributed world that we live in, although it’s a great tool for certain things.

Now you need things like our API. You need more complex distribution channels, because that’s the world we live in. Maybe it will get simpler somewhere down the road, but at the moment, that’s the evolution I see.

Sean: That ties into the topic of control of information. Pandora got a good licensing deal on streaming music and then turned right around and said, “Hey, radio stations, why do you pay less for your content stream?”

We are also starting to see people sue organizations like Facebook, insisting that they can’t lock up all that profile information and not give the user the ability to have their profile federated and available to them at any time.

I’m curious about your views in this area, although I understand that you’re not speaking on behalf of NPR or anyone else.

There will come a time, I like to joke with Scott, when the Supreme Court will say, “I don’t care what agreement you signed. In order to allow the free flow of commerce, these profiles and this information has to be distributed.”

I find it hard to believe that the Facebooks of the world will be allowed to grow into behemoths that own all that data without being forced to federate it with small organizations. At the same time, I see you giving it all away so people can use it if they want.

Is this is a conversation you have had over coffee at some point? Do you see this really as a small, anomalous moment in time where companies can have people sign an agreement that says, “All your family photos are ours for eternity,” and that eventually that just really won’t be able to stick?

Daniel: I’ll give you my personal take on it, although as you said, this does not represent NPR. I think that right now, we’re in a very ambiguous time in the marketplace.

Previously, there was all this lockdown, and there were subscription models that took content away, and you had to subscribe to it. No APIs were distributing this stuff out in a meaningful way, besides RSS, which was really a ploy to get users back onto people’s sites.

Then this whole API movement and widget distribution happened. I think Facebook is a contributor to this, in having the ability to create tools that enable content to be put into that framework.

All of that distribution has created a lot of ambiguity, in terms of who owns what content, whether it should be available, whether it can be locked down effectively, etc. I think that there is a real shift toward opening up all this content in meaningful ways. A lot of organizations are embracing it, although many are still locking down certain materials.

And of course, everyone still has a terms of use agreement attached to it, and your question really speaks in large part to when those terms of use will go away.

If you’re putting content on the web, it’s already out of your hands, and you’ve ceded control. If I’ve put a status update on Facebook, I have to assume that it will be widely distributed, even if it is not going to be. The fact that I’m publishing it in any form to a public space, whether it’s Facebook, NPR, my personal blog, or whatever, makes it open.

There might be some continued lock-down in terms of whether people can legally make money off of something without my permission or whether they have to give me an attribution. The murkiness lies in those details, but I have to assume that if I just put it out on the web, I’ve lost control of it.

Sean: I certainly concur with that. Do you have a read on what NPR’s position on that issue is?

Daniel: I’m not sure that NPR has quite so extreme a view on this, but I will say that we had many conversations as we talked about the API concerning whether we should release specific types of content. We talked a lot about what types of content we should hold back, in order not to miss certain business opportunities and things like that.

The prevailing argument was that, if material is already on our website, it’s a lot easier for the average person to copy and paste off of our web page than it is to parse an XML feed. So, if we’re concerned about people stealing our content in ways that we don’t want, what we really need is a really nice terms of use that enables people but also sets some boundaries.

Sean: Well, that’s a good place to wrap up, and we are at the end of our time. Thanks for talking with us today.

Daniel: Thank you.

Bookmark this:

Interviewee: Stefano Fornari

In this interview we talk with Stefano the CTO of Funambol. In specific, we talk about:

• Defining synchronization as a core technology for the future of mobile computing
• The protocols underlying synchronization scenarios
• Various mechanisms used to initiate synchronization
• Characteristics of the open source community specific to mobility
• Conventions that govern contributors and contributions
• Core technical problems in synchronization

Sean Campbell: Give us a little bit of background on you and on the company.

Stefano: All right. I am a cofounder as well as the current CTO of Funambol (the company). We started out in Italy at the end of 2002, and a while later Fabrizio, the CEO, relocated to the US to found the corporation, where the headquarters of Funambol is currently located. I am the current CTO.

Before that, I worked for a couple of years in a company called ATG, which at that time developed an application server. I was based in Reading, which is a little town close to London, and I acted as consultant for the region.

Before that, I had a very short experience with Compaq in Italy, during a period when they wanted to branch out into professional services on the software side. It was a very good experience for me to get familiar with mobile devices such as PDAs and so on, and I quickly became a reference for that kind of technology within Compaq in Milan.

Before Compaq, I worked at a software house in Italy that developed websites and stuff like that. That company was co-founded by Fabrizio, and I was actually an employee there.

Scott Swigart: Tell us a little bit about the open source project that you’re working on and what the software does.

Stefano: When I was working in ATG, Fabrizio and I had been thinking about what technology would be most important in the next five to ten years. We wanted to start a company that would innovate around that technology.

We wanted to be innovative in many ways–not only in terms of the technology itself, but also about how to develop software, and about the business model. Of course, one of the most evident technology directions at the time was mobile computing.

When we started to think about mobile computing and requirements we could meet to help drive its future, it became evident to us that mobile phones need to have local data storage to run their applications.

A very simple example is the address book, because at that time, but even today, you want to make sure that your phone applications continue to work smoothly, regardless of whether or not it has network connectivity. You must also make sure that your application doesn’t break dramatically if you run out of battery.

Thinking along those lines, we recognized that most applications will be mobilized, so we need a framework to make it possible for those applications to have a local database on the mobile phone, and that they can synchronize with a server.

The server, then, can in turn synchronize data with many other things, including Outlook, CRM, or an enterprise application server. As we moved forward, the first thing I researched was whether there was already a technology or a product to provide this service, or whether we should start from scratch.

At that time, the SyncML protocol had just kicked off, which is basically a synchronization protocol that attempted for the first time to standardize data synchronization among different sources.

After a while, that protocol was adopted by the OMA–the Open Mobile Alliance–which is one of the most important standardization bodies in the mobile area. Since there was already a protocol in place, we decided not to start from scratch and reinvent the wheel. We just needed to implement that protocol, so we focused on how we wanted to do that.

Open source was starting to become more visible and prevalent, and Fabrizio and I had always been very supportive of open source, so we decided to go that route to implement the protocol.

That was the real origin of the product, and today, the open source project has two primary parts. There’s a synchronization platform with a server side component that we like to call a mobile application server, and there’s a client component for the devices that have a SyncML client on board.

On top of these, we developed over time a more complete application suite, and today we have what we call a MobileWe, the open source and multiplatform MobileMe. It is a kind of Apple Mobile Me, but open source, with support for a very broad variety of devices.

Scott: You mentioned a client side portion. Does it include a client type database or is it really more focused on the sync protocol on the server side?

Stefano: Today, we are most concerned with synchronization. That is the reason we mainly took the approach of trying to integrate with existing applications on the mobile phone.

We synchronized the data that those applications already stored in the database. There is one remarkable exception, which is a J2ME email client,which instead implements a full email client for Java mobile; in this case, we could not count on an existing application into which we could plug in, and we developed the storage too.

Scott: What are some of the key applications?

Stefano: Mostly the PIM and email applications that are on the phones–calendar, email, tasks, and usually also notes.

Scott: Tell us a little bit about the server-side component of this and what that synchronizes out to.

Stefano: SyncML is a data-agnostic synchronization protocol, so you can really synchronize everything you want with that one protocol. In fact, our SDKs–both on the server side and the client side–also allows you to begin an application from scratch that is able to synchronize with whatever you want.

In a complex server deployment, the server will consist of many blocks. First of all, we have the data synchronization service, which is physically the block that receives the SyncML requests. This component also includes the synchronization protocols, the synchronization logic, and the connector architecture that allows us to delegate to connectors the actual access to the data.

This allows developers to develop their own connectors, without knowing anything about synchronization or mobile phones. They just implement a very simple Java interface that will be called when the synchronization takes place.

This connector architecture allows the decoupling of synchronization logic from the data access logic technology. We implemented our own connectors for our PIM personal information management database and email.

These will be enough for a simple synchronization scenario.

Scott: I imagine that synchronizations become much more interesting when you introduce the push side of the synchronization.

Stefano: That’s correct. This aspect allows us to make the synchronization completely invisible to the user. It means that if the user changes something on the device, the change is automatically synchronized to the server. If something changes on the server, the change is automatically synchronized to the device. That’s the part that is more interesting.

These are what we call the client server push, which is automatic synchronization from the client to the server, or server to client push, which is the synchronization from the server to the client. In order to achieve this synchronization, we need additional components that are able to detect that there are changes into the backhand.

We call these component listeners, and out of the box we provide the ready made email listener, which we call the Inbox Listener Service. This service is able to monitor user’s inboxes and detect when there are new emails.

There is another service called the PM listener service, which detects changes in our PIM database. When one of these two listeners detects a change, they know to which user the change relates, and they tell the Data Synchronization service to notify the clients used by the user, that there is something new to synchronize. This notification could be done in many ways. It’s very dependent on the kind of device, because different devices and operator networks have different capabilities.

Scott: What are the mechanisms that initiate these server-to-client push actions?

Stefano: We have three ways of doing it. The first is through SMS. If you are an operator, you already have a way to send SMSs, so we can send an SMS as a trigger to start a new synchronization.

Most of today’s mobile phones have a SyncML client on board, so we don’t have to install any software on them. They work out of the box with SyncML and usually also have this kind of notification implemented.

When this type of solution is not implemented, as in the case of Windows Mobile to take a very common example, we provide our client software that is able to address the other two types of server to client push: connection oriented push and connectionless oriented push.

Connectionless oriented push is when the device is able to start in a TCP/IP listener on a particular port and the operator network gives to the device a routable IP address: in other words, an IP address that the server is able to access directly.

If this is possible, then the server connects to the client and delivers the notification. This is our preferred way of doing push, because it’s least demanding on the battery since it doesn’t keep any connection open. When there’s something new, the server connects to the client and tells the client to start the synchronization.

The drawback of this method is that the IP address of the client must be addressable from the server, and not all networks give devices an IP address that is routable. It may not be a public IP address, there might be a firewall in the middle, or in the case of a WAP connection, there might be a gateway in the middle.

In those cases, you are not able to connect the device directly, so our client starts a connection to the server and keeps it open, so that the server can write the notification package when there is something new. The client keeps the connection open in a read mode, and when it reads the notification sent by the server, it can start the new synchronization.

In order to do this, there’s an additional component called the Connection oriented push service, which the client connects to.

Scott: Now that we have that architectural description in place, let’s switch gears a bit and talk about the open source nature of the project. How much of it is developed by you as a company versus by the community? What benefits do you get from having the project be open source?

Stefano: We get many benefits from the open source community, particularly with regard to the mobile space. We serve a big variety of devices and operators in different countries and so on.

One of the most important benefits that we get from the community is people using the software all over the world. They test our software in many countries, and they report issues and successes.

In a more traditional company, you would have to either test in every country or pay a lot of money to one of those mobile testing houses that have a pretty good coverage of the many different environments.

Another area of benefits, of course, is to the code itself. We have gotten very significant contributions, even though I would say that the majority of the code is still from employees.

Of course, that fact is partly because in many cases, people make those contributions after we’ve hired them, using the project as a sort of recruiting mechanism. It’s been a great help in finding and selecting people that want to work with us.

We had very remarkable contributions from developers that wanted to develop something on their own and found something useful in our software, so they contributed to our software–in particular on the client side–to improve the client SDK for example, or to port it to different platforms.

We also get great contributions from companies that are using our software and that want to contribute back code that they have built; they want to work to develop features together with the community.

So you see, the benefit is not only the code that is being contributed, but it is also building an ecosystem around the core of the project. We have a whole architecture that includes the connectors and so on, so people can build different applications based on some pieces of our code.

When people who use our software help us to validate it in another environment, we also consider that a contribution. They use it; they test it, and they help us improve it.

Scott: One of the areas where open source does really well is where there are lots of variables such as different devices and carriers. It would obviously be very difficult, if not impossible, for one software company to support all of those combinations.

You’ve got a great community that really helps test all of those different cases and contribute to the code, and of course, a lot of times it isn’t a major change to get it to support something else. Do you find that someone can contribute a lot to the project by doing just a little bit of work to get it to work better for a particular device?

Stefano: Absolutely, that’s the way we see it, and at Funambol, we try to encourage that as much as possible.

For example, we launched a program that we call the “device sniper,” where we give a very small amount of money to people to test a specific device based on a test script that we provide and to report back to us. This approach has worked well, and we have found a number of issues.

Scott: When someone contributes a patch, how is it looked at to determine whether it is of sufficient quality to be included?

Stefano: For small contributions, patches, and stuff like that, we ask people to file the bug and attach the fix as a patch file (a diff) so that it can be reviewed and then maybe integrated into the trunk.

The core developers review the patch, and if it’s good, they just include it into the trunk; if not, they ask the contributor to make the changes that they think are needed.

For larger changes, the process could be similar, but in that case, the core team will be more interested in having the contributor stay around and maintain it. In some cases, that type of contributor even becomes one of the core developers of the project.

In cases of those larger changes, we need to work much more tightly with them, and so they have to be much more active. They have to present their proposed changes more deeply, and they have to discuss with the other developers the associated design and architectural changes.

When they have developed the trust of the core developer team, we give them the write access to the code repository so they become part of the enlarged team that works on the code, and they can contribute directly into the trunk.

It is a very meritocratic process. Once we see that someone is really committed to the project, and that we share a common set of goals, they can make the transition to become core developers.

Scott: Synchronization in general is a hard problem. There’s a lot of logic that goes into synchronization, especially to handle issues like what to do if the same thing has been changed on both ends. Talk a little bit about some of those challenges.

Stefano: I’m glad you ask, because I think that some people have the perception that synchronization is an easy thing. Assuming it works properly, of course, at the end your address book in Outlook matches the address book on your phone, and so you take it for granted. Working seamlessly just means that it works well, though, and not that it is simple.

There is actually a great deal of complexity in doing this well, because you must not lose data nor create duplicates. Some challenges are related to the protocol itself, and others are due to different interpretations of the protocol or even with bugs that the phones may have.

In order to address this challenge, we implemented what we call the synclet technology. Before incoming SyncML messages get processed by the synchronization engine, they pass through what we call an input synclet pipeline.

The pipeline is basically a pipe of synclets that are executed based on the specific phone that is synchronizing. It gives us a hook to attach specific processing for a specific device. Thus, for example, if we know that a particular device has a bug in handling the addition of new items, then we can fix some of those things into a synclet.

We basically can do the same in output for outgoing messages, so we have an output synclet pipeline as well, such that if we send a message to a device and we know that a particular device, for example, interprets the protocol differently from others, we can fix the message before it gets sent to the mobile.

The same is true for the representation of contacts, for example, or events. There are standards, but there are also different interpretations, as well as bugs. For instance, a vcard may work on one phone but not be accepted by another phone. We fix this kind of thing in our pipeline.

Scott: What about issues around the actual data on the phone? For example, what if I have the contact “Stefano” on my mobile phone and “Stefano” in Outlook?

Stefano: That does happen, and moreover, when you synchronize different devices into a single address book, we need to detect what is actually the same information, even if there are different data.

That’s challenging, because for example, you may detect a person that has the same or a very similar name, but they are not the same person. You might also have different data for different people, such as a home phone number and a mobile phone number.

There are a number of challenges here that people usually don’t see, but that are quite interesting.

Scott: Synchronization has always struck me as a very complicated engineering challenge with a lot of problems that there aren’t necessarily great answers for. Thanks a lot for taking the time to chat with us about what you do and about the project.

Stefano: Thank you.

Bookmark this:

Interviewee: Matt Mullenwegr

In this interview we talk with Matt Mullenweg of WordPress and Automattic. In specific, we talk about:

• Successfully making the culture shift when an open source project is acquired by a corporation
• Scaling skills and physical resources effectively as a project gets larger
• Handling comment spam in blogging environments
• Achieving usability for a technically diverse user base
• The value of an open source community in recruiting people
• The place of blogs in a world of open-ended communication options

Sean Campbell: Matt, could you start us off by introducing yourself?

Matt Mullenweg: Sure. I am a co-founder of WordPress, and I currently lead the development effort. WordPress is open source publishing software that started out as blogging software, but is now starting to be used as a content management system as well.

Sean: There’s been a lot of talk lately about Red Hat’s business model; a lot of people hold them up as the gold standard for a successful model of growth and profitability using open source. There are also rumors that IBM’s going to acquire Sun, and of course, Sun recently acquired MySQL.

Against that backdrop, what can you say about the right way for a corporation to acquire an open source company, especially in terms of getting people to stay with the effort after the acquisition is complete?

Matt: I think that you have to be very cognizant of the existing culture and, ideally, try to change stuff as little as possible. For example, my understanding of MySQL is that 70 percent of people work from home. Trying to change that could be very disruptive.

There’s synthesis with all acquisitions, and it’s important not to break what’s working well. Google kept YouTube with separate offices–basically as a separate operation–because they had already done super well. In general, I think it’s best to change as little possible when you acquire something, particularly if it’s open source.

Sean: Are there areas you think are safer to change than others? It seems that large organizations are particularly likely to want to make changes, for better or worse.

One area where it seems there might be an opportunity there is in the case of a small up-and-coming project that may not have the ideal depth of hardware and back-end resources. How would an acquiring company find the right balance between showering them with unneeded technology, versus dedicating additional, bright people to grow the development side of the project?

In other words, how does the company know how to help, beyond just leaving them more or less alone?

Matt: Depending on how it’s done, making more resources available could be great, although I’d be hesitant about giving them to people directly. Still, having more headcount and infrastructure can spur development.

I don’t know if the guidance here is all that specific to open source, actually, versus general acquisitions, since I think that in the software world, the expectations are the same. In either case, whatever you’re hoping to change or integrate should be one of the first conversations you ever have. That helps minimize the degree to which people feel things like, “Ah, crap. We’ve been acquired, and now I’ve got to figure out their stupid user system.”

If you talk about those issues from the beginning, you avoid making people feel blindsided. There will be places where changes have to happen, but if you say at the outset, “OK, we have 100 million users; you have two. It makes sense to integrate those systems,” then you can avoid unnecessary surprises.

A big part of it is that companies really need to think about how easy it is for people to do things on a day-to-day basis. How easy or hard is it to buy a computer, go to a conference, get a new email account, or bring a new server online? None of those issues really fit into a normal design review process or anything like that, but they’re stuff that people deal with every day.

Little annoyances add up cumulatively, and people’s general job satisfaction can depend completely on whether they have to deal with 1,000 annoyances and huge levels of bureaucracy to do basic, simple things. Particularly if they were very simple before the acquisition and difficult afterward, people will quickly get frustrated.

Sean: When WordPress was growing, how did you scale up the IT expertise you needed alongside the core development skill set? It must have been a complicated transition to go from a strictly developer mindset to managing the large number of servers you have now, and obviously, you had to tackle issues about smart growth, reliability, and uptime, which I assume were not core considerations at the beginning of the project.

Matt: Originally, I and a few of the other early developers were doing all the systems administration, just because we had to. We were lucky in the sense that, a few months into it, we started to get help from a fellow named Barry, who had more of a real systems administration background than we did.

He’s very good about making things super reliable and fast, although he didn’t initially have experience with systems at that large a scale. It was certainly a learning process along the way, but we all agreed from the beginning on commonsense goals like wanting the site to be fast, available, and super reliable, so that was a good foundation.

Actually, to this day, Barry is still our only full time systems guy.

Scott Swigart: Of course, not every open source project is backed by a company that offers hosting as WordPress does. As the company offered hosting of WordPress blogs and that took off and became popular, how much did that drive the development to make sure that this open source project was really being built in such a way that would facilitate it scaling? And how much do you think the decision to host has had to do with the overall success of the project?

Matt: I think hosting could have had a negative impact. We made the decision very early on that the structure of WordPress.com would be basically identical to WordPress.org, which honestly made it more difficult in terms of scaling and a few other things.

At the same time, though, it ensures that recruitment and scalability and everything else that was done to WordPress.com has fed directly into the back-end open source project.

I don’t know how much of an impact hosting has actually had on WordPress’s success, simply because 99.99% of blogs never reach the point where the level of traffic is an issue. On the usability and interface side, it has provided more flexibility, because we are able to test things out in a very time-independent way.

We can do things as a plugin for .org or .com and put it out there and let the market decide whether it is something worth having or not.

Sean: From some of the interviews I have seen, it sounds like you’ve had kind of an evolving relationship with comment spam. Obviously, the problem isn’t going to go away soon. What do you think is the next set of punches and counter-punches in that area?

What do you think the next level is that those folks might try, and looking ahead, how might you try to keep that problem at a low roar or squelch it?

Matt: Luckily, our anti-spam product, Akismet, has remained at a very high level of accuracy. I think the thing that is starting to impact us more is that the spam is being made to look more like human comments. Entries may comment on previous comments, and in some cases, spammers actually pay people to write the entries in.

People don’t realize the extent of this sort of thing. The spammer may leave a compliment on the blog, and the blog owner is flattered. They may notice that the URL looks a little weird, but that seems secondary to them.

Unfortunately, if you are going to keep spam off your blog, you have to be a little bit paranoid and really check all the URLs and everything.

Sean: What’s the next technological piece you are thinking of adding to deal with the spam that really doesn’t look at all like spam, like when you get a comment that says, “I loved your blog,” and the URL is Bank of America without the F? What’s the next logical move in that area?

Matt: We’re not really planning any major technological changes right now, because we’re catching that stuff. The biggest thing needed is not technological, but educational– just informing people, and educating them about what really is spam or not.

Sean: I have to wonder how solvable some of these problems are. Blogging has progressed from more technical people to less technical people, and that’s what you want with technology, right? You want it to become mainstream and pervasive and really a new paradigm.

You want it to be approachable by people who aren’t technical, because they have interesting things to say, too. At the same time, they are inherently not going to understand some of this stuff. So, how do you guys think about that balance?

Matt: We try to treat our users as much as possible like they’re intelligent people, and we don’t try to dumb it down. Our approach is to provide the opportunity for people to enter into it and learn from the system, and they really do.

On one hand, there is the impulse to make everything super simple, to ensure it’s accessible. On the other hand, you have to consider something like World of Warcraft, which is an incredibly complex system that has tens of millions of people paying $10 a month to use it.

Part of the secret there, of course, is that it’s fun, and I think anything that we can make fun will allow people to have a true learning process.

Sean: A related issue is that it doesn’t have to be for everybody–you don’t have to try to make it one size fits all. I’m sure there are people who can’t figure out World of Warcraft, but many of them don’t particularly have any interest in it. That’s OK–it can still be a massive success.

Matt: I think if you try to build for everyone, you are pleasing no one.

Sean: Usability-wise, WordPress has always been great. I don’t claim to know every single widget and option, but I have a reasonably solid understanding of it, and it has never been painful to figure those things out. You couldn’t say that necessarily about a lot of other open source software.

How are you guys structured internally to maintain high usability? Typically, that gets difficult as a project grows, unless somebody’s acting as a benevolent dictator.

Do you have a final launch authority on usability, or do you test it on user groups, or something else?

Matt: One of the most important things is that everyone developing the software uses it every day. That sounds basic, but I think most software actually doesn’t have that. If you’re developing a word processor, you need to write something in your word processor every day, and if you’re developing blogging software, you need to blog every day.

You also have to realize that for some software, that’s really hard. If you’re developing a Sarbanes-Oxley management system, most of your staff are not going to need that at a personal level.

Sean: [laughs] Right. Not all your developers are bankers in their evening jobs, so they’re not going to be able to understand all the ramifications of some UI change.

Matt: Still, in consumer software, that’s typically completely possible, because every developer can also be a beta tester, and I think that’s really important. The output of that is that you get everyone thinking from a user point of view, versus an implementation point of view.

Part of my job is also finding the best people in the world, who can give some really powerful input into what we’re doing. But more, I think you just need to have sort of a set of shared principles that everyone agrees with and everyone can believe in, like a philosophy.

One of our philosophies is to include as few options as possible. We try to make things work by default and not burden the user with lots of options. That translates into everyone thinking about things in a slightly different way, and being able to articulate those things into the minds of every single person adds a bit of nuance that would be lost if we had separate people who did development and others who did design.

Sean: I see companies that build “line of business” apps trying to add more and more design, in fits and starts, to their development process. You’ve got trends like consumerization of IT, and you’ve got a workforce that expects apps to have a certain fidelity. If your users walk in and see some ugly, heinous line of business accounting or HR or travel app that’s been around for 10 years, they’ll say, “You’re going to make me use this?”

In terms of hiring the right people, do you feel that open source, in general, makes it easier to identify candidates that rise above the middle and get to the top? In corporate development, you can say you worked on a project, but it can be difficult to communicate what you did and to prove it and to really show a long stream of communications in solving problems and interacting with a community.

It would seem like that makes the job of hiring easier than if you were trying to draw on a team of closed source candidates for a closed source project.

Matt: I agree.

Scott: What are some examples of how that has panned out for you?

Matt: When you’re doing any sort of hiring, if it’s a person you’ve worked with in the past, you know how they’re going to operate in difficult situations. In reality, it’s more important than anything on a resume to know how they’re going to interact with other people and the product, day to day.

Open source allows anyone to jump into that process, without any gatekeepers, and the pool of people is often broader and deeper. There’ll be people in Uruguay or Brazil or Mexico or some place in Europe, and when it comes time to hire, you’re like, “Well, so and so would be perfect.”

And it’s a complete meritocracy.

Sean: In a commercial environment, there’s often the supposition that if somebody comes from XYZ company, they are used to a certain corporate culture, and that often becomes a consideration in the hiring process.

When an open source team is hiring people from other open source teams, does a similar sort of interplay tend to happen? For example, does it come into play when you’re looking at a given effort, how that project tends to organize its discussion, its voting process, and its way of arbitrating between usability versus technological issues?

Matt: Absolutely. For example, we’ve worked with some people who came from the Mozilla project, and they have a certain way of doing things that involves a little more bureaucracy than most open source projects, just as a result of their size.

A lot of factors like that contribute to someone’s background, although there’s never a direct correlation that tells you that, if someone worked on X, they’ll be perfect (or not) for Y. There’s always a learning process as they’re becoming part of the culture, so there’s no secret sauce there.

Sean: Considering various ways of communicating on a computer, there’s blogging, and Twittering, and so on. It seems that the average user probably has a limited number of ways they want to go about communicating on a given day, and most of them adopt just a few; e-mail and IM are the other obvious examples.

Do you see any ripples in the future, where people might be getting closer to moving away from blogging toward something else? Maybe toward something that’s more rapid fire, or less asynchronous?

I’m not trying to position Twitter or any other specific option in that place, because I see them as more complementary than anything else. At the same time, I also see that with all the different options people have for communication, eventually Joe User will get saturated and be forced to pick just a couple.

Matt: I think that complementariness is foremost right now. Twitter and WordPress aren’t super tied together, but you can imagine that every blog post I do should be posted to Twitter, and probably some of my Twitters should go back inside my blogs.

The reason I personally like blogs is because they serve as the best online profile I could create. More than my Facebook, my Friendster profile, or any of these others, my blog is really a representation of me. Because I have complete control over it, it’s a bit of me. By now, it contains seven or eight years of history of things I was interested in, including photos and all kinds of things.

As I interact with things like Flickr, Twitter, and Facebook, I find that I want to aggregate them into my blog, because that is sort of like my home base. I think that people online will always need a home base like that, although the other things that they interact with will be constantly changing.

I did not see YouTube coming at all, but it’s hard to imagine something that has had a bigger cultural impact in the past couple of years, and it’s another way that people now share, aggregate, and interact with one another.

Sean: Blogs have very rapidly gone from being a new thing that geeks did, to spreading out to non-technical people, to really having an impact on politics, policy, and news. In some ways, people probably view blogs as being kind of mundane.

You don’t run into a blog and say, “Oh, what’s that?” What still surprises you? What have you run into recently around the notion of blogs that is fresh and new, and catches your attention?

Matt: I am always fascinated by things that capture peoples’ hearts and minds. In the past couple of years, Twitter has captured a huge amount of press, and I think a good chunk of peoples’ thoughts, but if you compare it to Facebook or YouTube, it’s a drop in the bucket.

When you can make the interactions with the application or website into kind of a game, that’s when I think people start to really make it part of their everyday life. If you’re a part of the game, you’re also facilitating communication or imparting information, or people are learning or reading, and that can be very rewarding.

Facebook in particular has incorporated game-like elements to the nth degree. It is literally addictive. I have peers–particularly those who are still in college–who spend hours per day on it. How is that possible? It’s ridiculous.

Sean: Right. It’s not a game in the World of Warcraft sense, but it has a similar semantic feel that makes you want to play along. You get drawn into the interface, and even if the interface has a hiccup or two, you just drive around the pothole, because you want to play the game.

Scott: That is what we need to solve the recession–we just need a game-based economy.

[laughter]

Sean: Well, I’m glad we solved the economic crisis. This conversation has been time well spent.

Matt: Thanks. I enjoyed talking to you both.

Bookmark this:

Interviewers: Scott Swigart and Sean Campbell

Interviewee: Denis Lussier

In this interview we talk with Denis. In specific, we talk about:

• The role of PostgreSQL in the enterprise database sphere
• Building performance to expand upon the capabilities of LAMP
• Creating a hybrid open source/corporate business model
• The role of various license types in enabling business models
• Sun’s relationship with PostgreSQL

Sean Campbell: Please give us a little bit of background about yourself, both with and prior to EnterpriseDB.

Denis: Sure. In 1994, I founded a small boutique system integrations firm called Fusion Technologies that grew to be about 250 people by 2004. In 2003, as a bored CEO running a bigger company, I started an open source practice area at Fusion Technologies. That is where I stumbled across PostgreSQL and MySQL.

I came to realize in that process that PostgreSQL was a superior database to MySQL, although it was not as popular as MySQL. And neither of them were really all that popular compared to Linux in the operating systems space or compared to JBoss in the app server space, just because anti compatibility was not that big of a deal in the database space, since because nobody really has it and nobody cares about it.

Ultimately, I spun off EnterpriseDB as a separate company. I took venture capital financing in early 2005, and we have done several rounds since then, producing what you now know as EnterpriseDB. I sold Fusion Technologies to a larger company, so I am not involved in that anymore.

Sean: I was a database guy for a long time, and I wrote my share of code, although Scott wrote more than me. Where does EnterpriseDB fit into the grand pantheon that includes everything from DB2, to Oracle, to MySQL?

Denis: We are definitely going after the enterprise space, because we are built on rock solid PostgreSQL, which is enterprise class. If you look at the headlines, MySQL might be referred to as “The world’s popular open source database,” while PostgreSQL will be referred to as “The World’s most advanced open source database.”

PostgreSQL appeals to the open source UNIX and BSD types of guys, and it is the engineers’ choice for an enterprise class open source database management. MySQL is not silly enough to go directly after Oracle–even now that they are with Sun–and neither are we.

Just as you didn’t rip and replace Solaris to go with Linux, you don’t just rip and replace Oracle and go with EnterpriseDB, because we are PostgreSQL with Oracle compatibility. Migrations like that can take years.

Even though going after the enterprise market takes years of development, we have an enterprise class product that is appropriate for the enterprise, especially for up and coming companies. For existing companies, we have got to get in with some developmental type projects and earn our stripes through true enterprise class replacement, which again takes years.

Whereas MySQL is sort of positioned as the web database, we do a lot with scale out and so forth, for companies like hi5 and Skype. Guys that are doing huge things with PostgreSQL understand what you have got to do to truly scale out a database, which is PostgreSQL’s strength.

You also have to scale up the database in terms of OLTP type transactions. Huge numbers of users and transactions are PostgreSQL’s traditional strength. If you want to do industrial class scale out, PostgreSQL also excels at that.

We fancy ourselves as “The PostgreSQL Company.” We are the largest sponsor of PostgreSQL core committers, and we also have a value added product on top of that, which is EnterpriseDB’s Postgres Plus Advanced Server.

That offering includes all of PostgreSQL, plus advanced features that enterprises are interested in–hence the name. Other than a couple parts of it being closed source in much the same way as Red Hat Enterprise Linux versus Fedora, anyone can kind of do whatever they want whenever they want, without paying us.

Sean: It’s an interesting time to be opposite MySQL, with Sun buying that project to essentially buy a letter in the LAMP stack as a play for some open source credibility.

Denis: That’s not working out so well with all of MySQL’s executives leaving Sun.

Sean: I think they found an HR department that was not to their liking, although I have a feeling that the situation stems 25% from the entrepreneur going to work for a large company and 75% from Sun.

Denis: We all love Sun and Unix and stuff like that, but Sun and software don’t do so well, and the MySQL guys in a big corporate culture don’t do so well, so there is nothing particularly surprising there.

Sean: When we talked to Dries over at Acquia, who is smaller than you guys by far, he said his goal is essentially to be the Red Hat to Drupal, the Red Hat to portals.

It kind of sounds like you want to be the Red Hat to the database stack, modifying people’s perception of an M in that LAMP stack as you deal with more scale out in Web 2.0 and aid them in moving beyond being just small sites that run PHP in MySQL.

Denis: There is definitely an element of that. Among the features we are about to announce, there is one called “Infinite Cache.” Basically, we take memcached, and rather than putting it in front of a database and requiring you to change your application, we take it as the basis.

It is BSD, so you can do whatever you want with it. We take it as the basis and put it behind the database so you can still do your relational queries. We take what is effectively a hypothetical database and put memcached behind it so that we do the caching and you don’t have to worry about doing special things in your application.

We use a distributed cache behind the scenes. When we say a Web 2.0 database, what that really means to us is that you can do stuff like memcached, but we make it transparent behind the database.

If you want to use a huge scale-out database, like Skype or hi5 does, and you use this technology behind the database, it will automatically cache everything that you put in the distributed cache.

The remaining part that isn’t the read–maybe 90% of a typical web based application–is an OLTP application. PostgreSQL and, therefore, EnterpriseDB based on it, is one of the best databases in the world for that.

The architecture underneath is very, very similar to Oracle in terms of how you do distributed locking and how you do row level locking versus page level locking, and all that stuff. So, we have the traditional strengths of Oracle in that area, but we leverage various open source projects when creating Postgres Plus Advanced Server. It is similar in approach to Red Hat enterprise Linux.

Sean: Assuming the analogy still holds true, Red Hat has been doing really well. They had a good quarter for a technology company in a quarter where many did not. They have been growing continually, but particularly in the last quarter. What would you like to follow or not follow in terms of their example?

Denis: They obviously defined themselves as an enterprise open source company, but they try to combine the best of the open source world and the non open source world. We seek to follow that; for example, they defined the successful subscription model that we follow, and there is a lot to follow there.

Ed Boyajian, who is our CEO now, ran North American operations for Red Hat, so we have an ongoing affinity for Red Hat. Everything that we do moves towards open source, but we don’t rush making it open source before it is time, because we want to get the economic benefit out of some of our features.

Our features move to open source at the rate that makes sense. We don’t have a policy that says everything must be GPL2 immediately. We want to emulate Red Hat’s success, but frankly the database market is somewhere between six to eight times bigger than the operating system market.

We would hope to be as successful as Red Hat someday, but we dream of being even more successful because of the bigger market.

Scott Swigart: How do you make that decision about what makes sense to keep in your proprietary offering and when it makes sense to push those patches out to PostgreSQL?

Denis: To speak in sweeping generalities for a moment, if we have got a feature that we have made closed source, and somebody else is creating a competitive feature that would cause the source code to fork unnecessarily, we will contribute our feature.

Sometimes, we open source defensively like that. We also open source sometimes because we have a feature that is going to be enterprise class, but we need a much wider QA community before we can make it enterprise class. That’s more of an offensive reason to open source.

In general, the PostgreSQL community is all about anti compatibility. PostgreSQL is the strongest anti database in the world. Our customers who care about Oracle compatibility features are obsessed with the money saving aspect of open source and that it be enterprise class based on open source.

They understand that some of our Oracle compatibility features are not open source. To open source those features, we would really have to try and force them on the community, because they don’t really want them anyway.

The decision about whether to open source a specific feature is sort of complicated, but most decisions we make about how, when, and where to open source something make sense if you break it down into those three categories. The decision that you make and the timing become self evident.

Scott: Does the fact that PostgreSQL is under the BSD license help to enable your business model? In other words, GPL wouldn’t really work for what you are doing.

Denis: It helps to look at things in terms of generalities. There is GPL that is not controlled by a single commercial interest, such as Linux. There is also GPL that is dominated by a single company, such as JBoss and MySQL.

Then there is BSD. There is BSD in Apache, so the licenses are more permissive, so you can commercialize them and make your own decision as to how, when, and what you open source.

I chose PostgreSQL when I founded EnterpriseDB because, from an engineer’s perspective, PostgreSQL is a better database, but from a license perspective, it also provides flexibility for our business model to combine open source vision with commercial pragmatism.

Scott: How has it been beneficial for PostgreSQL itself, in terms of its evolution? I know there are some people who would say that a GPL license would be best for a given project, because people have to share their derivative work.

On the other hand, there are people who will say that you have to make enough money to stay in business, and that by growing a company, they are going to be able to contribute a lot more to that core project.

Denis: Both models can and do work. They have their strengths and weaknesses. You can’t really directly sell a subscription to something like PostgreSQL, the Apache server, Tomcat, or anything with an Apache type license, a BSD type license, or an MIT license.

You can be a professional services company that specializes in that, but the actual software is very high quality and it is free. That makes it harder for a company to make money on it, but on the other hand, it is easy for a lot of companies to get behind contributing to it, because they want to invest their time and they want to use it for free.

Therefore, the Apache, BSD, and MIT licenses encourage people to trade time for money. GPL, on the other hand, lets a company get behind it. I would argue that PostgreSQL is the world’s best open source database and MySQL is the world’s most popular. It was the most popular because there is a commercial entity behind it.

We don’t control the PostgreSQL community that EnterpriseDB is a big contributor to. It is a very independent community. Several of the core committers work for us and so on, but we don’t control it, much as IBM doesn’t strictly control the Apache community. They are a major contributor to it, but they don’t control it, and they don’t need to.

You can sell WebSphere, whereas you can’t sell Apache or Tomcat. MySQL is successful and it is a good product (although, naturally, I think that PostgreSQL is better). We think that models like PostgreSQL, Apache, BSD, and MIT can be successfully commercialized by a company that understands open source, the way IBM does. They can have some closed source aspects of their product and still be a good open source company.

IBM is an investor in our company, and we frankly try to emulate their success in the open source market.

Scott: Sun used to be, or maybe even still is, a fairly big supporter of PostgreSQL. What is that relationship like?

Denis: Sun is trying to be a good open source company in the model of IBM, but once they spent a billion dollars on MySQL, they slowly and steadily backed away from PostgreSQL, which is what you would expect.

We are steadily getting into the Solaris space ourselves, rather than just doing a third party contract through Sun, which has basically expired.

Scott: Who is your target customer? Are they the kind of people who would have bought Oracle in the past, but just don’t see the justification for spending that kind of money for an enterprise class database?

Denis: There are startup companies that see the value of PostgreSQL but want people who are true experts at it. They are not going to build it themselves, and they get all the build optimizations and cache update service, and that kind of stuff.

There are also companies that are spending, in some cases, tens or hundreds of millions of dollars on Oracle per year that want to, over time, decrease that dependency. And then you have everything in the middle.

We recognize that big companies can’t just replace Oracle or SQL Server overnight, but we think we have something that they can move toward over time, and we have been very successful with that model.

In fact, that is very similar to the Red Hat Linux model. They targeted Solaris, but you didn’t just rip and replace Solaris eight years ago, even though you might now.

Scott: Well, we have come to the end of our time, but I want to thank you for a great conversation.

Denis: Thank you. I enjoyed it.

Bookmark this:

Interviewee: Ricky Zhou

In this interview we talk with Ricky. In specific, we talk about:

• Identity of the Fedora community and its relationship with Red Hat
• Relationship between Fedora and other distributions
• Upstream projects as they relate to Fedora
• Public opinion about the Fedora project
• Open source involvement in the software industry and university sphere

Scott Swigart: To get us started, give us a little bit of an introduction on yourself.

Ricky Zhou: OK. I’m currently a freshman at Carnegie Mellon University. I’ve been contributing to Fedora since about March of my junior year. I’m mostly involved with the infrastructure team, which runs the servers that run Fedora and I’m kind of the leader of the website team, which is where I started out. I also do some packaging, among other things.

Scott: How did you get started with Fedora? Why Fedora versus some other distro, and which areas of contribution did you get started in with the project?

Ricky: I had been a Fedora user for a while before I got involved. My first involvement was with the Fedora website. I noticed that the Fedora wiki CSS had some browser issues, so I subscribed to fedora-websites-list and sent a patch to fix it.

It just so happened that this was right around Fedora 7. At this point, the Fedora website was running entirely on a MoinMoin wiki, which could not easily handle the load on release day. The team was working on some static HTML pages to lessen load,, which I started helping out with. After I helped out there for a while, Mike McGrath the Infrastructure team leader sponsored me into the sysadmin-web group, which runs Fedora’s application and proxy servers, and that’s how I became involved in the Infrastructure team.

Scott: Talk a little bit about the areas where you’re focusing and contributing today. You’ve mentioned the website. What are some of your current areas of interest around the Fedora project?

Ricky: As I said, I’m probably most active in the infrastructure team. I worked on the latest rewrite of the Fedora Account System a few months ago, for example. We’ve worked hard to make the signup process easy for new people looking to get involved with Fedora. I also help maintain and patch some of the other web applications that we’re running.

For example, one thing I’ve been very interested in is Transifex (transifex.org), a web application we are using to accept translations. I’ve been helping to keep these applications up and running.

Scott: What’s your impression of the Fedora community? Obviously, there are people that work for the Red Hat company that help out with Fedora. There are also lots of people who aren’t Red Hat employees who work on it, of course.

Give us a little bit of the flavor of the community and the kind of people who are helping out with the Fedora project.

Ricky: The divide between Red Hat and Fedora is actually pretty transparent. In fact, a lot of Red Hat employees that work on Fedora don’t even use their Red Hat email accounts. Before Fedora 7, back when Fedora was divided into Core and Extras repositories, there was a significant divide, as only Red Hat employees were able to commit to Core packages. Since then, a ton of work has been done to merge Core and Extras to put all community members on an equal footing.

The Fedora community is also very open. For example, in infrastructure, we exclusively use open source software, and furthermore only software that has been packaged for Fedora. This ensures that all of the software packages we’re using will be useful to the entire Fedora community as well.

In general, we put a lot of emphasis on building the systems so anybody can get involved, and the amount of work that people are willing to put into this has really impressed me.

Scott: The difference is interesting between Fedora and something like Ubuntu, which is very focused on users, who may be fairly non-technical. Fedora, on the other hand, seems to be very focused on the community that actually develops and packages the open source.

That’s not to say that there’s not due attention paid to usability as well, but it seems like the focus is a lot more on growing a vibrant community of contributors. Another distro like Ubuntu may be a lot less focused on the actual contributions and more focused on the end user experience.

Do I have that right? And as a related question, what do you feel gives Fedora its essential identity, versus some of the other popular distros out there?

Ricky: I think your description is pretty accurate. There have been a lot of discussions within the Fedora community about exactly who our target audience is, although I don’t think anyone has made a definitive statement there.

Our ambassador team has been doing a lot of great work in trying to get people involved in Fedora, and recently, some Fedora contributors have started holding IRC classroom sessions, which includes some classes designed to help people gain the knowledge needed to contribute to Fedora. I do believe that the best user is someone who has recently been involved in helping build the project, and it is definitely a goal of the project to try and convert the casual user into a valuable contributor.

Scott: If you take a look at where Fedora is popular, my sense would be that it’s particularly popular with developers. That’s probably especially so for developers who are writing software for Linux that’s going to run on Red Hat or Cent OS or something like that.

It may also be popular with IP administrators who are really well versed in the Red Hat products in the data center, computer science students like you, and so forth.

Do you have a sense for where Fedora is a popular distribution and for a more or less typical profile of the people who tend to gravitate toward it?

Ricky: In one sense, Fedora tends to be a showcase for the newest developments in software. Hobbyists are interested in the vibrancy of the development process around Fedora. On the other hand, Fedora isn’t just a distribution; it’s also an umbrella for a lot of development that benefits other projects.

For example, we have the EPEL project, which takes Fedora packages and makes them available in a repository for CentOS and Red Hat Enterprise Linux users. A lot of people outside of the developer/hobbyists audience have really benefited from that (for example, we use a lot of EPEL packages within Fedora’s Infrastructure).

Scott: Another thing that’s always interesting to us is the relationship between a community distribution and a commercial distribution. You’ve got Fedora and Red Hat Enterprise Linux. You’ve got openSUSE and SUSE Linux Enterprise. You’ve got OpenSolaris and Solaris. That’s a UNIX distro and not a Linux distro, but they’re making it feel very Linux like.

What’s your impression, having spent a lot of time on the community distro side of things, about how community distributions and commercial distributions tend to relate to one another?

Ricky: Right now, as you know, Red Hat Enterprise Linux comes from a version of Fedora (so Fedora is upstream for RHEL). The goals for RHEL and Fedora are slightly different though. Fedora has a support life (as in, we keep releasing package updates) of 13 months, and we tend to focus our development efforts on displaying the newest developments in open source software, RHEL is focused more on stability, and each release is supported for seven years. As a result of this, we run a mix of Fedora and RHEL in Fedora’s Infrastructure.

Scott: Talk a little bit about Fedora as a distro and its relationship with the upstream projects. It’s always interesting to hear about the relationships between distributions and the upstream projects they are associated with, which have their own cultures and communities.

Obviously, when people are doing work on the distro, a lot of the work they’re doing is actually submitting patches to GNOME or the Linux kernel or other myriad projects. Describe that relationship in the case of Fedora a little bit.

Ricky: A central philosophy of Fedora is that we are committed to contributing as much to those upstream projects as possible, as opposed to keeping patches on our side—this is made in our packaging policies, but also manifests itself in other areas. I mentioned Transifex before; one of its benefits is that it is designed to commit submitted translations directly to upstream source repositories, which ensures that translations are sent where they can benefit the largest group possible.

Even in the Infrastructure team, I see a lot of members who are actively involved upstream for a lot of the Python software that we use. One nice benefit of this is that there’s always some expert to turn to whenever I have a problem with something.

Scott: I don’t know if it’s entirely accurate, but we’ve heard people go far as to say that the goal is that if it isn’t upstream, it isn’t in Fedora. In other words, Fedora tries to stay very true to what’s upstream and views that as the best way to build a distro. They try very hard to work with the upstream project and not carry a lot of distro-specific patches.

With a lot of the other distros, that’s not always the philosophy. Other distros may feel that if it’s in the interest of their users or community, it’s fine to carry distro-specific patches. They may try to get them upstream, but at the end of the day, if it fixes a problem for their users, they’ll carry their own patch. Talk a little bit about that philosophy.

Ricky: This philosophy makes a lot of sense to me. In most cases, if there is a change that would benefit our users, these changes would benefit a far larger audience if pushed upstream. Apart from this, there are many practical reasons to avoid having heavily patched software. In many cases, bugs can be more useful and easier to debug if we are running identical code to what’s in upstream. By resorting to carrying Fedora-specific patches, we can lose out on a lot of opportunities to help out upstream projects.

Scott: This is a pretty different topic, but I’d like to ask whether you feel that Fedora gets the respect it deserves.

When I ask that, I’m thinking of times when I see a new Linux distribution come out that touts its great wireless support or great support for EBDO, and so forth. The distro’s community may be very proud of the work that it has done on something like the network manager, but I wonder if they keep in mind that the project manager matured in a distro like Fedora that started including it very early on. SE Linux showed up in Fedora very early, for example, too.

A lot of these projects later become mature and popular and show up in a lot of distros, but many showed up in Fedora first. Do you feel like Fedora gets the acknowledgment it deserves for helping to make those projects mature?

Ricky: I think that Fedora definitely gets credit for that. If you look on a lot of news sites, you’ll see that a lot of people are fairly aware of how and where things have come from. Of course, there are plenty of places where people are off the mark, but that’s to be expected, too. Overall, Fedora does have a good reputation for being an early adopter of many useful features. I’ve seen people mention in a few places that a lot of software has improved and stabilized a lot after being included in Fedora.

Scott: From your vantage point as a computer science major at Carnegie Mellon, what do you see in terms of the thinking around software? Software’s been dominated by proprietary software vendors, whether it’s IBM, Microsoft, or Oracle; it’s been licensed, and people have had to pay for it.

As the open source community has become more visible, there has been an increasing amount of software available that’s completely free, whether it’s Fedora, openSUSE, Mandriva, or many others.

Obviously, a lot of businesses are also wrapping themselves around open source, providing value in the form of the peace of mind that it’s fully supported or various types of add-ons that maybe aren’t completely free and open source.

Red Hat fits in there as a pretty pure play open source vendor, whereas some companies have much more of an approach where they have a free community edition, but also an “enterprise edition” with lots of additional features that aren’t entirely open source.

In your environment at Carnegie Mellon, what do you find to be the prevailing philosophies around software and how it should be made available?

Ricky: In some of my introductory classes, we are definitely taught how to use open source tools for development, but I’ve noticed that there is much less of an emphasis on the ideology of open source. On one hand, this is something that I’d like to see more of—I know that there are some higher level classes where students are required to make significant contributions to open source projects. I know that some people in Fedora have shown a lot of interest in pushing open source in college education, one of the results of this is http://teachingopensource.org/, which I know at least one of my professors is keeping up with.

On the other hand, it’s possible that some of the ideals of open source are already somewhat ingrained into my generation. At CMU, I took a humanities course about intellectual property, and during the many of our discussions, I found that almost all of my fellow students support and understand the general ideas behind open source software.

Scott: Do you find different levels of corporate involvement at the university level among either the companies that have wrapped themselves around an open source business model, like Red Hat, Novell, or Sun Microsystems, as opposed to companies that are more proprietary like IBM, Oracle, and Microsoft?

Do you find that some of these are more present at the university than others, in terms of actually interfacing with the university to have business-academic partnerships and that kind of thing?

Ricky: I’m not sure whether there’s a difference in terms of campus presence depending on whether companies have more or less of an open source involvement. The main corporate presence that I know of on campus is during the several job fairs that we have, and I haven’t really heard of that many open source companies being at those.

I guess in many cases, open source companies can find a lot of prospective employees by looking at who’s active in the open source community (for instance, I’m currently an intern at Red Hat continuing the same work that I’ve been doing in Fedora). I would certainly love to see more open source companies getting involved with universities though. One recent example I’ve seen of this from Red Hat is POSSE (https://fedoraproject.org/wiki/Professors_Open_Source_Summer_Experience).

Scott: OK. Well, we’ve come to the end of our time. This has been a great interview from my end. I appreciate you taking some time to chat today.

Bookmark this:

Interviewee: Krishna

In this interview we talk with Krishna. In specific, we talk about:

• The birth of Novell-Microsoft NOS interoperability
• Participating in the development of key Windows technologies
• Making Linux and UNIX first class citizens in a Windows network.
• The relationship between Likewise and Samba
• Key aspects of Active Directory functionality for UNIX/Linux
• Making the decision between making a code base open source or closed
• Deciding which features to offer for free in a multi-tier model
• The role of Group Policy in Likewise

Sean Campbell: Can you give us a little bit of background on you, your relationship to Likewise, and the company itself?

Krishna Ganugapati: I started my career in the Windows NT Development Group at Microsoft in 1993, straight out of college. My specialty was in distributed systems and networking. My personal dream was to see a mainstream, industrial strength desktop operating system in the consumer space. At the time, there was probably no better place to work on operating systems. I got to work in Dave Cutler’s group, who joined Microsoft in 1989 to build Windows NT. I think it would be safe to say that the experience in the NT development team would be hard to top.

In 1993, the NT group had 90 people. There were 30 of us in the Windows NT Graphics and Printing Team led by Leif Pedersen. There were also 30 people in Windows networking led by Dave Thompson, and there were 30 people in the Windows NT Base team led by Lou Perazzoli. There were probably another 90+ people putting together the basic foundation of what would become NT and the next generation of Windows operating systems. It was an incredibly gifted group of people to work with. I suspect that someday, people will compare Dave Cutler to Kelly Johnson or Robert Oppenheimer

My goal was to work with all three of Dave Cutler’s development managers. I started as the NT print spooler developer in Leif Pederson’s graphics and printing group. In early 1995, I switched groups to join the NT networking. I ended up working for two out of three development managers, so that was pretty good. When I went to work in the Window Networking Group, I joined the beginnings of what would end up becoming the NT Distributed systems group. Our sub group was called the Mars Group. The Mars Group, basically, was a group that focused on NetWare, around 1995 or 1996. Mars being the red planet (Novell had this big red logo).

Sean: NetWare was probably the dominant network operating system at the time, right? How did you approach that competitive situation?

Krishna: That’s correct; they had a significantly greater market share in the NOS business than Microsoft. There were two products that we started there, one was called Client Services for NetWare (CSNW), and the other was called File and Print Servers for NetWare (FPNW).

Our main goal was to build technologies to make Windows NT work with the NetWare file server, and the core problem was how to make Windows NT file clients talk to Netware File servers and how to make NT file servers look like NetWare file servers.

That was important, because Novell in those days would rather not see that happen. Novell being the dominant network operating system, the easiest way to shut out Windows NT and Windows 95 was to not provide file system client software to talk to their servers. CSNW and FPNW were interoperability products that allowed Windows NT clients to be first class citizens in Novell networks. The FPNW product was particularly compelling in that once your Windows NT clients could talk to a Netware File and Print Server, the FPNW add-on would now make a Windows NT Server system serve as a replacement for a NetWare server. I believe that the CSNW and FPNW products were critical in the battle for the NOS supremacy. There was a team of five of us in the Mars group. I should add that my job was not directly related to CSNW and FPNW. I was designing a directory agnostic client side directory API. Novell had moved beyond file and print services and had just released its Novell Directory Services 5.0 (NDS for short), now called eDirectory. We, at Microsoft didn’t yet have a directory at all! How do you then compete with a directory product when you don’t have a directory?

My project ultimately became Active Directory Services Interfaces (ADSI). ADSI’s strategy was about building a client side multi-provider architecture that would provide a common directory service programming API at the upper edge, and individual providers would communicate with a variety of directory services. We wrote providers for Windows NT (we made the NT SAM and even local machines look like directory services), Netware 3.12, NDS, and LDAP. Later on, ISVs wrote providers for all sorts of data stores and backend. The upper edge of ADSI was IDispatchable, which meant that Visual Basic programmers, COM programmers, and .NET programmers today could program any directory service. The neat thing was that applications written to one directory–say NDS–could easily be made to work against another directory, even a directory that hadn’t yet shipped! Active Directory still hadn’t happened.

ADSI was one of the first client side component object model (COM) libraries that would actually make it to the mainstream in the NT platform. ADSI debuted in NT 4.0 in 1996. Eventually, it became the API to Active Directory.

Around late 1996, the Cairo project was pretty much shutting down. Consequently, the next generation–which would then become NT5 (Windows 2000)–would basically absorb and assume the technologies that were in Cairo, and those technologies would end up becoming part of it. Cairo was supposed to the next generation operating system with distributed systems out of the box. At this point, the Windows NT Networking Group subsumed Cairo’s distributed system technologies and the combined Networking and Distributed Systems Group came under Dave Thompson. This is the group that delivered Active Directory. That would make Dave pretty much the father of Active Directory.

By about 1998, ADSI was pretty much done and I was ready to move on to something else. The Networking and Distributed System Group had grown so large that it had become two groups. Jawad Khaki, whom I’d known for quite some time, was leading the Windows Networking Group. I joined his team and took ownership of the Windows IPSec project. Microsoft was in a joint collaboration project with Cisco to build the first widespread implementation of IPSec. The project had hit some roadblocks, and they brought me on as someone with a reputation for moving projects along.

I ended up shipping Windows 2000 with two projects under my belt: IPSec and ADSI.

Sean: In that timeframe, wireless security would have been a hot topic. Did you have much involvement with that?

Krishna: Like all good things, my involvement into wireless technologies was accidental. After Windows 2000, the networking group was working with a variety of partners implementing the 802.1x protocol. The 802.1x protocol was designed for securing Ethernet switches (layer 2 security). Coincidentally, the same protocol was used for securing wireless 802.11 networks in the enterprise. For some reason, the IPSec group and the team that had begun this effort were merged and I became the development manager for both efforts. As time passed, the challenge of securing the wireless link took on the problem of automatically detecting and configuring your wireless enabled laptop to an available access point. Access points could be non-secured, secured with a key, secured with 802.1x which required certificates on the client machine and the access point, and finally secured with the end user’s Active Directory credentials. The wireless security problem had now morphed into a buffet of cryptic configuration choices for the end user.

To solve this problem, we put together a new piece of software called wireless zero configuration. The goal of wireless zero configuration was to ensure that your laptop could seamlessly detect and connect to any available access point. The zero configuration system would pop up a UI if the laptop detected a new access point and prompt you to connect. This was about six months before Windows XP shipped.

The Windows Client UI guys were really unhappy with us, because they didn’t know that we’d built this UI. But the cool thing was that wireless zero configuration made it into Windows XP, and it was touted as probably the one of the most game changing features in Windows XP. We were even featured on the Windows XP container: a singular honor! So now you know how the Windows XP wireless zero configuration came to be.

Sean: That’s a great story. What other common Windows technologies did you have a hand in during your time at Microsoft, and how did you eventually make your way over to the open source side of things?

Krishna: I spent my last two years at Microsoft as an architect in the Windows Real-time communications group. RTC had become the next major focus at Microsoft. The RTC group would integrate the erstwhile Hailstorm group and the NetMeeting folks (the Exchange Real-time communications group). All three groups had very different perspectives on where real-time communications should go. My job was to help build consensus and reconcile three different perspectives into one cohesive vision for real-time communications. The Windows RTC group believed that the future of real-time communications would be around standards-based SIP for controlling and signaling for disparate real-time communication streams. Ultimately, SIP did win the day at Microsoft and it is very gratifying to see that the Office Live Communications Group has become one of the leading providers of SIP based technologies.

When Windows XP shipped in 2003, we had that mainstream industrial-strength, consumer desktop operating system. I decided it was time for me to try my hand at building a company. So I left Microsoft in 2003 and started incubating a startup of my own and spent two years doing that. We had a really neat value proposition of building a high-end storage augmented residential/small business gateway. But we were unable to secure funding and I was beginning the process of winding down my company.

During that time, I met with Cameron Myhrvold and Richard Fade of Ignition Partners. They were invested in a company that was trying to accelerate the adoption of Linux into Windows-centric networks by building easy-to use administration tools for Windows system administrators to manage Linux file print and web servers. Part of Likewise’s initial value proposition was about extending Microsoft’s MMC management infrastructure to do this. MMC was technology designed in the Windows distributed systems group; the underlying distributed system technologies were things that I was familiar with and so I joined Likewise in December of 2005.

Around the time I joined Likewise, we changed the value proposition to be more ambitious: to making Linux systems first class citizens in Windows networks. For me, the parallels were hard to miss: in my first act, I was involved in making Windows clients and servers first class citizens in a NetWare NOS dominated intranet. This time, I would be involved in making Linux clients and servers first class citizens in a Windows dominated intranet.

Sean: If I understand Likewise’s business model, it seems a bit as if Samba, for example, had a company sitting on top of it, kind of the way Aquia sits on top of Drupal. Red Hat is obviously a much longer standing example of that, on top of the Linux kernel, building out their own distributions.

Likewise seems to draw from this idea of taking something that’s out there in the open and finding a way to build a model on top of it, and providing some type of open source solution that customers can use to get familiar with the product and deploy it in their environments. Then you also offer an enterprise product that sits on top of that.

To take the example of Samba again, that’s one of the things they don’t have. They’re sometimes criticized for not having the same level of GUI support as Windows, although that’s obviously not the Samba project’s specific goal.

Do you consider Likewise to be at all similar to Samba? It’s a different type of networking service you provide to an organization, but you also provide a level of enterprise tool support that they don’t, whereas they might just say, “Well, that’s what the config file is for.”

Krishna: Your observations are very insightful. And yes, the original value proposition for Likewise was just that. Linux had Samba for Windows-interoperable file and print and Apache for web serving. While Likewise, the company is very distinct and separate from Samba or the Apache foundation, the thesis was to be a company that could provide business solutions around technologies provided by Samba. Samba provided a file server and a print server. However configuring Samba could be very complicated. In this sense, Samba is more of a set of technologies that an OEM or solution provider needed to adapt for customers. As you’ve noted, Samba does not provide any GUI style tools. Likewise’s original goal was let Windows system administrators be able to point to a Linux server and quickly set up a file, print, or web server. The best possible way was to make them be able to use their existing familiar tools but instead point those tools at a Linux server.

However this was easier said than done. The smallest things can radically affect your value proposition. There was a social aspect we’d probably not considered. In the Linux world, systems administrators really get into the nitty-gritty of things. They’re different than Windows administrators. I’m not saying one is better than the other, but Linux administrators have always been very happy to roll up their sleeves and fire up a command line to get the job done, whatever it is. So building graphical things for UNIX and Linux didn’t really catch on in a big way. I think increasingly it’s going to happen, and in fact we’re starting to see some tools of that type, but at the time, there was really no interest.

However one thing did really catch on. People really liked the ability to join Linux server machines into Active Directory. Most of the Fortune 1000 and Global 2000 had large Active Directory installations and the Linux machines could not participate in the benefits that Active Directory was providing to the Windows clients and servers. In a sense, they were locked out of participating more fully within the corporate intranet, because of their inability to communicate with Active Directory.

Sean: What aspects of Active Directory seemed to be key to your user base?

Krishna: Three things immediately caught on. Interestingly, all of them were around securing Linux and UNIX servers

The first was single-user, single-password. Across corporate environments, Active Directory is the dominant NOS directory. If a customer wishes to deploy Linux servers in this environment, they could log in to the Linux server with their Active Directory credentials. This removed the need to have to deploy and manage a second directory.

Second was single sign-on. Once you entered your credentials once, you would not be prompted to enter your credentials again. Active directory group membership would be honored on Linux servers so access control to resources on Linux servers could be controlled through AD.

Third was centralized policy management. Because a machine was joined to Active Directory, we could push centrally configured policies out to groups of machines. Linux administrators were particularly interested in pushing out sudoers files . These are files that controlled what resources a user had access to on a Linux server.

All three of these feature requirements could be solved by integrating a Linux system into Active Directory. It appeared, from our first product line, that we had teased out a successful value proposition. Let me explain this a little further.

First, we were now providing a “must-have” product offering. By consolidating Linux, UNIX, and Mac systems into Active Directory, we were bringing in huge operational efficiencies. An existing, widely available NOS directory’s benefits were being extended to other operating systems. We were ensuring that Linux systems were now secure with single-user, single password, and single sign-on now available to all systems. Relative to our authentication offering, our management product suite was a “nice-to-have”. There are lots of examples of must-have versus nice-to-have offerings. For example, in the Windows ecosystem, one would consider Active Directory as a must-have relative to Windows Group Policy which is a nice-to-have.

Second, we were now solving an acute customer pain point, not something chronic. An acute pain offering is one where a customer is willing to pay you for alleviating his pain. On the other hand, customers have gotten used to their chronic pain, so they feel like it’s OK to live with it. For example, UNIX administrators might love to put together a bunch of convoluted Perl scripts or Python scripts or shell scripts to manage something. Giving them graphical tools, where they were comfortable with their scripts was us trying to solve chronic pain. There’s no appreciable benefit to them, offering them an alternative.

Third, you have to figure out a way to at least grow toward making the potential return on investment exponential as opposed to linear. If you’re writing MMC plug-ins, the way you add value is to write more and more plug-ins. Your return on investment is linear to the number of plugins that you build. Thus you want to be in the depth play business as opposed to the breadth play business. VMWare is an example of a great depth play. The core hypervisor technology is extraordinarily complex and hard to replicate, and so it is no wonder that they’ve captured the lion’s share of the market. Core authentication is a depth play – it is complex software; hard to replicate.

With the Likewise authentication product, we were providing a must-have, acute pain relieving, depth play product. We were securing Linux clients and servers in Windows dominated corporate intranets and reducing TCO. And the one thing that tied all these benefits was joining non-Windows systems into Active Directory.

Sean: So that brings us back to your relationship with Samba. Is that why joining Linux machines into Active Directory became so important to the overall Likewise value proposition?

Krishna: Samba consists of three key pieces of Windows interoperability technology: the file server, the print server, and the little known winbind authentication engine. The first two pieces–file and print–have always received a greater amount of attention than the authentication engine. In its first incarnation, Likewise would use winbind as the foundational technology to join Linux, UNIX, and Mac users to Active Directory. Once you joined machines to Active Directory, a Linux or Mac could benefit from all of the advantages that a Windows client would have in an Active Directory centric corporate intranet.

However, we did run into several challenges. For one, as you noted, Samba is not necessarily the easiest to use technology, and we made a significant number of changes to the code base. At one point, we were probably the largest contributor of software patches back into winbind. We also needed the authentication engine to be a programmable platform, so that we could create a larger developer ecosystem. Trying to push the kinds of changes we needed upstream could be quite time consuming and a challenge in itself.

We came to realize that most successful open source companies must be in a position where they control their own technology destiny. This allows them to be more nimble and adapt to their own market requirements in a timely manner. For example, Xen has XenSource (now Citrix) behind it and Mono is pretty much run by Novell. Other open source entities like Apache and Mozilla and Ubuntu are run almost like corporate businesses, the way they organize themselves.

So about two years ago, we decided we’d go ahead and we’d write our own authentication software system from the ground up. In January, 2008, a year later, we had written our own authentication software for Linux, which we called LWIS – the LikeWise Identity Service.

LWIS was written as an industrial strength, programmable, and diagnosable daemon. The goal was to ensure that customers could diagnose authentication failures rapidly. Suddenly, a lot of things started coming together for us. Now we had to make the decision: were we going to be an open source company or not?

Sean: Clearly, you decided to go the open source route. What fed into that decision?

Krishna: I am very proud of our company’s decision to go the open source route. Earlier with using Samba’s winbind technology, we were ethically and legally obliged to provide all of our source code changes upstream. With LWIS, we could have chosen differently. But we recognized that being an open source company made sound business sense.

First, it gave us the ability to reach a huge number of prospective customers almost instantaneously. Giving an open source dimension to your company is really valuable for purposes of seeding the market, so that’s what we did.

Another reason was our desire to create a platform developer ecosystem, in addition to an end user ecosystem. We wanted developers to program to our platform. We’d experimented with this before we released LWIS with our Likewise DCE/RPC system. A year before that, Novell had re-open sourced its version of the OSF DCE/RPC stack. DCE/RPC had pretty much languished over the past decade. This was probably one of the great ironies of platform interoperability. Microsoft’s MSRPC system is basically a fully compatible implementation of the OSF’s DCE/RPC stack. So when Novell open sourced its DCE/RPC stack, we thought “This is not exactly the best for us here, but let’s take it and clean it up, let’s rehabilitate the DCE/RPC.” So we fixed it and happily gave it back to the community. What this meant is that for the first time in something like 15 years, you could write Microsoft RPC-compatible DCE/RPC applications as first class entities on just about every UNIX platform.

We now owned our own intellectual property, we held the copyright to all our source code, and we found this amazing effect on all of the storage solution vendors and OEMs out there, who came to us and said, “Can we use this stuff? It looks pretty cool.” Our developer ecosystem had surely, but slowly begun to happen.

We’ve since continued down that path of building out this open source programmable Windows compatible distributed systems platform.

Scott Swigart: Just to back up a little bit, you said you had the choice of going with an open source approach or being a closed source proprietary company. You mentioned that one advantage of going that route was in getting exposure for your technology. Can you elaborate on that?

Krishna: One important advantage is the idea of seeding the market. There are a lot of wonderful things you can do with open source, and we all know that from a development perspective, it’s a great thing to do. But, here is why it makes sound business sense.

When you’re in the platform infrastructure business, your target markets are the IT administrators, system administrators, and application developers. At the end of the day, these individuals need to be comfortable with your product. With LWIS, we provided downloadable binaries for more than 130 platforms of Linux/UNIX and Mac operating systems. A sysadmin can now download the authentication agent for his platform install within minutes and have his Linux server authenticating to his corporate Active Directory installation. There is no pressure from us. He now gets to experiment, research, and ensure that our offering is appropriate and meets the requirements of his environment. This is really the power of open source. Customers are now in the driver’s seat, just the way it should be.

What happens next is that you actually find out that a lot of people have become champions for your product line in their organizations. If you do your job right, these people start adopting it, and the viral installation effect you can expect is rapid and enormous. So when we go into companies, we already have a little edge. I don’t have to go to a customer and talk about my value proposition. When I meet with a new customer, they often times say, “We’ve already downloaded and installed your system, and it’s working great.” An open source distribution model allows us to directly go into the demand fulfillment phase of a sale, rather than spend expensive cycles in demand creation and market education.

In addition, seeding the market with our authentication product puts us in a great upsell position with our customers. We’re in the business of building the best, the easiest to use, the easiest to deploy, and the best performing authentication engine to Active Directory you can find. What happens is that enterprise class customers need several additional features that help them roll out our authentication stack throughout their corporate networks. These include features like remote management, remote diagnosis, and centralized security policies. Indeed, there are a lot of upsell features we can actually sell to customers.

It’s really remarkable, because you quickly end up with contented and willing customers.

Scott: There is a balance to be maintained there, and I think that many aspects of it aren’t necessarily unique to open source. With a lot of software, there is a free version and a paid version.

For instance, consider what Microsoft does with Visual Studio. They have the free Express editions as well as the full paid versions. And just as Likewise does, there are a lot of companies that have a free community edition as well as enterprise upsells.

Obviously, you guys are very successful at it, but it seems that it must be very difficult to figure out where exactly to draw the line. Pragmatically, if you give too much away, people won’t take advantage of the upsell. They might pay for support or something like that, but they won’t buy the upsell version.

On the other hand, if you don’t give enough away, you don’t get that viral effect you mentioned, and you don’t end up with these really strong evangelists for your product inside of companies. What kind of insights can you share about how to decide what goes in the free, open version, as opposed to what gets reserved for the paid enterprise version?

Krishna: You described the problem really well, and it’s a hard one to gauge effectively. I don’t know if there is any one good answer. The key for us was building upsell features that enterprises need when they are rolling out the authentication software to several hundreds or thousands of machines. Things like user provisioning, reporting, auditing, and administrator troubleshooting become must-have features for the large enterprise.

We spend countless hours reviewing our feature set and most importantly focusing our attention on alleviating customer pain. Over the past few years, we’ve formalized our thinking around what our enterprise customers are willing to pay for.

User provisioning is a good example to describe our thinking around upsell features.

In the Windows world, every user has something called a security identifier that designates his identity in the corporate environment. A security identifier is an elaborately long string of bits that makes the user globally unique.

In the UNIX world, to this day, you are basically restricted to a 32 bit user ID. What ends up happening is that you actually need to have some mechanism that converts that security identifier to a UID every time a UNIX user logs in.

With the open source version of the product, typically we are dealing with departmental administrators who do not have access to modify their Active Directory information. You want to give them the easiest way to join Active Directory without having to make changes to adapt AD to Linux and UNIX. In this case, we give them the ability to synthesize a UID from the Window’s security identifier. However for the enterprise version, we provide an add-on pack that gives the customer control over what type of UID it’s going to be and store that UID in AD. They can make sure that a particular user gets a specific UID, and we charge customers for that capability.

One thing we definitely cannot do is roll back a feature that we made available in the open source version. People in the community don’t take very kindly to the fact that you’re giving them a less than full featured version of your product after promising a whole lot more.

Scott: It also seems pretty clear that you can’t take a feature back once you have offered it for free.

Krishna: Exactly, and it’s also true that once you decide to be an open source company, it’s really important that you be true to the principles of open source that you signed up to.

In the open source world, your audience and subsequent customer base is made up of people who believe passionately in the merits of an open source model. Your products appeal to this audience because of your open source stance. To no small degree, they choose to do business with you because your products are open source. Thus, you really want to make sure you come across as an authentic open source company.

It is important to note that open source doesn’t necessarily mean free. I’ve never once been in a situation where a serious enterprise customer wanted to deploy our solutions for free just because the source code was available. Remember we’re building authentication software for a corporate network. The open source binaries are the same binaries that ship with the enterprise product. The same single binary goes through our rigorous QA test matrices. Yet, we’re building something that is fundamentally mission-critical to an enterprise. Customers therefore want to know that there is somebody contractually obligated to support them if needed and therefore, there are always willing to purchase product licenses.

Scott: You mentioned that Group Policy is sort of a nice-to-have element. What have you done around that?

Krishna: I should have said relative to core authentication, group policy is a nice-to-have element. However within an enterprise, centralized deployment of policies to groups of machines is increasingly a must have feature. We support a full implementation of group policy natively on all of the 130+ platforms we support.

If you have 3000+ servers in your corporate extranet, then you want to make sure the systems are configured with a certain set of policies and that you can discover and verify that those particular policies have been applied. We support several hundreds of UNIX group policies. Here we decided that we would make UNIX/Linux artifacts as first class entities in their own right. So you can deploy cron jobs, sudoer files, and selinux policy settings. This appeals to Linux administrators because as experts on Linux operating systems, they are the best people to decide what the policies should look like, yet they get to leverage the Active Directory group policy delivery system.

Scott: I want to be sensitive to the time, so is there anything that we haven’t touched on that you wish we had? Also, is there anything you would like to add about future directions that Likewise is taking?

Krishna: I believe that both the open source community and Microsoft need to continue to make real strides towards true interoperability. I also believe that both Microsoft and the open source community have a great deal to learn from each other and a greater degree of product interoperability results in happier customers.

I spent 10 very intense years working on Windows at Microsoft, and I spent the last five on the other side of the fence working with open source. Some people ask me, “You worked on Windows a lot, how come you’re working on open source now?” The truth is that the answer has less to do with the open source versus proprietary offerings than it has to do with a fundamental focus on customers. Most customers are definitely less than thrilled with both camps, because of the challenges they face making these disparate systems work together in their networks.

For example, you brought up Samba, which is some really brilliant technology developed by some really gifted individuals. The Samba engineers are no different from the great Microsoft engineers I’ve worked with. However, Samba’s lack of usability and ease of configuration limits mainstream Samba options. And on the other side, it is no longer viable for Microsoft to think that they don’t live in a heterogeneous world. In their own self interest, the Microsoft “Better Together” campaign needs to extend beyond its own product offerings.

Imagine if you are a customer running several hundreds or thousands of Red Hat/JBoss application servers in your extranet. Imagine that you also deploy Active Directory as their primary corporate NOS domain controller. It is terribly frustrating for you if you have to deploy a second directory just to manage your Linux servers. At that point, I think you’d hope that both vendors could have your interests in mind rather than engage in a zero-sum game.

As far as Likewise’s future is concerned, we will continue to espouse the customers’ interest by building the very best of class Windows compatible distributed systems fabric for non-Windows operating systems.

Scott: That’s a great place to close. Thanks very much.

Krishna: Thank you.

Bookmark this:

Interviewee: Agostino

In this interview we talk with Agostino. In specific, we talk about:

• The history of Wubi
• Providing a bridge for Windows users to try Linux easily
• Integrating Wubi into the Linux community and specifically Ubuntu
• Making the OS install and later upgrade simple for users
• Feedback from users and other Linux distributions

Sean Campbell: Tell us a little bit about your background, how you got involved with Ubuntu, and your relationship to Wubi.

Agostino: I started using Ubuntu very early on, even before the first release. Around 2003 or 2004, I was a Debian user. I was helping people on forums, and a lot of friends were asking what distribution to use. I was very happy with Debian, but I never felt comfortable suggesting Debian to first timers. So I decided to test some Debian-based distribution that was supposedly “user friendly.”

At the time, the two most popular derivatives were Mepis and Ubuntu. Now it may seem strange, but Mepis was more popular than Ubuntu. Nevertheless, because I liked the Ubuntu philosophy better and in particular its less compromising attitude toward software freedoms, I decided to try Ubuntu. I ended up using it myself, and I have never looked back. Everything I wanted from Debian was there, but with half the hassle.

Gradually, I got more involved with the community, helping people out with issues they ran up against. A typical one was that people had difficulty creating the ISO. Many users didn’t even know what an ISO was, some would literally copy the ISO file on the CD, others would burn at too high speed, others still had to modify their BIOS setup to be able to boot from CD. As much as I liked the idea of a live CD, it didn’t seem to be the right approach for many end users.

Another group of people complained about the partitioning. I had two personal friends who never went to Linux simply because of partitioning concerns. A lot of computers come with a single partition, which is often fully dedicated to Windows, and some people really don’t want to take the risk of modifying that set-up.

So I started thinking of some ways around those issues. I did like the approach of topologilinux and of instlux, but I always felt that they were only addressing half of the problem. What I thought was needed was an installer in a format Windows users were familiar with (i.e., a fairly standard installation wizard executable, such as instlux) but one which would install in a non disruptive way (similar to topologilinux).

As it happens, the underlying technologies that were necessary to implement that were maturing around that time. On one side, Grub4Dos was becoming quite a viable boot loader, and on the other, NTFS-3G (the Linux NTFS driver with write support) was reaching the release-candidate stage. As for the loop module, it had been around for several years. So the building blocks were there and I decided to give it a go.

I wrote a blueprint to illustrate the idea, then Geza Kovacs developed the first prototype. It was an initrd modified to boot from a file containing a pre-made image. The feedback from the users was encouraging, and I joined a few weeks later to fully implement the blueprint. Instead of using a pre-made image, I modified the Debian installer, to make it capable of installing inside of a file and generalized the booting code. The process was very similar to a standard installation from CD.

But because we did not have the resources to provide a customized ISO for each Ubuntu flavor, I had to modify the booting code of the installer so that it would patch a standard ISO image dynamically. There were other changes involved on the Linux side, particularly in the boot loader and sysvinit code to make a standard distribution capable of booting and rebooting from a loop installation. The original project that covered all this backend work was called “lupin” (for “loop-installer”).

This was basically Wubi 7.04.

With 8.04, Wubi became part of Ubuntu and a lot of the changes were incorporated directly into Ubuntu without any need for extra customizations. Today, in terms of files, a Wubi installation is basically identical to an installation performed from a CD.

Sean: What do you think the impact has been of having something like Wubi available to Ubuntu? It’s somewhat unique, although there are obviously live CDs, and you can put a Linux distribution on a USB key. All of these things can accelerate early tryout and early adoption without requiring people to re-partition their drive.

On the other hand, particularly for users new to Linux, this might be a way for them to try it out for an extended period of time, whereas a live CD, as cool as that is, is inherently a temporary experience.

With Wubi, you could continue to use your files and access things in a Windows partition, in a way that’s analogous to Bootcamp on the Mac.

Agostino: Wubi actually wasn’t designed to do long-term installations. The main aim was really to let people try out Ubuntu with confidence. Normally, users that start with Wubi tend to upgrade to a full installation to a dedicated partition at the next release cycle.

I always thought that a live CD is a wonderful idea, but it is still too difficult for many users. As mentioned previously, people do have issues with burning CDs and booting from them and even when they manage to do it, they sometimes get an inaccurate impression of the user experience, because the live CD is far slower and has some limitations (read only access by default, and so on).

What people like to see in the Windows world is an EXE, not an ISO. They download a file, double-click on it, and something gets installed. If they decide they don’t want it anymore, they uninstall it. This is what they are familiar with, and there’s no technical reason why we shouldn’t be able to provide a Linux installation in that form.

Sean: You mentioned that Wubi gave Windows users an easier means to try Ubuntu, but it sounded like that created an interesting initial influx of bug reports and reactions to using it. How did you handle that?

Agostino: I’m quite pleased with the fact that a lot of our Wubi users are first-time Linux users. This was our goal, although of course the fact that very few of them are Linux experts can make it difficult to get good bug reports. Detailed and precise bug reports are always welcome, but much more so when you deal with an OS installer…

In Ubuntu, you have a program that automatically extracts useful information when an application crashes and sends a log report that a developer can look at and get a good understanding of the problem. With OS installers, you run into a lot of situations that are difficult to reproduce, because people have all sorts of hardware, and often the problem manifests itself during the boot stage, so you have no logs whatsoever and users end up in a console with very limited functionality.

When that happens, even experienced users would have trouble extracting useful information to help us debug the issue, let alone first-timers. That can be quite a frustrating experience both for the user and the developer. It was really tricky in the beginning, when we were in the testing stage, and things weren’t always that smooth.

On top of that, in our case, the Ubuntu file system is inside NTFS. One of the current issues is that in Linux, there is no file system check for NTFS. This means that we cannot recover from a corrupted NTFS partition from within Linux, and that implies that we cannot boot from NTFS if it is corrupted. The user has to boot back into Windows, clean up the issue, and then boot Ubuntu again. This is certainly the most annoying aspect at the moment, and it also generates a lot of false positives.

Sean: I assume that for the same reason, you don’t support hibernate and suspend.

Agostino: Suspend is actually supported, but we don’t support hibernation because the current hibernation mechanism used in Ubuntu doesn’t support swap files. And of course we need to use a swap file inside NTFS.

You would have the same issue in regular Ubuntu if you used a swap file as opposed to a swap partition. You can fix that by using a swap partition, but then you might just as well migrate to a full installation.

Scott Swigart: It sounds like your background was primarily Linux, but when you set off on this project, you mentioned that you had to build a Windows installer and do integration with Windows. What was it like to learn enough Windows to get something like Ubuntu to boot and run?

Agostino: I started as a Windows user, but I have been using Linux exclusively since 2000.

Within the Wubi project, I was the developer doing the Linux side of programming. The actual installation in fact happens in Linux, after people reboot their machine, and that required work on the installer and on some standard packages that had to be modified.

In the very beginning, I didn’t use Windows at all, because you can basically boot from a loop file inside of an ext3 file system inside of a VM. All my tests were done with Linux inside of Linux, so the very first Wubi version was, in fact, Lubi, the Linux to Ubuntu Installer. [laughter]

Then I had to test NTFS and for testing I simply used a plain NTFS file system with a boot loader inside of a VM. But eventually, I ended up buying a Windows license (actually a full laptop) to do the final testing. Well, there was some Windows development too, because at the time, the programmer that helped with the original Windows front-end, Oliver Mattos, left and I had to take care of the Windows wizard.

I took the chance to redesign the interface. I always wanted to have a one-page wizard. To be precise, I wanted to have a one-click installer (which of course implies a one page wizard). This is what people see today.

Scott: At what point did Canonical and the broader Ubuntu community take notice of what you were doing?

Agostino: We released 7.04 first, which was completely independent. After we came out with the 7.04 release, which was a beta, we were contacted by Henrik Omma from Canonical.

They invited me to the Ubuntu developer summit, which is an event held every six months, where they do the planning for the next Ubuntu release. In the months preceding and following the meeting I received a lot of help from Colin Watson, who rewrote a good chunk of the installation code and Evan Dandrea who helped with several patches that I had to merge in Ubuntu.

At the summit, I had a breakfast with Mark Shuttleworth and explained, in a few words, what Wubi was about. Mark was very keen, but in general, there wasn’t much interest from most of the other developers. Only two people turned up during the Wubi discussion. I can easily understand that. Linux developers don’t actually need Wubi. In a sense, Wubi is a peculiar open source project. Open source usually works best for products that developers actually use.

And that’s part of why I got involved: I was afraid it would have been difficult to find someone willing to invest time in a project like Wubi. When it finally came out as an official installer, it received a lot of attention.

Even then there were people from the community that didn’t feel comfortable with the new approach. They started to see user reports that where talking about “installation drives”, unfamiliar boot loader configurations, NTFS issues. More veteran users felt a bit lost.

In fact, although most of the files actually installed are identical to the ones you would get in a standard installation, the configuration is slightly different. The first boot loader is actually the Windows one, which in turns loads Grub4Dos. Then, of course, the file system is different. Instead of booting from a partition directly, you first mount the FAT/NTFS partition, then you have to loop mount a file inside that partition, then you boot off that mounted file.

The combination of the loop module, the NTFS driver, the FAT driver, and Grub4Dos allowed us to create Wubi, so we stand on the shoulders of giants. The work on Wubi is minor compared to the work on those other pieces of software.

Scott: Some development depends on very hard, very low level kinds of engineering, but I always like to see an interesting combination of things that already exist. Wubi’s a great example of that.

In hindsight, it seems like an obvious thing to do–get Linux out there and let Windows users try it out, in a really safe way that doesn’t ask them to do very much. But, like you said, it wasn’t really obvious inside of the Linux community, because the people who were already using Linux didn’t have any need for it.

Agostino: Making things easy to install is only half the work. It must also be as easy to uninstall and revert everything back the way it was, which is something people expect of an installer. This is not generally required if you are going to take over the machine, but in a dual boot setup, which is our goal, it is a highly desirable feature.

Moreover, leveraging an existing operating system gives us a lot of advantages. There are a lot of questions that you don’t need to ask, because you can observe what the user is doing in the host operating system. So you can simplify the installation to a great degree. For example, I don’t need to ask what language and keyboard setup the user wants, because there is already an operating system running where such choices have been made.

And because people can easily uninstall , I don’t need to ask whether they are really, really, really sure they want to proceed. We do not have a point of no-return. Users just click “Install,” and that’s it.

This combination of factors creates an extremely simplified user interface. That’s very important, because Linux was plagued for ages by the idea that it was a difficult operating system. I think with Wubi, we may have the easiest operating system installer ever. You have one click, two reboots, and you are up and running.

I hope this will help dismiss all the myths about Linux’s difficulty. That used to be true, but today, we are far ahead on that.

Scott: And as you suggested, a Windows user is comfortable running a traditional Windows installer. Some people are willing to go on their own and figure out how to make their system dual boot and those kinds of things, but it’s a lot easier if it installs and uninstalls like an application.

Agostino: We are working on the ability to start with a Wubi installation and later upgrade that to a full installation within a dedicated partition. We didn’t have the time to finish that work for the next Ubuntu release (9.04). it will be probably be in 9.10.

This way the full installation process becomes a smooth and progressive task. The user will not have to decide up front whether she wants to modify her set-up to install Ubuntu; she can do that one step at the time, and go back along the way if she does not feel comfortable. So the decision to change partitions and bootloader is postponed to when the user has accumulated enough interest and confidence in the new OS, and when they do that, everything is migrated to the new dedicated partition.

Scott: What feedback have you heard from non-technical Windows users about how this has been helpful or interesting to them?

Agostino: There are thousands and thousands of blogs and Wubi stories, most of which are extremely positive. I’m really flattered about it. But, the type of forums I tend to follow with greater attention are the support ones, where people have problems.

This is more interesting for me, because it’s where I can get bug reports. Our typical users are not really familiar with bug reporting tools, and in many cases, the only way to get bug reports in the early stages of the cycle is to go into the dedicated forum and look at the questions people ask.

Scott: What have you seen in terms of interest from other distributions? I realize that right now it’s kind of welded to Ubuntu, or at least that’s the perception. I have to imagine that there would be interest in some of the other distributions, like Open Zeus or Pandora.

Agostino: Yes, a few Ubuntu derivatives contacted me for information on Wubi. Also a Fedora developer showed interest in the approach. As mentioned, it is easier for Ubuntu derivatives. For other distributions, it will probably get easier once we move to Grub 2. That will require less modifications to the core distribution as the bootloader itself is more capable.

Scott: That makes sense. The first time I saw it, I remember thinking of it as being very much like when Apple found small bridges to make people feel comfortable moving over. Wubi gives someone a little more confidence that they can do it, as opposed to, as you said before, just tossing them an ISO.

Agostino: Wubi has its own niche and target: providing users that already have an OS installed with a fully featured dual boot setup so that they can experience Linux for real and decide whether it is something they want to use. And if it is not something they want to use, it allows them to uninstall it. It is similar to but different from the Live CD, and no, it will never replace a full installation.

Sean: We’re getting near the end of our time. Do you have any closing thoughts?

Agostino: I would like to thank all the people that made Wubi possible and whose hard work is not always recognized as it should: Colin Watson and Evan Dandrea from Canonical, Szabolcs Szakacsits for all the work and support with NTFS, and all the grub4dos and grub2 developers, all the testers and translators. I hope we managed to bring people a step closer to the Linux world.

Sean: Well, it’s a great way to work toward the goal. Thanks for talking to us today.

Agostino: Thanks a lot. Have a good day.

Bookmark this:

Interviewee: Liz Danzico

In this interview we talk with Liz. In specific, we talk about:

• User experience consulting for open source versus closed source customers
• Pros and cons of working with small groups versus open communities
• Strategies for harnessing the wisdom of crowds
• Design by omission versus inclusion of features and achieving coherence with a dispersed team
• Clustering meaning: roles for both humans and computers
• Web acilitation of conversation: emulating pre-literate culture

Scott Swigart: Can you start us off with a little bit of your background, including your relationship to open source and WordPress?

Liz Danzico: All right. I am a user experience consultant, by and large, here in New York City. I do a significant amount of work with Happy Cog Studios, which is how I got involved with WordPress in the first place.

Happy Cog Studios did their redesign of WordPress 2.5 starting in 2007, which launched for that release in 2008. Then I was an independent consultant and continued to work with WordPress in a user experience capacity, doing user research and prototyping.

I have a number of other clients who I do work with in user research, information architecture, and a general user experience capacity. That’s one third of my job as a user experience consultant.

The second third of my job is that I’m chair of an MFA program in Interaction Design at the School of Visual Arts, which starts in the fall of 2009.

The final third of my job is that I do a lot of editing and writing in that general space.

Scott: What do you mean when you say “user experience?” Unpack that for us a little bit.

Liz: User experience includes every touch point that you have with a brand, whether it’s your experience with a website, a product, or a service. Your experience with every aspect of a brand is what we consider to be a user experience.

When we first started using the term “user experience,” when it was introduced in the work of people like Clement Mok in the late ’90s, we thought of it just in terms of a web experience. I think that now, we are starting to realize that the lines between products and services are kind of blurry.

That is, your experiences with digital things and physical things are blurring, and as a result, user experience is spilling outside of the digital experience into the physical experience.

Scott: When you work with a group like WordPress or others you consult for, what kinds of things are you brought in to do?

Liz: My work with WordPress is actually quite a bit different from the work I do with other people, for a number of reasons. In general, when I work with a client, there is a specific point person with whom I set up goals for an individual product or service and craft a plan that is very clear and has directives. I’m thinking about expectations, deadlines, and where and how the work is going to happen.

The same thing can happen for a project like WordPress–and it should–but one of the great problems of open source design is that the clients are as big as the products. As many users as WordPress has, that’s how many clients you have.

Of course, that doesn’t mean that the design is opened up to the millions and millions of WordPress users. At the same time, though, it sets up a different model, where ideas and concepts and iterations are opened up to the community, and feedback is broader and shallower than it would be with a typical client.

The relationship still has the same critical components, such as a leader, a problem definition, expectations, deadlines, and guidelines about where and how the work is taking place. But all of that is happening in the public sphere, like at WordCamps (which are locally organized WordPress conferences) or on the WordPress blog. That kind of feedback happens in a different place and in a different sort of timeframe than it would with a typical client.

So the tools you would use for having those conversations look a lot different than they would if I were working with The New York Times where things are happening on a much smaller scale but for an audience no less broad.

Scott: It sounds as if, when you work with a client that’s not an open source project, you interact with a much smaller group, and they make a lot more executive decisions about the direction the work takes.

On the other hand, with an open source project such as WordPress, there’s a lot more vetting the ideas and taking input from the community as the design progresses.

Talk a little bit about the pluses and minuses of each approach–working with a small team versus working with more of a community based effort.

Liz: Of course, this requires using stereotypes on both sides, since they both depend on hitting the right stride with the right people. But for the sake of discussion, let’s assume a best-case scenario on both sides:

Working with a small team in a closed scenario, assuming that you have the involvement of someone who has decision-making authority (or at least good access to the person with that authority), you can move very quickly with brainstorming and meeting the goals of the project. You also have a great sense of always knowing where you are at any given time.

If you have a strong project manager on your side, you can work against the deadlines set by that decision maker. That means you don’t need complicated tools to manage the project that can bog down the process.

Because there are very few people involved, everyone can meet together–virtually or in person–and everyone is clear about the directives that you’re trying to solve. Therefore, the product itself–which could be a physical product, a service, a combination of those, or even a brand or logo–can have a clarity of focus that is really hard to get when there are larger numbers of people involved.

That’s the key benefit of working on small teams in closed areas where there’s not a lot of sign-off that needs to happen. The drawback is that this group can be insular and isolated. If they’re truly visionary, you get the iPhone scenario, and you can get a really great result.

On the other hand, if this small group doesn’t have the right set of data or the right information at their disposal, then the drawback of is that they are misinformed. Even though they have a great lightweight process and can move very fast, if their work is predicated on the wrong information, the results can completely miss the mark.

In contrast, the open source model can have the inverse kind of problems. One of the signature problems of designing in the open is that there are often no clear leadership or problem statements. Typically, there are a lot of people with good ideas designing a lot of things, and the loudest, most visible person tends to get a lot of attention.

Understand that, by leader, I don’t mean an open source design evangelist. I am referring to someone who can actually take control over the project–or at least one idea–and move it forward. Without someone to capture those results, there is no momentum and there is no coherent trajectory.

That creates a domino effect of missed expectations, because everyone is working toward their own conception of the goal. There are no deadlines set, so there can be no momentum for all of this great work that is being done.

If deadlines aren’t set, a project misses out on one of the benefits of open source or working across EPs, which is that people can work staggered hours and in different ways, when it is suitable for them. Unless deadlines are set, no one really knows what they are working toward, which can tend to squander their effort and frustrate the contributors.

Open source projects tend to attract fantastic people, and in order to take advantage of their capabilities, it’s important that, at the beginning of the project, there needs to be a little bit of structure setting up clear parameters and a couple of leaders.

Some tools need to be set up, as well as things like a wiki, bug tracking, and processes to check code in and out. You don’t necessarily need complicated project management tools with this kind of work; lightweight ones can actually be far better.

The benefits of open source are clearly that you get a lot of feedback from people in real time, as well as the wisdom of crowds. But you need a leader to cull and edit the wisdom of these crowds, or else you’re just going to end up with a glut of information and no real sense of what to do with it.

Scott: “The wisdom of crowds” is a nice turn of phrase. In some ways, I think Shuttleworth tries to harness that with Ubuntu. Everyone once in awhile, the wisdom of the crowd washes over him, and he’s left holding his surfboard broken in half.

Still, I’ve always really admired what they’ve done with the Brainstorm site, and particularly with their latest refresh, they’ve made it even easier for users to participate.

I think user feedback is a weakness with a lot of open source projects. I’ll use OpenOffice as an example, even though there are any number of others that I could name. When we talked to them a couple of years ago, they talked about how they were very open to user requests, but I couldn’t help wondering where a mainstream user would make such a request, or even how they would know it was an option.

On the other hand, Ubuntu has built these external portals like Brainstorm that users can interact with and understand, without a lot of real work. They don’t have to think through mailing lists and IRC chats, and everything else.

How do you think those kinds of efforts feed into the types of concerns you are discussing?

Liz: I think that borrows from Digg and other models that people are familiar with, which aids in participation. It’s sort of a group editorial function, but honestly, I think there’s more needed than that. In terms of editing the wisdom of crowds, I’m really talking about a more centralized editing function with a bigger red pen.

Scott: There’s an interesting sense that some people see Shuttleworth in–sort of the keeper of the hip version of Linux that thinks about UI. He doesn’t always think about the Kernel, but yet I haven’t seen too many people ask themselves, “What more could he do?”

Liz: Remember the book “Paradox of Choice?” I interviewed Barry Schwartz once, and I recall asking him about what was needed in editorials. His perspective, at least when I interviewed him about a year ago, was that “We’re not coming up with better tools, or better search engines, or better filters for our data. We’re actually going to human editors for the work that we’re doing.”

We’re moving away from data-driven solutions and more toward human editors. I think that there’s only so much work that can be done before you have to kind of turn it over to people to make that data useful.

An actual person has to sort through and make sense of it, with an editorial perspective for people, instead of just having data just kind of represent itself over and over.

Sean Campbell: We went to Gartner in the fall, and there was a presentation on the “Evolution of Search” that included a hilarious little skit. He said, “Search is essentially a list, as if you walked up to a hole in a wall and yelled “penguins!” and a librarian on the other side held up a series of books for you to look at.

That’s the ultimate limitation of search as we know it. Google gives you pages and pages of results, but ultimately, you still need to sift through more information than you’re really equipped to handle. People are drowning in it, and for the most part, they don’t really know how to navigate through it.

Scott: I recall an interview we did with someone whose company developed a product with in-house developers and released it under open-source license. We asked him whether he could build that product without having the traditional ability to bring his teams together in meeting rooms with white boards and so forth. His answer was that he felt the product was too complex to be built by a dispersed team.

I mentioned to him that, at OSCON, in a discussion about open source usability, someone asked, “When is there going to be an open source iPhone?”

The perspective that he gave was that there’s a trade-off inherent to open source, because development tends to be very incremental, and features accrete over time. That’s great for the Linux kernel, because it adds more and more device drivers, while not really getting rid of any of the old ones, so it’ll run on very old hardware and very new hardware.

Likewise, it’s great for the Apache web server, because configuration options become available for more and more scenarios.

On the other hand, the brilliance of the iPhone doesn’t come about because of more and more added features–it’s the result of ruthless cutting. From his perspective, that’s very hard to do with open source, because the process wants to have that one more feature that’s going to be useful to two percent of the user base.

Sean: I don’t remember the person’s name, but there was a famous designer who said (to paraphrase), “I know I’m finished when I’ve removed the last thing I can remove.” That’s the opposite approach from open source, where it often seems like we’re done when we’ve added the latest 10 things.

Scott: Sean just recently got a new MacBook, and instead of having one button, it has no buttons. They decided that they would simplify the thing beyond having just one button, and make the whole thing a button.

[laughter]

What are your observations around that design notion of removing versus adding things?

Liz: I am actually a writer by background, and my first job was writing user manuals—the first for a washing machine. I have spent a lot of time and energy in both professional and academic contexts studying how to communicate ideas to an audience in the fewest possible words.

As technical writers, we approach the problem by looking at all kinds of documents–bank statements, rental agreements, and washing machine instructions. By understanding the audience’s prior knowledge and what situation they were in when they were trying to understand the information or ideas, we were able to economize on words.

That training and background predisposes me very much to the mindset that one should build a product and then think about how to take away features, rather than adding them.

My favorite example is not one of building products, but of managing customers and thinking about how to listen to them. Until around the beginning of 2007, Jason Fried of 37signals never had any customer service representatives; he answered the 125-150 customer service emails himself.

He would get questions like, “How do you manage all of those customer service emails, and how do you listen to customers? How do you know who to listen to and who not to when it comes to feature requests?” Because if you’ve ever used a product that you love or one that you hate, you’re probably at some point going to say, “You know what I wish I had is this button or this feature, and I’m going to tell them about it.”

Jason has openly admitted that he responds to emails with customer service issues, but he just deletes emails with feature requests. The philosophy is that if a feature request is popular enough, it will come up again, and if it keeps on coming up, then it’s time to start paying attention to it.

If something is popular or critical enough that it keeps coming up, that is the reason to consider not trimming it away, even if you have to regard it in kind of a stricken way.

I always thought that was pretty alert, because it’s fairly standard, especially at large companies, for there to be very elaborate customer service measures to categorize and tag and file information like that, at substantial cost.

Sean: That reminds me of someone we knew who worked at Microsoft and was gone for four weeks. When he got back, he deleted his whole inbox, under the premise that anyone who really needed to get in touch with him would send another message.

I don’t have the guts to do that, but I’m glad somebody does.

In your example, he’s trusting himself to kind of aggregate things into meaningful buckets. He’s relying on that human neural network to do stuff that computers aren’t really up for. We have interviews transcribed through a service called CastingWords, which uses Amazon Mechanical Turk.

Again, the idea is basically that they’ll chop the interview up, and people will sign on, grab a piece of work, transcribe a bit of this conversation, and post the results back. It works extremely well, compared to other transcription services we’ve used, and it doesn’t try to use software to do something that it’s not well suited to yet.

Liz: Mental clustering.

Scott: I think that to exist, we have to be able to look for patterns and aggregate information from a lot of sources and boil it down to ascertain meaning, since we can’t really process all of it. From a design standpoint, a lot of bad software tries to force you to process way too much.

For example, it may present huge lists or grids of information, trying to make the human do what the software ought to be better at, rather than leaving the higher functioning stuff for the user.

Liz: If that is the case, it comes back to needing better editors, because in general, there’s going to be a bunch of information, and clearly, we don’t yet have the tools–whether it’s Brainstorm or whatever–to provide the patience or trust to allow someone else to be the editor.

Maybe what we’re going to need is to train each person to be a better editor. We’re all going to be in charge of our own data, so we need to develop these skills to have better mental clustering and pattern recognition and so forth.

Maybe that’s one of the new skills, and we should start training people to do that kind of thing instead of relying on external tools to do it for us.

Sean: I think there’s a lot of truth to that. As language evolved, it became a vital skill to be able to communicate effectively. Looking back to the 18th and 19th centuries, that was a skill everyone aspired to.

The situation now is that creation is just as much about aggregating what’s out there and deciphering what it really means as it is actually writing up the output.

We’ve all seen the situation where some people don’t have the tools or the framework they need. If you’re web savvy, you can recognize what a screen worth of data means in a blink of an eye, and you will immediately ascertain what the next logical step is. Someone else might not find the relevant information on that screen for five minutes.

Scott: Our model for handling that information keeps changing, too. Early on, Yahoo! had the notion that humans and not machines were the key to categorizing the material on the web. They undertook the task of manually categorizing all these web pages, and then Google came along with a lot of data and a killer algorithm and just trounced them and everyone else.

Now, in order to use Google effectively, you have to know what to put in to get out what you want, and it isn’t always necessarily the most obvious keyword.

Technology leapfrogs human effort, and then people leapfrog the technology, because now they’ve got a foundation to go farther than they could go before. That takes us back to the notion of people driving things.

Liz: Alex Wright wrote a fantastic book called “Glut: Mastering Information Through the Ages.” More than one of the chapters in the book talk about oral traditions among pre-literate societies and the idea that we are now seeing a sort of return to pre-literacy and oral traditions.

The idea is that we’re writing online content as a means of having conversations, as opposed to the one-way nature of previous forms of written literature. That’s very much how things worked in the oral traditions, and Wright makes a great argument for why that is in the last chapter of that book.

Sean: That’s a great example of what we’re talking about.

We’re running short of time, so are there any closing thoughts you’d like to add?

Liz: I feel like we’re at a point of comfort with these tools, including things that have been popular for now a while like Flickr, as well as sharing tools like Basecamp, or even tracking and version-control tools.

People have become comfortable with the idea of putting data out there to be shared. That suggests that open source design, which has been controversial for a long time, might have a new chance for success.

I think the next year or two is going to bring some interesting new changes in the ways that we present design work to one another.

Sean: That seems like a good place to wrap up. Thank you for taking the time to talk today.

Liz: Thank you.

Bookmark this:

Interviewee: Alexey Rusakov

In this interview we talk with Alexey. In specific, we talk about:

• Tracing a history of involvement as a Linux contributor
• The rise of Free Software in Russia
• The role of ALT Linux in advancing Free Software
• The expansion of Free Software in schools and other Russian institutions
• The Russian government’s role in technological change

Sean Campbell: Alexey, please tell us a bit about your background.

Alexey: Around 1999, when I was a student at Moscow State University and working at a company where they used Microsoft Windows, I started to learn that there were other operating systems. At that time, almost every end user in our country believed that there were no alternatives to Windows for the desktop. System administrators (often called sysadmins), however, were aware of UNIX and OS/2, and some of them knew of a comparatively young system called Linux.

Some of my friends were working as sysadmins, and they introduced me to OS/2, UNIX, and Linux, which grabbed my interest immediately. My first Linux system was Red Hat Linux 5.2, if my memory serves me correctly. I learned of the UNIX way of doing things, and it was a new approach to working with the computer for me.

The idea of joining commands into pipelines that convert, process, and format data appeared to be very powerful, and it impressed me very much. I was very interested with this direction of computer technology, and I started digging deeper.

My serious experience with Linux started a year later, in 2000, when I signed a contract with a company that used SUSE Linux in their everyday work. They did work for some German institutions, developing a search and retrieval system that worked with biological data. This was my real start in Linux and my first exposure to the Linux kernel, Vim editor, GNOME, KDE, and other applications.

It was a very interesting experience. I have worked in that company for four years and learned Linux very deeply. In general, I liked it very much, but there were some very problematic areas in SUSE Linux, particularly in their packaging system and out-of-the-box configuration.

Sean: What was wrong with their packaging system?

Alexey: The packaging system was based on RPM and, concerning the command line, there was almost nothing to facilitate working with it. There were no applications or scripts that could help resolve dependencies, and I had to go through the entire process of installing RPMs almost on my own. There was, however, a graphical tool called YaST that somewhat eased the work of installing RPMs, but there was yet another problem–the lack of a consistent and fresh package repository.

Actually, it was not only a problem of SUSE, the problem of GNU/Linux package repositories was very common in those days. Even if a repository for some distribution existed, it was more or less a pile of packages, nothing more.

Sean: How did you become involved in ALT Linux?

Alexey: Well, I started searching for other distributions, and I learned more about Mandrake, Red Hat, and Debian, as well as our local Russian distributions–namely ASP Linux and ALT Linux. I tried ASP Linux first, but it didn’t impress me, and then I tried ALT Linux.

ALT Linux was quite peculiar in some ways. It started as just a Linux localization of Mandrake Linux, but the differences increased, and in 2001, ALT Linux became a separate distribution: a fork of Mandrake.

I installed that Linux on my home machine, started using it, and was very much delighted. The packaging system in ALT Linux was based on RPM, too, but the distribution authors added a framework over RPM, called APT. They borrowed it from Conectiva, the Brazilian Linux company, and Conectiva, in their turn, had taken APT from Debian, and adapted it to work with RPM instead of Debian’s dpkg. Conectiva was bought by Mandrake shortly afterwards; the result of this merger is now known as Mandriva.

I got very interested in the approach of ALT Linux to building distributions. They paid a lot of attention to what is actually delivered to the user. Besides that, they had a very strong, although not large, community, which used mailing lists for communication. By and by, I became more involved in talks about ALT Linux, its bugs and features, and, well, I think I fell in love with it. After two years of working with ALT Linux, I decided to become one of the package maintainers, which is our equivalent of a “Debian Developer”.

A maintainer is actually one of the people that builds packages within the community. He is responsible for building new versions, fixing critical bugs, and plugging security holes in it, although he is not paid by the ALT Linux company for this. I started maintaining one or two packages, and then I got more deeply involved. Eventually, one of the very prominent maintainers who was building GNOME packages for ALT Linux resigned from the project and asked if someone could build GNOME instead of him.

The project needed someone to build GNOME, and I volunteered, because GNOME was my main working environment (I’m still using it now). That was a very challenging experience, because there were 100 or more packages, and the project pays a lot of attention to the consistency of packages–they are almost hand crafted, I’d say. Besides, there were no tools that could really help work with a bundle of packages at once; they appeared later.

I managed several stable builds of GNOME, and one day I was contacted by directors at ALT Linux and started working there as an employee.

Now I’m not building packages much; I mainly work as a project manager. I’m also dealing with most of our international relations. GNOME is maintained by a team of volunteers now. I still build some packages once every week or two, just for fun.

Sean: You are the first person we have talked to in Russia who is involved in Open Source. Could you tell us more about Open Source’s general position in Russia? How prevalent is it, and how does ALT Linux fit into that?

Alexey: I’d like to use the term “Free Software” instead of “Open Source.” Open Source typically means that the source can be viewed, whereas Free Software gives anyone the ability to see the source code of the program, to alter it, and to distribute the altered form, if they wish. That freedom, of course, also carries the ability to run the software and to copy it without limitation. I’m a little bit picky about these terms, because it was quite common to misuse and even abuse them not so long ago.

Until 2004 or even 2005, Free Software was a fairly marginal concept in Russia, although there were Linux user groups (LUGs) spread out around the country, and of course, one could learn about various distributions over the Internet and get involved.

The Free Software community in Russia formed around a number of people, mainly system administrators and higher school students, who got together in LUGs, exchanged their experience, and got more proficient in creating better configurations. Ultimately, some of them took those configurations further, to the point where they started creating Linux distributions.

There were maybe a dozen notable projects that had created Russian Linux distributions by the end of the 1990s, although most of them are gone now. Most were discontinued within two or three years, but a couple of distributions outlived the others.

ALT Linux was originally both scientific and commercial. The Institute of Logic, Cognitive Science and Development of Personality, a nonprofit organization founded by the Russian philosopher Vladimir Smirnov, was a central institution associated with the scientific part. His son, Alexey Smirnov, is now the CEO of ALT Linux.

The commercial part involved IPLabs company, a computer hardware retailer. In 1998, they cooperated and created a Free Software project that was named IPLabs Linux Team. They started localizing Mandrake and other distributions, and publishing these distributions in Russia.

That team did great work improving core parts of Linux, including the kernel, RPM, X Window System, KDE, GNOME, and others. The project greatly influenced the future of Free Software and GNU/Linux in Russia, and its development model still works–and evolves–10 years later.

In 2001, IPLabs Linux team and another team of volunteers called Linux RuNet decided to join forces. The resulting company was named ALT Linux.

At the same time, a decision was made to fork from Mandrake altogether. The last distribution of IPLabs Linux team was Linux Mandrake 2001 Spring Edition. It was very different from the original Mandrake, actually. The hardware business of IPLabs was discontinued in 2001, and the main work of the new company was creating software. One of the first things to be done in ALT Linux was the creation of a consistent repository of RPM packages that still exists and evolves today.

That repository is called Sisyphus. The name is quite fitting, since Sisyphus is a figure in Greek mythology who endlessly pushes a boulder up a hill, only to see it roll down again. That unceasing work is something like what happens to the packages in this repository.

Our maintainers constantly build new versions of software and push them to the repository. The work never ends, because new versions of software are constantly released.

Of course, the main part of the work is done when a maintainer initially packages some software for the repository, and after that, the software is actually just rebuilt, with slight modifications of the building process from version to version. ALT Linux have made great strides in automating this process, and our infrastructure lets us tolerate even very serious changes in upstream projects.

The repository is forever “unstable”, meaning that it contains the newest versions of packages, which are not necessarily stable ones. This may cause problems if you use Sisyphus carelessly in everyday operations. Testing and bugfixing is actually performed by those who use Sisyphus as the source of packages for their systems (including maintainers).

Sean: When ALT Linux started to gain momentum as a distribution in its own right, I imagine that it must have had to create an identity for itself. How did that evolve?

Alexey: Since 2001, ALT Linux have been constantly promoting Free Software. We believe that Free Software is the only possible choice for government and other public formal structures. The reasoning for this position is that the government needs to have non exclusive rights to avoid being locked in to a single provider, and they need to have access to the source code as a matter of national security.

ALT Linux made great progress in promoting this point of view within Russia. In 2007, the Russian government started reviewing a document called “The Concept of Free Software Development in the Russian Federation”, and the accompanying Glossary.

In 2008, the Russian government approved this document. This approval enforced the term “Free Software” and some other terms legally, although the main license of Free Software, the GNU General Public License, still cannot be referred to by lawyers in Russia since there is no adequate and approved translation of this license to the Russian language.

Before that time, the terms concerned with Open Source and Free Software were often misunderstood, misused and even abused by software giants–they obviously did not like the Free Software movement. Particularly, Microsoft’s SharedSource initiative comes to my mind, as well as Sun’s license for Java 1.5.0 and former versions. The glossary has set things straight.

In 2007, the Sisyphus repository gained the very important ability of branching. Before 2007, we largely had to stop receiving new versions of packages, at least for major versions, before preparing a release. That problem was solved in 2007, so that we could speed up the release cycle and release distributions more often without stopping development on the main repository.

It was also very good for our community, because the community actually loved new versions. People who use Sisyphus really like to have the latest and greatest versions, and they get nervous about big freezes.

On the other hand, using stable branches for releasing distributions allowed other companies to release their distributions on these branches that have more or less predictable functionality. Actually, we’ve created a truly open model of Linux distributions development.

Sean: Was the government activity in the field of Free Software limited to approving strategic documents?

Alexey: Of course, no. Also in 2007, the government invited bids for a contract with companies who are willing to develop and deploy a Free Software suite in the schools of three Russian regions: Tatarstan, the Perm Territory, and the Tomsk Region.

Together with some other Free Software companies, under the auspices of Armada group, we won the competition. The Free Software Suite for Educational Institutions has been deployed to 1000 schools in the three regions. This suite actually consists of several distributions for different hardware and use cases, all created on the same stable repository branch.

The problem of outdated hardware is a very important one in our country.

Outside Moscow and St. Petersburg, the two greatest cities in Russia, there is a lot of outdated hardware, especially in schools and other institutions that live on the state budget.

So, the distribution for the weakest configuration manages to work even on systems based on Pentium processors and 96 megabytes of memory (and it’s still a modern distribution, just a lightweight one). The next distribution is targeted at more powerful configurations, such as a Pentium II or Pentium III processor with 256 megabytes of RAM. The top distribution is intended for more modern systems. Yet another distribution was crafted to be used in terminal classes, and two Live-discs were supplied to help teachers and students try the Suite without installing anything.

This suite was distributed in these three regions. We got a lot of feedback that has allowed us to improve our distributions. The popularity of this project was so large that schools from other regions began to ask to participate in the project. So the Federal Agency of Education issued a letter that allowed schools from other regions, in addition to these three regions, to join the project.

In addition to the original 1000 schools, another 1000 schools from all over Russia got a box with the suite and deployed it. There are some plans to continue development and support of software for educational purposes in 2009 and in 2010.

Sean: There’s an interesting undercurrent in your responses about the interplay between change and stasis, in terms of the adoption of Free Software. How do socio-political forces in Russia play a role in governing technological change?

Alexey: Many things in Russia are started and led by the government, which I believe provides quite a conservative structure concerning public institutions. Local people aren’t apt to change something for themselves, but when the government suggests to institutions that something should be done, they tend to simply go ahead and make those changes, without necessarily thinking about it too closely. There are both good and bad consequences of that.

Anyway, since 2005 or so, the situation in Russia with regard to Free Software has changed quite a bit. It has changed toward awareness, first of all. The people have become aware, much more than before, about the existence of alternatives to the Windows platform, and about copyright issues too.

One story, which began in 2006 and ended at the very end of 2008, has added to this awareness greatly. Alexander Ponosov, a school headmaster, was accused of using pirated versions of Microsoft Windows, Office and some other programs installed on computers of his school. He got involved in a lawsuit, and he was even about to be prosecuted, but after several complaints the accuse has been dismissed completely.

This story became well-known to the public. Many people suddenly learned about alternatives to Microsoft Windows, because Ponosov started searching for something else, and found Linux. On the other hand, people also learned that failure to follow licensing requirements (which are sometimes rather intricate) can lead to legal issues.

Sean: What were the long-term effects of that series of events, beyond the initial increased realization by people of the potential consequences of software piracy?

Alexey: We believe that competition started to be fairer as a result. Before that, most people didn’t have an idea that copying software could be illegal. Hence, the scale of software piracy in Russia was enormous. But since 2005, various government structures started pressing the public very seriously (sometimes, as in case of Ponosov, too seriously) about using only licensed software, and people became more careful to do so.

As a result, people started thinking more carefully about exactly what software and functionality they need. They found that high-end software like Photoshop or Microsoft Office was overkill for most people. They could buy a lot less functional programs which are not so expensive and be fine with them.

Another way out appeared to be using free alternatives that, however, might be not fully compatible with their closed-source rivals. So people began to discover Mozilla, OpenOffice.org, GNU/Linux, and other programs that were not only free of charge (as in “free beer”) but also free to modify (as in “free speech”).

They started using this software first of all because they didn’t have to pay for it–not because it was Open Source and could be changed and distributed. But then many of them discovered added value in the form of helpful community and ability to fix simple bugs in software or at least ask someone to fix them without long unfruitful waiting for attention from a large software vendor.

The success of GNU/Linux in Russia was very much concerned with these ideas that software should be obtained legally and that some software can be distributed and modified legally. Copying or fixing something for each other is a usual thing in Russia, and Free Software fits better to our mentality.

At the present time, there is already a serious movement toward Free Software in areas where there is not much money. These areas are very significant, including medical, educational, and other government-controlled institutions.

Business structures also started adopting GNU/Linux, mainly because it gives more freedom in copying and installing as well as in changing and customizing it as needed.

Sean: We’re running short of time, but thank you for talking to us today.

Alexey: It was my pleasure.

Bookmark this:

Interviewee: Gwyn Fisher

In this interview we talk with Gwyn. In specific, we talk about:

• The value of automated code review
• The advantage of code review at the developer level
• Judging the security of open source codebases
• Coordinating bug resolution between distros and upstream projects
• Educating users about vulnerability in the face of complacency
• The changing face of computer attacks
• Why isn’t everyone using code-validation tools?

Sean Campbell: Gwyn, why don’t you start with your background and tell us a bit about Klocwork?

Gwyn Fisher: Sure. I’m the Chief Technology Officer at Klocwork. I’ve been with Klocwork for a couple of years now, and as a consultant, actually, working with them for several years before that.

Klocwork was founded in 2001 as a spinoff from Nortel Networks. The Nortel research group that created our basic technology was centered on the head of the CASE department at ISPRAS, part of the Russian Academy of Sciences and affiliated with the Moscow State University, who was very interested in the notion of very large scale software architecture, model discovery, efficiencies and inefficiencies in incredibly large source code bases.

As it happened, Nortel had a specific business problem that they were trying to deal with that required massive re-engineering on one of their switch operating systems, and so as part of the existing partnership between Nortel and ISPRAS, he was funded to help solve this problem. Following the spin-off and creation of our company, he and most of the team relocated to North America, where many of them continue today to be the kernel of the ever-expanding domestic product development group within Klocwork.

We stayed in that space for several years, helping customers in very similar situations, who had requirements for re-engineering very large, very old, very complicated code bases, in order to simplify, componentize, and improve their software for newer devices, reuse, maintainability, and all that good stuff.

And then some of those same customers–and these are all very recognizable brand names–came back to us and asked us to do defect analysis for them. So, in addition to finding things that are wrong with their architecture and the fundamental way they’re building their software, we also started locating things like where they were leaking memory, doing bad things with pointers, generating buffer overflows, and other basic software problems.

After a couple years of research, we brought the first iteration of a product suite to market for defect and security vulnerability location and analysis.

So, since we were founded in 2001, we have sort of morphed over the years, away from a pure architecture and model discovery kind of technology, into a much fuller footprint in terms of what we do around defect analysis. That includes operational defects and security vulnerabilities, as well as continuing to emphasize our strength in architectural and maintainability issues.

There’s also that whole backbone of model discovery, visualization, and re-engineering–all the great stuff our customers need to be able to do in these very large code bases to manage them over time. We help them keep building out across more languages, more platforms, more tool chains, and all the good stuff that goes with working in a production software development environment.

Scott Swigart: What are the things that are really hard to find with standard code reviews and QA that tools like yours can quickly make apparent?

Gwyn: The quick, sort of palm-to-forehead, kind of “Why wouldn’t we have found that? Why didn’t we think of that?” sort of issues tend to be in the area of what is called inter-procedural analysis.

If you have a very large system, it’s made up of a whole bunch of functions, methods, and modules all linked together into large system images. And the problem, whether you’re using code review, runtime testing, or whatever, is finding enough combinations so you can trace through, from source to sink point, exactly what’s going to go on with a pointer, buffer, array, or whatever it happens to be.

We can point out across multiple different function boundaries, that this pointer you’re grabbing from over here, you’re using inappropriately over here, or this memory you’ve allocated here, you’re never releasing on this particular thread. Those sorts of instances would require your average high level architect to really bend their mind around them for days or weeks at a time to try and find manually. They’re the sorts of things that we can point out to someone and they can understand it very easily within 30 seconds or a minute.

Obviously, we all bring our own biases to bear, in terms of how we think about our products and how we think about our tools. And it’s perhaps obvious to think about a tool like this as a QA tool, and certainly the heritage of tools like this has been very much driven by a downstream audit or an adjunct to code review. But, really, that’s not where this technology comes from–this is a developer’s tool. The first one of these was Lint, back in the ’70s.

And as a developer’s tool, well, Lint’s got a pretty bad name. Let’s face it.

[laughter]

As sort of a history, we’ve all sat there and watched the CRTs desperately trying to refresh as acres upon acres of warning messages come out of these things. But when tools like Klocwork bring modern analysis technology and accuracy and complexity into that equation and deliver it back to the developer, all of a sudden you’re not just inundating them with all kinds of stuff that they can’t deal with. You’re actually giving them a few very useful bits of information before they check code in.

That’s our approach, instead of trying to kick in downstream and then figure out who the right developer is to blame, instead our goal is to build analysis much more organically into the development process so it becomes an enabling environment rather than a blaming environment.

The uptake within an organization changes overnight, as soon as you can get through that cultural issue. Rather than giving QA a new tool for telling the developer that they’re an idiot, we are giving the developer a tool that will help them prevent being called an idiot. [laughs]

Scott: In some of our conversations with Microsoft, they’ve talked about modeling tools that they use similarly to what you’re describing; before code ever gets checked in, the developer does analysis of the code and fixes issues that are flagged.

Talk a little bit about the benefit of having that done at the developer level versus, for instance, in a QA process that happens once or twice a year.

Gwyn: If you do take a milestone approach or do it just before you roll out, the challenge tends to be that you get an overwhelming number of issues to deal with, often all at a critical time in the product life cycle. From the individual developer’s point of view, it can be an awful ordeal.

Scott: In other words, the developer would tend to build up a big pile of issues that all come back for resolution at once, when they are periodically unearthed. On the other hand, if the issues are dealt with proactively, you never accumulate that big, ugly pile of bugs.

Gwyn: Right. The other thing to consider is that this ongoing system means that the developer is dealing with mistakes that they made just recently, instead of six months ago, so it’s still fresh in their mind.

Sean: You mentioned that you guys do some work with different open source communities. Educate me about your input into their process–are there open source analysis tools that are commonly used?

Do you find that developers are running tools that they have available prior to submitting code to the mailing list, or is it more the case that they really just trust the mailing list and are also interested in the scans that you and other vendors do and what those uncover?

Gwyn: There is no doubt about the wisdom of the thousand eyes approach, particularly for the more mature open source projects, and people do use as many techniques and tools as they can get their hands on.

In the Java space, there’s a very nice open source analysis tool—FindBugs–that can really help developers get started with analysis. It doesn’t perform inter-procedural analysis, so in the words of the tool’s developer, they’re really focused on “low hanging fruit”, but it’s definitely a great place to start.

In the “C” world, well, not so much. There are a couple of tools that check coding style and things like that, but while they can tell you things like you should declare your copy constructor as private or you might generate memory leaks, their ability to find actual defects (as opposed to poor coding constructs) is very limited, unfortunately.

Sean: And sometimes, it’s really hard to spot with the naked eye looking at apps committed to a mailing list.

Gwyn: Exactly.

Sean: This is in kind of a different area, but given your perspective and what you’ve done in terms of looking at a variety of codebases over the years, what tools do you think organizations should utilize to ascertain whether a given open source project is secure?

That’s assuming they’re not going to go buy a tool, do a massive code review or utilize a third party; they’re just out there having the internal conversation of Apache versus IIS, or MySQL versus SQL Server, or MySQL versus Oracle.

Different people make different claims about which is more secure, but how do we prove it? You end up in this kind of absence of evidence argument, I think it’s termed. Scott did a look through one of the mailing lists for one of the major projects, and he didn’t see anything about threat modeling. That doesn’t mean it’s not occurring, but that also doesn’t mean it is.

What’s your take on that? What should an organization do to really ascertain whether a given project has a robust development lifecycle around security?

Gwyn: To be blunt, they should do everything they possibly can. You just have to assume that the guys on the other side of the fence haven’t done it yet. Do you believe Coverity, who says their favorite 11 open source projects are perfect? Or do you believe Fortify, who comes out two weeks later and says the sky is falling and they’re all insecure? Do you believe Mozilla, who says they have a thousand eyeballs looking at their code all the time? Or do you believe the users who say, “Yeah, but I filed almost as many bugs against you as I do against Microsoft”?

Nobody has the answer to this question. I’ve talked to several people over the years who said, “Wouldn’t it be nice if we could put together an indemnification agency that would use tools like yours to help some of our customers have more confidence in open source?” The idea would be to say, “If you want to use Apache (for example), here’s everything you need to know about it. Here’s the support protocol around it, and most importantly here’s a security rating for it.” In essence, they’d be attempting to provide a warranty for a particular revision of a particular open source project.

There are various organizations around the country and around the world who do little bits of this, and some of them are quite successful, but no one has ever gotten to the point of taking responsibility for saying “Tick or cross: you should use this.” It all comes down to providing a lot of data for the customer to evaluate, because no one is willing to take on the legal liability of saying what people should consider safe to use.

Because of all the sources out there and because all the tools and techniques are out there, you’d assume somebody would at some point do it. But there are enough litigious arguments against it that I don’t think it will ever happen, unfortunately.

I think it would be a great thing to do, and I think that most of the senior committers on those mature projects would want to see that happen as well. But they’re giving their time for free; it’s not something they’ll ever want to do themselves. And I think any commercial entity is just going to be scared witless of some bank somewhere going, “You know what, you told me to use X.Y.Z of Apache, and it turned out it had this vulnerability in it. We’d like you to compensate us for Grandma’s password finding its way to the Ukraine.”

Scott: We’ve talked a little bit about the value of static analysis to define things that aren’t necessarily easy to see with the human eye, and the value of doing it up front. One of the interesting things I find in open source is that with a proprietary company like Microsoft or Oracle, for the most part they own the code from soup to nuts. If there’s a bug, there’s a place to file the bug. They fix it, or they don’t fix it, but it’s their codebase.

When you’re dealing with something like a Linux distro, on one hand you’ve got this downstream distro that’s packaging a lot of upstream projects. And so, you’ve got the distro provider who has hired people to do the packaging and things like that. But, they’re also paying people to work on these upstream projects, whether it’s the shell, the web server, the kernel, or whatever. Then, when a vulnerability happens, there’s a relationship necessary for a fix posted to the distro to get posted to the upstream project.

Distros also talk about things to make these upstream projects enterprise ready. In your experience, do you find at the distro level that people are using tools like yours so that if an audio player for example didn’t go through the same kind of analysis upstream, they maybe are doing it downstream and working with the upstream project to improve the quality of the code?

Gwyn: Wouldn’t that be a good way of organizing life? Unfortunately, it doesn’t happen so much. Some of the distributions today are backed by serious individuals or serious companies. It’s not just two guys off in Europe trying to bang a CD together, and in some ways, they’re doing phenomenal things. Look what Ubuntu has done around the user experience. They’re doing amazing stuff for the operating system as a whole.

I would say the distro manufacturers as a whole certainly validate in an industrial strength way and do the needed extra development, but they don’t really take responsibility. In terms of indemnification, guarantees of quality, or anything like that, they tend to fall back on a very dinosaurish kind of mentality. Got a problem? Hey, it’s only a patch, suck it up.

In just the same way as indemnification players don’t really want to step up and say it’s secure, distro vendors really don’t want to take this step of guaranteeing not just observed quality but absolute ground-up code quality, because of the exposure that would entail with all kinds of people over whom they have no control. Who do they go to if they have to build a fix on some kind of SLA? Of those thousand eyes, half of them are asleep at any given time. And the other half work for somebody else.

I’m certainly not saying you should go and buy commercial software all the time because you can always go and get hold of somebody. At the same time, though, the community just doesn’t have the level of leadership yet that lets it take that particular kind of responsibility for the quality and security of the software they’re producing. Perhaps the fact is that there’s just not that level of requirement being surfaced by the enterprise customers.

At the end of the day, nothing’s ever free. I think that open source has absolutely proved that software can be as free as you like, but that’s just one small component of the total cost of ownership of these things.

Scott: I imagine that you run into the idea of a security IQ that starts out fairly low, where people say “Well, we haven’t had a problem, so we must be doing things right.”

Gwyn: [laughs] And my five users are all very happy with it.

Scott: Right. So, how do you approach educating somebody about the fact that just because nothing has blown up and it hasn’t been the end of the world, they might not want to assume that everything’s hunky dory.

Gwyn: It is actually fascinating how you can never feel anybody else’s pain. I think that’s a rule of life, whether it’s physical or intellectual. You can’t actually empathize with all the people who are going through it until you’ve done it yourself.

At one point early in my career, we got this very polite email from the DOD saying that they had taken 65 servers offline because they found a vulnerability in one of the products I was responsible for. And this was back ages before anyone was really worried about such things.

It was a tremendous wakeup call to realize that we had just taken away a significant capability from a very important agency just because of an error on our part.

They were very polite about it and didn’t intend to cause any significant commercial ramification, but as professionals, how were we to get to the root cause of it? How would we stop it from happening again in the future? Until that moment, I had never felt that pain, and I didn’t really understand it.

Increasingly, we get people coming to us from the device space to ask specifically about security. And they’re not coming to say “OK, we know we need to worry about this;” it’s “why would I have to worry about this? Everyone’s talking about this and I don’t get it.”

So, we were talking with this one medical device manufacturer, and they make truly scary stuff. If this stuff goes wrong, it’s not going to be happy days for anybody. Their senior architect was sitting in front of me and he said, “Marketing tells me I have to put an IP cable on the back of this box because he wants the paramedics to be able to send stuff off to the hospital before the guy gets there.”

And I asked him, “OK, and what are you doing about making sure that your device is doing what you think it’s doing and hasn’t been hijacked or something?” And he responded, “Why would anyone ever want to do that?”

Explaining to people that just because they would never think to go and hijack somebody’s super-fridge or their pacemaker or whatever it happens to be, doesn’t mean somebody else out there doesn’t have too much time on their hands and would just think it’s a huge joke.

In many ways, it’s that first moment of enlightenment that’s the hardest for people. You’ve got to get across that notion that just because they would never do it, and just because they can’t understand why anybody would, doesn’t mean it’s not going to happen to their system or device.

Once they’re through that, then it becomes a very pragmatic process, but that very first thing you’ve got to have is for that senior architect, VP, or whoever to feel the pain.

Scott: It seems to me that in a way, Microsoft got a little bit lucky (though I’m sure they didn’t feel that way at the time) back in early 2000, when Code Red, NIMDA, Blaster, and Slammer came out. These really high profile exploits were very embarrassing, and they were the kind of exploits that brought a whole company network down. And so, they decided to just pour dollars and resources into solving the problem.

Today, it seems like the nature of exploits is very different. They are far less likely to absolutely saturate your network and let you know that you’ve been hit. It’s more about setting up a botnet and what some people call crime as a service.

It’s a lot more valuable to have Sally not know her computer’s been compromised, or to have some Linux server in a data center doing command and control for a botnet, without its admins knowing about it.

Like I said, Microsoft got lucky in the timing and the nature of the exploits. In some of these projects, there’s a false sense of security because they say “Well, where’s our NIMDA? We’ve never had one.” Well, they’re never going to have one, because that’s not the kind of exploits people are writing.

Gwyn: Yeah, exactly. Over the last ten years, we have moved from script kiddies doing dumb things to these really very, very smart attack vectors. It is a whole new day.

We can’t go a whole interview without me poking at Microsoft. [laughs] Look at the animated cursor type of thing that hit the public eye a year ago. This was vulnerable code that had been in Windows since day one, but all of a sudden, out of nowhere, you don’t have to see the person, you never have to see their machine, you can do everything remotely, and all of the sudden that thing is yours.

And what’s more, if you’re smart enough, the person who’s getting compromised doesn’t know what’s going on. It sort of became that perfect storm of an attack vector. And at the end of the day, what is it? It’s a simple exploit of a buffer overflow vulnerability, albeit a phenomenally complicated one. Those guys should get a PhD just for creating those things. [laughter]

You’ve got to give them kudos, right? But, that happened at the height of Microsoft’s secure development life cycle, and that bug had been reported. It was in their queue, and they knew all about it, but it had been de-prioritized and was buried in a pile of sixty eight thousand other things they had to worry about.

So, they are at one end of the spectrum because they have so many products and so much going on, but if you take the whole open source domain as a thing, there’s probably more code there. And you know that same level or magnitude of vulnerability is out there.

Scott: That’s one of the benefits of tools like yours. On one hand, there’s putting it in the hands of every developer so that they can have a ‘get clean, stay clean’ sort of mentality. But, the other value I see is that the most dangerous code in the world is probably the old code that nobody is looking at anymore, because it was written before people knew about integer overflows and buffer overruns.

I remember, way back when, we bought these Unix systems from a vendor. We were writing some low level network code. We did something that crashed the server. We had a bug in our code, and it was sending a packet where the frame was the wrong size or something. The response from the Unix vendor was basically, “Hey, there’s an RFC, and if you’re operating outside that and bad stuff happens, that’s not something we’re going to worry about.”

[laughter]

Gwyn: There you go. Good rationale. On that same tack, IBM’s X Force does all these amazing things all the time, like showing how forcing an invalid pointer usage can compromise Flash applications, etc. — seriously smart people.

One of the things they do is to issue a “top ten” offenders list for attacks during a year and something that struck me was, I think, in last year’s report. No prizes for guessing who is the topmost vulnerable vendor, because Microsoft wins every year. But, of all the reports that they had coming to their NOC over the year, Microsoft products were responsible for three percent. Three percent of all the reports had something to do with Microsoft software, somewhere along the way.

The fascinating thing I saw was at that number five or number six–I forget exactly where they sat–was Mozilla. How many products do they make? They have one fantastic browser I use every day, and some other random stuff that is much less popular, but they are responsible for 1.4 percent, 1.3 percent … don’t quote me on the figures, but some relatively close number of available vulnerabilities, compared to all the things that Microsoft puts out.

Now, in reality, how many of the things that Microsoft puts out are actually getting attacked at the same level that Firefox is getting attacked? But still, it makes you think… Oh, and that top 10 was only responsible for about 15 percent of all reported attacks, so over 80 percent were against software and vendors you’ve never heard of.

Scott: Well, I want to be sensitive to the time. We’ve covered a lot of great stuff, but what have we left out?

Gwyn: If I were in your place doing the interview, I would ask what’s wrong with these tools? At face value, they are no brainers. Everybody should be using them, so why not? What’s the barrier to acceptance? Is it just because they are very new? What’s really going on here?

We try to answer this question for ourselves all the time. Why is nobody in this market a Microsoft yet? Why are we still a very nascent segment? Obviously, there are some elements of market maturity there. Most of the companies in this space have been around for a very short time.

It doesn’t matter which community you are working with, whether open or commercial. There is an overwhelming thought process involved around analysis. People tend to look at this process the same way that we started this conversation about it today–thinking of this as an audit environment.

People tend to regard it as a milestone activity and as something that is an onerous, annoying part of the software development process akin to code review. I can’t imagine anything more mind numbing than having to revisit the days in which I did code reviews all the time.

If this kind of technology is equated to that area of things, that’s a problem for this industry, and for the service technology type. It behooves companies like us–and open source projects who are trying to do the same sort of thing–to get over that, and to engage with the developer in a meaningful discussion around what you actually need out of one of these tools.

Is it that you need the most accurate tool with the lowest noise ratio? Or do you need something that finds every bug imaginable, regardless of the noise ratio? Do you need something that’s really fast, or that integrates with every environment, or something you never see and just pops up when you do something dumb?

For various vendors in this space, we have markedly different models as to how we engage with development organizations. As for ourselves, we are very upstream. Everybody we ever talk to is in development. Other people in our space are very downstream. They talk to auditors and CFOs–guys who in the morning are buying firewalls, and in the afternoon, they are buying software validation tools. There is a completely different feel to how you engage with a development organization, when you start getting that viral distribution down at the volume end of the buying pyramid.

I think it’s an interesting challenge in this space–how to get that massive distribution. When did IDEs suddenly hit the tipping point from being a bizarre luxury that a few companies used, to being something that every developer on the planet sees as a right and a prerogative?

It’s the same with runtime profilers. It’s the same with memory profilers. It’s the same with all these tools, which are just part of today’s normal, everyday SDLC.

When did they get there? How long had the market been around? Is it just a volume-based thing? Is it a market approach thing? I think that’s the fascinating question in full scale analysis now. Obviously, it has value and is having great effect in both commercial and open source, so when does it go big time?

Scott: That’s a great closing thought. Thanks for taking the time to talk with us.

Gwyn: You bet. Thank you.

Bookmark this:

Interviewee: Chris Messina

In this interview we talk with Chris. In specific, we talk about:

• Meritocracy and flexibility in the open-source approach
• Bringing together designers and programmers into a cooperative whole
• The new wave of portable devices and the role of Linux
• Deciding how many features are useful and where bloat sets in
• Analogies from the natural world in project evolution
• Seeking balance between backward compatibility, future-proofness, and open innovation
• The next revolution in personal media on the Internet

Sean Campbell: Chris, can you get us started by telling us about your background?

Chris Messina: Sure. After I graduated from college in Pittsburgh, I came out west and started working for myself as a web designer. My background is actually in design.

I ended up working for a little non-profit called CivicSpace and then volunteering for the Mozilla Foundation. That was my first real foray into open source software, and I was immediately taken with how things are done and the way meritocracy rules.

If you have time, motivation, dedication, and you produce good work–or you just produce a lot of it with a couple of hits here and there–you tend to be able to rise up and influence projects.

I started volunteering for the Mozilla project and working on a project called SpreadFirefox, which ended up becoming a community marketing hub for the launch of Firefox.

Being involved in that project really gave me an understanding of the power of open source development’s decentralized nature. It also showed me some of the shortcomings of that nature, primarily in terms of getting the message right in talking to non-geeks and non-nerds.

Firefox is a breakaway success for a number of reasons, and it just so happens that I was able to ride in right at the right moment. My background in design and some of my specific interests were a good fit.

I was very inspired by the Howard Dean presidential campaign and how it was run. We wanted to do a lot of the same things with Firefox–make Firefox a “campaign for president” on the Internet.

I tried to get hired by Mozilla, but instead I ended up helping to co-found the social browser based on Mozilla called Flock. That’s because I also believed that the browser needed to become a much more social vehicle for the web. That was where the next generation of stuff was going. It was around 2005, I think.

During that summer, I also helped to organize an event called BarCamp. We took a lot of inspiration from open source software processes, applying them to the real world, creating an event model that we wanted other people to be able to pick up and run themselves.

That has also become a very successful decentralized example of what we call a “starfish initiative,” where people break off from the main effort and create a whole new organism that has its own characteristics and that thrives in its own environment.

I did Flock for about nine months and became a little bit disillusioned with the direction. I ended up leaving and starting my own company with Tara Hunt where we did consulting around social media and social marketing, project design and project strategy, using an open source approach.

We started what we called coworking space that initiated this other movement to create independent workspaces for people, again bringing an open source approach to it. That’s been a fairly successful project as well.

I’ve been doing a lot of work on the social web. We started our project a year ago, called the DiSo Project, trying to create a series of open source, non-proprietary building blocks for building social web-sites in a decentralized way.

I’ve done a lot of work with microformats and OpenID and so on, and so to come full circle, in May I began to work full time on the DiSo Project.

A lot of what I’m doing is trying to make these social web building blocks usable while maintaining a decentralized approach where there’s no one central authority–essentially modeling the experience after the way the Internet has developed.

Sean: You mentioned that there was a connection between being a designer and getting involved in an open source effort. Given that you’ve interacted with and helped to kick off a bunch of open source related efforts, what do you think about how projects view the role of designers?

Chris: There are a lot of reasons why designers and engineers, specifically coders, are like oil and water when they come together. I think we have problems with language, where we talk about things in very different ways.

There’s a degree of art in both disciplines that the other side might not fully appreciate. You can imagine a coder having written some algorithm that’s not very efficient, so a number of other developers might come in and propose alternatives or provide patches and improve that algorithm. Once they have committed the change, everyone can see that the process has increased the quality of the product.

From a design perspective, it’s much more subjective. You can objectively increase the performance of a code project. You can objectively patch bugs, increase security, and so on.

With design, quality tends to be in the eye of the beholder. It requires a certain type of design leadership that knows the rationale and reason for design. They might say things like, “Well, it’s just better. It’s good.”

Sean: Rounded corners felt good to me here.

Chris: That’s right–much of it is simply intuitive.

Sean: And everybody looks at it and says it looks good but then whether you can rationalize that point of view is the challenge.

Chris: Rationalize it but also quantify it. A lot of developers tend to prefer paring things back to their most bare essence, without embellishment. I think that that can be somewhat jarring for people if they are not initiated or not familiar with a certain idea.

We’re dealing with that issue right now with OpenID, where there is a visceral need that some people are experiencing where OpenID could represent a type of convenience for them. For example, whenever you go to the grocery store checkout and you hand over your credit card, you don’t type in the 16 digits on the credit card; you swipe it.

That magnetic strip has those 16 numbers encoded in it, not because you can’t type numbers but because it’s more convenient. Eventually we will get there with OpenID, where people realize this is a little bit easier than having to remember all these different passwords and things like that.

It also ties to something of greater value, but communicating that from the land of developers where it came from to the land of individuals who may really not care or for whom the technology is confusing enough to obscure the benefit is a long jaunt.

Those elements of subjectivity in design can drive developers insane.

I also think that designers can tend to be megalomaniacs around control over seeing their vision executed in a certain way. When you deal with open source, it’s never mostly about ego–or at least it shouldn’t be.

To go back to that meritocratic approach, if someone else comes up with a better design, even if it’s less aesthetically pleasing but more effective, it’s hard for designers to have their designs shot down by people who are not trained in design.

I know that I had that problem where I used to want to design things for Firefox, and it had to be absolutely perfect before I put it out to the world. And yet, open source teaches you to get stuff out there as soon as it works, to see if people have solved the same problem and so on.

It takes a very different ego orientation to solve problems effectively in open source, and that’s not something that necessarily comes from design training.

Sean: In my experience, many designers like to create in isolation and then bring their work forward to the world once it’s completely done. The iterative, collaborative nature of open source projects doesn’t tend to work that way, and there’s not really a good interface point for you if you like to work that way.

Mozilla has made an elegant browser, and they don’t let things get too much in the way. Even when you pull down an extension, most of them work reasonably well, and you move things out of the way pretty fast.

Ubuntu is more likely to create an improved presence menu than they are to, let’s say, put some more technical implementation pieces in. How do you think those projects get a good interface point for designers, whereas other projects just never quite seem to get there?

Chris: WordPress is, of course, another one of the exceptions out there, and I’ve been involved to some degree in that community. I’ve also had experience with the Drupal community, and I’m going to New Orleans next week to speak at the Do it with Drupal conference.

I think part of it comes down to DNA–where a project came from and the founders’ preferences or appreciation for design. I also think that it has to do with how close the developers or designers are in sympathy to the users. Developers tend to develop for themselves, and they have a certain mindset of the world.

One way that designers can approach the problem in terms of advocating on behalf of users is using data. Developers love data, and they don’t like to argue with it.

I think Steve Jobs is at the top of making sure that the stuff works for people, regardless of what developers might think is best for the customer. Google is another good example, where both the designers and the engineers have a huge appreciation for design, in terms of being able to determine whether it will actually get used by people.

And that appreciation leads to understanding that you can add 4000 features, but maybe you will lose 20 to 50 users or potential users for each feature you add. That changes your perspective.

You also need to have someone that has vision and is willing to take the arrows in the back from all the interim parties, if they really believe and somehow have justified a certain design or design approach.

Plenty of designs have been really unintuitive at first, and counter to what people would use or do, that ended up becoming very, very common and very useful. Sometimes that’s a matter of mad design genius that’s willing to take that risk and push forward and do something good.

If, on the other hand, you have a vacillating design aesthetic where no one is really leading (or everyone is leading), then I think it can be very, very damaging. That creates a very unstable circumstance where designers can’t actually contribute anything meaningful.

With Firefox, they actually put a lot of complexity on the part of the extensions, and they actually absolved themselves of a lot of those issues. Also, open source does a really good job of commoditizing existing solutions and making them a little bit better.

They basically took Internet Explorer and Netscape, cut out all the crap that developers really hated from the Netscape code, released a pretty solid product, and really didn’t add all that much complexity.

Firefox didn’t spring out of the head of Zeus in perfect form; they already had a model that they saw wasn’t really working that great. Netscape already had tabs. They supported web standards, which is great for the web standards community, and created a product that was cross platform.

So in some ways, Firefox absolutely gets credit for cutting out all the crap that the Netscape people and the advertising and marketing people put into that product. But you can also say that they had a keen eye on their users and did a great job of making something that their moms wouldn’t cry over using.

Sean: With the introduction of Netbooks and very intelligent phone-based devices, there’s a lot of discussion–and rightly so–about tuning Linux for these new form factors. There’s a lot of stir about Linux on these devices, but at the same time, lots of users don’t really want to know whether it’s Linux or something else.

They just want to use it, in the same way that they use Tivo, where they wouldn’t know or care that it was Linux under the hood, unless someone happened to mention it to them. It seems like there is a little bit of a potential speed bump ahead, and I wonder if you can see around it.

It’s great to repackage Linux because of the way it’s set up, but at the same time, there are some challenges creating a design aesthetic early in the life cycle when you have fast moving devices and form factors.

I heard somebody refer to the Android as a garage door opener, in terms of its looks. But then the funny thing is that somebody created an open source program that made it work as a garage door opener, literally. And somebody else got Debian running on it.

So there’s a weird mismatch between extreme usefulness and flexibility, paired with the desire on the part of some users to have the slick feel of something like an iPhone, right out of the box. You really don’t see that pairing of requirements in many Open Source products; how do you think we can migrate around it?

Chris: Either fortunately or unfortunately, open source tends to be equated with freedom, flexibility, and choice. That type of freedom and flexibility unleashes all types of possibilities and potential, but it can actually be debilitating if you don’t know where to begin and you become overwhelmed.

I think that’s the real tension within a new type of projects. Consider Facebook Connect versus Google Friend Connect or OpenSocial. Facebook is providing a constrained environment with fewer choices. You have one way of connecting, and there’s one style of font that is used to view a connection.

It promises a very strict set of expectations associated with the experience. On the other hand, OpenSocial and Friend Connect and some of the stuff that I’m trying to do with Vidoop projects don’t really constrain you the same way. They provide a much greater potential for expressiveness.

But that also means overloading the user with having to cut through all those different opportunities and so on. As a result, flexibility can actually end up discouraging users. The iPhone excels in providing a solid but restricted set of choices.

Through these constraints a great deal of potential was unleashed, but these are all constraints put upon the developer in order to, let’s say, create an interface that pushed people down certain pathways.

I think the original iPhone had one home screen, space for a few extra icons, and that was it. None of this sliding springboards around or things like that. How much more simple can you get? On the other hand, Android and other software focuses on maximizing the potential for expansion.

This is another working area of difficulty, where open source projects have to take away the freedom and expression that a lot of open source advocates tend to expect.

Sean: In an earlier conversation in this series, we talked about the kinds of things that a company can do with people sitting in a room and a white board, as opposed to the kinds of things that the open source community is really good at.

There is a certain amount of tension there, because traditional open projects like Apache and the Linux kernel accrete functionality over time, with one more configuration option, one more setting in the config files, and one more module. They just grow organically over time.

The point that earlier interviewee made is, I think, exactly the point that you make. A lot of times, usability isn’t about building functionality; it’s about paring it back to what is essential and intuitive and discoverable.

You have pointed to a lot of examples–iPhone, Google, WordPress–that are really good at that. Talk a little bit about that inherent tension between people wanting things to be very usable and the tendency for open source projects to get incrementally larger and more complex as they evolve.

Chris: I have a couple of interesting anecdotes about that. Jamie Zawinsky said that all software expands to support email.

[laughter]

Today, of course, all websites expand to until they support social networking. We’re seeing the same thing happening, where it becomes so easy to add on new features and new functionality. Does it make the software more usable, or does it just increase complexity?

I forget who told me this, but I remember hearing that if you get to the point where you need to add another configuration button, you have kind of failed.

To go back to the Firefox example, if you enter about:config in the URL bar, you can access a bunch of options that Firefox makes available to developers and crazy people who like to peek under the hood and switch the size of the nuts in the engine.

For the most part, the browser just supports the basic protocols, and the protocols work on both ends. You can tweak the settings all you like, and it may or may not make a big difference.

We’ve come really far in the expectation that people have a fairly powerful CPU and a good bit of RAM. And we’ve got lots of cache in the buffers now that enables software to continue to increase in complexity.

That wasn’t true back when they were creating the Apollo lander and stuff like that running on a 286 or whatever, and you had to count your bits. Today, a TechCrunch web page is a two megabyte download, just for text and graphics.

When I was doing web design in 2004 and before that, every little byte that I could shave off a GIF saved me all kinds of trouble, because you had to worry about download speeds. Now that has gone out the window.

My point is that there is always a balance to be maintained between helping people do the things that they came to do, and getting out of their way so they can move on. This is one of the fundamental challenges that we’re trying to cope with in OpenID.

People couldn’t really care less about how they sign in or who they give their passwords to, as long as the next step is that they’ve done something cool or interesting, or connected with a friend to do something social.

If we can make that process more secure for them and more convenient, then we’re helping the ecosystem to grow. That’s our challenge, not to add additional burdens as we’re changing things. There is a new breed, I think, that I call the hybrid developer/designer. The present time might be known as the birth of The Web Arts, just as we had Art Nouveau and stuff like that back in the day.

Developers and designers are starting to achieve a certain mastery over some of the technical implementation tools that with things like Django and Rails that allow them to create very interesting pieces of work.

These actually work for people, because they have that sensitivity about reducing complexity, and an appreciation for typography and for creating messages that people can actually read and understand. I think it’s going to be very interesting to see how that type of feedback can create better software that creates a model that open source developers can emulate.

As I’ve pointed out before, open source is really good at commoditizing successful solutions that are usually proprietary or cost a lot of money, so I think that would be a good thing.

Sean: It seems that as certain projects increase in complexity, they become hopeless and something completely new has to come out. Everybody realizes that they really didn’t want all the stuff that they were asking for, and they move on to the new thing that doesn’t actually have all that stuff but actually does what they need it to do just exactly right.

As a development framework, Rails is very intriguing to some people.

That makes me think of MySQL and Drizzle. If you look at Monty’s recent posts, he’s none too happy about where MySQL is going. That doesn’t mean that MySQL is a bad project. It just means that it has reached a point of diminishing returns from his perspective, in terms of what he wants to accomplish with his developer effort.

Scott Swigart: A tree eventually just falls over, collapses, and rots under its own weight. But hopefully brand new ones have grown around it. Do you see that evolution?

Chris: I tend to look at the way that nature and biology work to try to understand how something will be successful in the wild, and how something might grow organically on its own. I think your analogy fits very well with the one that I tend to use, which is one of a forest fire.

There used to be a lot less understanding of the cycles in the great American forests, like those in the national parks. Standard policy was to try to put out fires as fast as possible, which resulted in underbrush accumulating over the years, so that eventually, much larger, more destructive fires happened.

We had interfered with the natural cycle of fires in the forest that get rid of the deadwood and things that just don’t make sense biologically and evolutionarily anymore.

Firefox is a great example of this. The original project name was “Phoenix,” which made a lot of sense in terms of having forest fires in projects to essentially burn out a lot of stuff that doesn’t make sense anymore, that no longer works in the environment.

One needs to start from the DNA that helps a system to work well, and to adapt it to build a new environment. I think we’re seeing a similar process as Rails reaches a point of maturity.

I look at things like Django, which are more recent still, and I see new ways of developing things that draw on a lot of the lessons that came out of what makes Rails an interesting development framework.

The fact that you can store things forever and keep on increasing the complexity of things is actually not the way that nature works. Nature tends to be the most resilient when it is reduced to its barest essence and it passes on its core elements to the next generation.

You see some legacy systems that just don’t work anymore, because everything in the environment has changed but their core makeup has not.

Sean: Just to continue to stretch the analogy, what about when the forest fire never goes out, and it burns things up almost as fast as they’re being created. The organism doesn’t really ever reach maturity before the next thing comes along.

I was sitting through a webcast today–“everything you need to know about Drupal but were afraid to ask.” A notion came up that I hear over and over again, which is that backward compatibility isn’t really that important. It’s more important to get things right in the next version, and just continue to move it forward.

Does that create a barrier where people will hesitate to build on foundations that don’t have a certain level of future-proofness to them?

Chris: I think it depends on their expectations. To go back to the biological model, somebody told me yesterday about the way that numbering works in Apache projects. I’ve seen 1.2.8 type version numbers forever. I never really understood what makes something 1.1.4 versus, let’s say, 2.0.4.

I guess Apache had a pretty smart way of numbering things, where incrementing the number after the second decimal point basically indicates an incremental patch release, where full compatibility is maintained.

Incrementing the number right after the first decimal point represents a change that is a little bit more significant. Some APIs might break, but for the most part, things are going to be consistent between that release and the previous one.

A change to the number before the first decimal point means that all bets are off. It could be a totally different product.

Drupal has been very interesting in the sense that they have not tended to maintain backward compatibility, which has actually been good for them. Had they tried to maintain backward compatibility all the way back to 3.0 and 4.0, not only would that have impeded growth, but I think it would have actually led to a great deal of stagnation.

If you look at successive generations of humans, it’s important to leave the past behind in some ways from one generation to the next. We need to teach the kids a slightly different way of thinking about things and help them be a little more worldly. Otherwise, we end up with stagnating societies, which can be problematic.

That analogy also holds in software development. A project continues to grow and change, and to cut off pieces of itself that don’t make sense, in order to renew itself. That’s a very positive and productive path. If you’re only worried about future proofing your stuff and never upgrading it, that starts to look a little bit like death.

Moreover, human beings should be doing productive and interesting things, so it’s not altogether bad that we work on these projects for software add-ons and items that give us something interesting to do instead of sitting around and letting the robots do the work for us.

You can also specialize. If my body were a single-celled organism, I wouldn’t be able to carry on this conversation. Instead, as it has grown and gotten more complex, my body has then shunted off functionality to different specialized systems.

So now, for example, instead of IT directors having to manage storage, bandwidth, and getting content all over the world, they can use Amazon for storage, which takes care of that particular brand of headache. The IT director just has an abstract API to deal with, and they use Google Apps for their domain.

We have specialized organizations that essentially do all these things that a lot of other people have to look for. They also continue to rev themselves, and continue to grow, change, and adapt to the environment, because that’s what they’re supposed to do. That actually is more of an argument for pieces loosely joined in some sense.

Scott: But those individual pieces can only rev so fast. If Amazon S3 did something that broke every single thing that was using Amazon S3, or even 40% of it, people would have a lot of trouble trusting the next Amazon S3.

Chris: That’s possible, although those types of changes don’t tend to happen, which I think has to do with communication and transparency. I think it also speaks to smaller, incremental, and agile changes. Again, that design philosophy of trying to create the perfect, immaculate solution and unleashing it on the world tends to create a lot of brittleness.

A lot of these companies are also more accessible than before, and they are using social media as a feedback loop. Even though it was excruciating for all of us on social media using Twitter over the past summer, when it was going down all the time, most of us have forgotten about that now.

As long as you get your stuff together within a couple of months–or ideally a couple of days or hours–things move on. I guess I hear and appreciate the argument that you’re making, but on the other hand, I would still rather have Twitter doing what Twitter does as opposed to running my own Twitter server.

Scott: Absolutely. Well, I want to be sensitive to the time. The way we usually finish up is to ask if there’s something in particular that you’d like to close with.

Chris: Yes. We’re standing on the brink of the next big revolution or evolution on the web in personal media and the way that people understand and relate to technology and to the way in which it can connect people. There are going to be a lot of new challenges, threats, and opportunities that will bring with them a lot of promise in terms of what people will be able to do with very low cost.

This new generation of hybrid developer/designers I mentioned will be able to create very straight-forward solutions for people that hopefully will draw on the best elements of open source, namely its ability to collaborate, to promote things from meritocracy, and to redistribute knowledge. Those solutions will emulate what Google has done, in the sense of increasing the amount of knowledge that’s accessible to the world while still finding success for itself.

We’re going through a technological revolution that changes the way the social web works, which changes how people can make a living and work together. I think we’re being connected internationally in a way that we haven’t been before.

A lot of what’s taken for granted in the open source community, in terms of ethics and processes, will become a lot more widespread.

In fact, I wouldn’t be surprised if elements of the Obama administration, for example, look to what has been successful in the world of open source as a model. The networked world in which open source has grown up and matured has become a cultural force unto itself.

Those are things that I’m looking forward to and am personally very excited about. I think that open source has become sort of a culture within itself that has its own processes that work, and we’re going to start to see its impact on education and other aspects of the world as well.

Scott: I agree–one of the ways to view that transformation is that the distance between an idea and everything else has become so much shorter. It’s especially related to your ability to effectively use all the building blocks that are available. We have the S3 and the Mechanical Turk and all of these organs of the Internet.

Using those effectively can drastically reduce the cost and time needed to take a great idea and develop it into something functional.

Chris: And on the flip side of that, the barrier to offering real value in terms of cost is going to go up considerably. You’ll be able to solve your problems building great little web apps that do a lot of really low complexity tasks for you.

At the same time, though, we’ve democratized technology dramatically and lowered the cost of publishing for everybody. It seems like everybody’s got a blog and everybody’s got a Twitter account.

There’s going to be so much information out there and so much data being produced that it creates the next generation of opportunity for a big Google type of business to emerge. That is, in helping people make sense of all the stuff that’s coming out as well as crystallizing the essence of content.

While we’re lowering the cost considerably to do things that used to be pretty hard and pretty nerdy on the web, we’ve now moved the level of complexity that’s necessary on the programming side, up several notches.

Sean: That’s a great place to wrap up. Thanks for taking the time to talk with us today.

Chris: Thank you.

Bookmark this:

Interviewee: Paul Cooper

In this interview we talk with Paul. In specific, we talk about:

• Getting started with the GNOME project
• Devices targeted by GNOME Mobile
• Differences between GNOME and GNOME Mobile
• Establishing a design approach for meeting the needs of users and devices
• The potential for virtualization on mobile devices
• The relationship between GNOME and providing kernel and hardware support

Sean Campbell: Paul, tell us a bit about your background, your relationship to GNOME, and how you got involved.

Paul Cooper: I’ve been working in GNOME and open source IT for a bit more than 10 years now.

I guess it started as a career when I was working as a systems and database administrator in the Mathematical Institute at Warwick University. We used Linux and open source quite heavily in our infrastructure. We also had a department head who liked to make promises but didn’t always like to assign budget to them. So, sometimes, we had to make things happen with little or no budget, which led us to use open source software quite heavily.

I found out about GNOME purely as an interested user; it looked like it would make my life using the computer more enjoyable and easier. When we rolled out a new lab right around the time that GNOME 1.0 was coming out, we decided to use GNOME.

Putting that lab together, creating user accounts, setting up a default look and feel in terms of how the panel and applets were loaded, and various other aspects of administration was my first real involvement with GNOME.

On the basis of that experience, I wrote the first system administrators’ guide to GNOME, which I hope is no longer in any Google cache, Wayback Machine, or the like. [laughs]

A lot of other, much smarter folks have rewritten all of that documentation, and obviously, the technology has changed completely since those days.

Beyond that, I wrote and edited little bits of documentation here and there, but I was really more of a hanger-on to the project at that point–someone who was there evangelizing and tracking. I was talking to people about it, helping in the local Linux user groups to submit bug reports, and that kind of stuff.

I was not heavily involved in the project until almost 10 years later, when a bunch of us here in the UK decided to put together a bid to host the annual GNOME conference, which is called GUADEC.

The conference travels much like the Olympics, and every year people bid to host it. It was probably about three years ago now that a bunch of us, primarily myself and a now-colleague called Thomas Word, decided to put together a bid to host that conference.

Thomas and I had met while doing various trade shows, where we put together a GNOME stand to help get feedback from people who were using it and to show new people about GNOME. Somehow, our bid to host GUADEC in the UK was successful, and we hosted GUADEC in Birmingham last year, 2007.

Those are my community involvements with GNOME. Quite separate from all of that, I started work at a company called OpenedHand. We help companies build amazing devices where they sell software primarily based on the GNOME Mobile project.

I helped with business development and just generally building the company, which has been acquired by Intel. Now, we’re working on the Moblin project which, again, is based on GNOME.

Scott Swigart: You alluded to the fact that GNOME is particularly well suited to mobile devices.

Paul: The GNOME Mobile project is really a subset of the GNOME Desktop technology, with the addition of some other pieces that are ideally suited to mobile devices.

When we talk about mobile, we’re not necessarily talking about mobile phone handsets, although it could well be used for that. We mean a much broader range of consumer electronics and embedded devices–really, anything that has a UI.

Scott: When people think of mobile devices, some people immediately think about netbooks, and some people think of handsets. From the perspective of GNOME Mobile, what falls under the umbrella of mobile?

Paul: Netbooks are at the large end of the spectrum, and the OLPC devices are also built on GNOME Mobile technology. Then you have things like the iRex eReaders and some of the Garmin GPS devices, which are also built on GNOME Mobile stuff and Linux.

Then there are so-called MID devices like the Nokia N810 and similar Atom-based devices.

Another thing we worked on was the Vernier LabQuest, which is an educational device for data capture and analysis. They have a catalogue of something like sixty or seventy different types of sensors for things like wind speed, temperature, weight, acidity, water flow, and others.

Then at the other end of size and power are mobile phone handsets, where OpenMoko, LiMO, Purple Labs, and Azingo are using subsets of the GNOME Mobile stack.

There’s really a very broad span of devices.

Scott: What are some of the fundamental differences between GNOME Mobile and the parent GNOME project?

Paul: We strip out a lot of the old deprecated stuff that still needs to hang around on the desktop for backward compatibility, and there are also mobile-specific components like matchbox, a window manager that’s highly tuned for mobile integrated devices. But the idea is that we use, and optimize, the mainline codebase as used in the Desktop, in the same way you have the same core Linux kernel from a phone up to a huge supercomputer.

Scott: How does the decision making process go around that? For example, one project might choose Empathy as their IM client, while another one will choose Pidgin. Someone also has to choose to make one particular change to the user interface over a different one.

Is there a distinct decision-making process for GNOME Mobile, or is it just all rolled up under GNOME’s broader decision-making process?

Paul: One of the differences between GNOME Mobile and the GNOME Desktop is that with GNOME Mobile, we’re really trying to build the foundational pieces that someone would use to create a specific device, rather than creating the entire interface and application space as well.

Whereas if you download all of the source for GNOME Desktop, and use something like Gentoo or Linux or Scratch and compile it all yourself and build it up, then you end up with a desktop that you would recognize with applications that you can use.

Within GNOME Mobile, we’re more about the foundational pieces that someone could use and custom tailor and build on top of to create a unique experience, so it’s really not the same type of end product as the desktop.

Sean: What are some of the more interesting observations or findings you’ve come across as you build for mobile devices?

Lots of companies have gone through various gyrations over the years, attempting to build functionality that would make a PC easier to use in a small form factor. Typically, they either try to up-level some platform that’s for a small device, or they down-level some functionality that is more common in a PC environment.

They go through various flashes of inspiration or pain, depending on how you want to look at it, and generally, they figure out that their approach doesn’t really work.

Since you’ve been involved in the project, what are the things that you’ve bumped into? What have been those flashes of pain or inspiration in trying to work with the mobile device, as opposed to a traditional PC?

Paul: One of the most important things I’ve learned is that the things that tend to be most successful don’t start with the technology–they start with what they’re trying to enable for the user. They have a user centered design approach and let the technology fit into that.

Scott: Considering somebody like Nokia, to some degree they take on the responsibility for figuring out what their end users want.

They own the experience, but they have the GNOME Mobile community-developed codebase as a foundation to build upon, so they’re not starting from scratch.

Do you think of GNOME Mobile as more about providing a good foundation for device manufacturers, rather than trying to be the complete end user experience?

Paul: Yes; our approach is very much to provide building blocks that they can mold to their device’s capabilities and what their users want to do with it.

Scott: That seems quite different from the approach being taken by GNOME for the full desktop, which tries to provide more of a complete experience that is fairly consistent from one distro to another.

Mobile GNOME might be so heavily customized that the user doesn’t even really realize that it’s GNOME underneath.

Paul: Absolutely. The basic reason for that difference is that all desktop or laptop machines are fundamentally the same, in terms of broad capabilities and how you interact with them.

There’s a keyboard, mouse, or trackpad, you have a screen at least 800×600 or bigger, and users have a certain expectation. The environment is very consistent, compared to mobile customer devices.

I’ve got a Sony Ericsson K850I with a quite small screen. I also have a Nokia N800 with a much bigger screen, but no keyboard or keypad of any kind. The phone doesn’t have a touchscreen, but the Nokia does. Some phones may have accelerometers, some may have GPS.

The usage models and how you interact with various devices is very different. It’s almost impossible to have a single interaction model or interface design.

Sean: Have you seen the recent news about VMware’s push to get into a virtualization stack for mobile devices? They made a big announcement the other day about working in that area to make it easier for users to switch among profiles and different OS architectures.

People have been talking about this type of advance for some time, but VMware seems fairly aggressive about it.

There may not be a direct impact on your efforts, but do you have any observations around virtualization on mobile devices, and what it might help you achieve in the future?

Paul: I saw the headlines, although I haven’t researched this story too far yet. Like everything in the mobile space, what we were doing with desktop machines five or 10 years ago is roughly where we are in terms of processor and storage capabilities in small devices today.

Virtualization is clearly a possibility on devices, and I am a bit curious to understand how exactly it will be beneficial–what the end user benefit of virtualization is.

Most of the benefit I’ve seen from virtualization has been on the server side. When I’ve seen it used on client machines, it was for IT professionals and developers to have multiple environments for development and testing–like running Windows and Linux on a Mac or multiple versions of a particular operating system to test with.

I’m not sure your average man in the street is even using virtualization on their desktop, so I really wonder why they would want to use it on their phone.

Sean: It seems that it’s predominantly targeted at the OEMs, but there are potential end user benefits, as well.

It will be interesting to see how it works in practice, but I’ve seen some discussion of being able to switch your user profiles between handsets from different manufacturers, regardless of what OS is running.

Because it’s abstracted, you could theoretically move your profile information from a Blackberry to a device running Windows Mobile or Symbian. It’s the same as when you hear people talking about compartmentalizing desktop apps into a VM, so migrating to another bare metal machine is simply moving the VM.

Scott: My impression is that a lot of input devices are handled deep down in the kernel, so that multi-touch functionality like the iPhone uses would require pretty deep changes to a system, beyond something like the GNOME shell.

Similarly, an OEM could design a stylus that might have a pen on one end and an eraser on the other end, and it would know how hard you’re pressing. I know at least on Windows that requires support really deep down.

Do you have sense of how to support those new human interface interactions if the GNOME project has to work with deeper things like the Linux kernel? How does that collaboration happen?

Paul: There would be some drivers that would need to be written in the kernel, but work would also be needed at the level of X. That would need involvement of the X.org project.

There’s already support for stuff like styluses and Wacom tablets in there, and we support those types of devices. Pressure sensitive input, having a pen and an eraser, and multiple pens that do different things are already supported, and there is development work going on within X to support multi pointer systems.

There is a conference called the Linux Plumbers Conference, which is one of the forums where this type of collaboration across the kernel, X, and the plumbing takes place. It’s not just for display devices, but also for sound, printing, and anything that cuts across these various areas.

Scott: So, the Linux Plumbers Conference is basically for these people at different layers of the stack who have to collaborate. It requires something deep down, but it needs to be surfaced in an architecturally sound way so that the things above it can really take full advantage of it. Is that where a lot of that work happens?

Paul: Yes. It also provides an easy way–even within companies that employ developers across those various projects–to get in touch and cross those boundaries.

Scott: Talk a little bit about the kinds of things that the hardware manufacturers contribute. I would guess that Nokia would have very specific things that they would be interested in to deliver the experience they want. Intel is obviously building chipsets and wants things to make use of the capabilities they put in the silicon.

Is a lot of what you see from the corporations that kind of deep hardware support?

Paul: It’s really a whole range of things. Most companies involved with GNOME Mobile provide some basic level of maintenance to the core components. Others push the capabilities and add features in areas that are particularly important to them.

In terms of hardware companies, it’s not something we may see in GNOME Mobile, but obviously they’ll be adding driver support in the kernel and X where it’s relevant to their hardware.

Other companies are involved that aren’t necessarily silicon providers. It can be the organizational structural support of building into their purchasing agreements, or putting pressure on their suppliers to provide open source drivers for their hardware.

Work in that area can be very useful. Although it’s almost tangential to GNOME in a sense, because we’re not at the level of hardware drivers, without them, we can’t run on a platform.

It also includes coming up with new software to add into the library. The GNOME Desktop and GNOME Mobile are not static projects–we have a six-month release cycle. We just released in October the first coordinated GNOME Mobile release that ties in with the standard six-month release cycle.

It was a base platform to get started, and now we’re considering new components to add to the platform. There’s a long list of incubator projects under consideration. Various companies involved in GNOME and GNOME Mobile have been working on parts of these libraries, developing them, using them, and adapting them for their devices.

Scott: We’ve asked a lot of stuff that’s interesting to us, but what do you feel is the more interesting stuff going on around GNOME Mobile right now?

Paul: We’re in an interesting period. A lot of what we’ve been doing recently is just getting our house in order, as it were, and tying the base platform into the standard release schedule. Now we can start to add to the platform and build out more capabilities.

It could be something like Clutter, which is a library for building interesting UI interfaces and user experiences based on OpenGL and GLES chipsets. Or it could be something like GuPNP for integrating uPNP services in devices.

Now, the pieces that we can make to build new devices become much more interesting.

Scott: Thanks, Paul. That looks like a pretty good place to wrap up.

Bookmark this:

Interviewee: Stormy Peters

In this interview we talk with Stormy. In specific, we talk about:

• History and scope of the GNOME umbrella project
• The relationship between GNOME and the public
• Branding an open source project in a world of mixed solutions
• Competition and collaboration between open source projects and other software
• Enhancing communication between developers and non-technical users
• Individual versus collective contribution to product development
• Client-side Linux and the rise of mobile devices
• Emerging relationships between device manufacturers, carriers, and users

Sean Campbell: Stormy, to get us started, can you give a little bit of your background?

Stormy Peters: Sure. I got into open source software when I was managing the desktop team at HP and I had a team of engineers in India working on the CDE desktop. All we had time to do was fix the defects that our customers were calling in.

This was about 1999, and Linux was becoming really popular. I realized that the Linux desktop did everything our customers needed, as well as other features like OpenOffice, and I thought, “Well, why don’t we just use that one?”

It turned out to be a very easy technical process to port the GNOME desktop to HP-UX. The main issues we had to contend with were non-technical. When we proposed our plans to management, their main concern, initially, was that we were going to accidently copyleft HP-UX.

Pretty soon, I’d worked myself into a new job creating the open source program office for HP with the Open Source Review Board, and policy and strategy. Then I ended up helping HP customers and partners, and I eventually left to go to OpenLogic.

OpenLogic is a software startup that helps Fortune Global 5000 companies use open source software. I helped set up a plan with the community to support the projects those companies were using, as well as to help OpenLogic’s customers set up open source software policies of their own.

Then, back in April, I was at the Linux Collaboration Summit in Austin, and a couple of the GNOME guys approached me to work with them. I thought about it, and I realized that it would be really fun–a great mission and passionate people to work with.

I really like working closer with the community, and the GNOME folks are great people. One thing led to another, and now I’m Executive Director of the GNOME Foundation.

Scott Swigart: I think a lot of our readers will be pretty familiar with GNOME, but in your own words, could you describe the project and tell us a bit more about your specific involvement in it?

Stormy: The GNOME project is very large; you can think of it as an umbrella project with a couple of hundred modules in it. If you use Linux, you probably use GNOME, which is the desktop for Linux with all the windows and the menus and look and feel that you see when you log into Linux. KDE is the other Linux desktop environment, but in most of the Linux distributions, GNOME is the default.

GNOME includes a lot more than just on the desktop environment for Linux. It’s also a development environment that developers can use to build applications that run on Linux, OpenSolaris, or other Unix platforms. It’s also got all sorts of things from web browsers and media players to the ability to play TV or do instant messaging. A huge array of projects fall under the umbrella of GNOME.

Scott: It strikes me that people tend to think of GNOME as just the shell, but it is really a set of other stand-alone, independent applications, as well. Talk a little bit about that relationship between GNOME the shell and all those other projects, as well as what it means for something to be under that GNOME umbrella.

Stormy: GNOME started out as the desktop. That was its primary focus, but the GNOME developers started to find needs like being able to read their mail on Linux, so they wrote Evolution, which is a mail reader. Then they decided they needed an open source browser, so they wrote a browser, Epiphany.

So you see, GNOME’s growth beyond being just a desktop is because the community of GNOME developers continues to define new requirements and opportunities. It’s even a set of applications that run on cell phones and handheld devices called GNOME Mobile.

I asked the question, “What does it mean to be a GNOME project, and how does a project become one?” when I started, and to be honest, it’s still not terribly clear to me.

Scott: [laughs]

Stormy: I’m waiting to see the next project that becomes a GNOME project, to observe how that happens. There are more than 100 projects in there, and I think the core identity of what it means to be a GNOME project is that it participates in the community.

People who work on those projects work a lot with the other GNOME developers, and they follow GNOME values and interface guidelines. GNOME is really good at making sure that everything we develop is accessible, internationalized, and usable by everybody.

Scott: It seems like a fair analogy to say that in some ways GNOME is almost like a distro. How do you manage the necessity of making sure that 100 projects are all ready on the same day? Does GNOME just freeze it at some point and say that whatever projects are ready to be included when a new GNOME release comes out are included, and the others aren’t?

Stormy: You touched on something that GNOME has done quite well in the past, which has obviously been picked up by all the Linux distributions. The Linux distributions are now large companies with lots of customers, and they need regular releases.

For the past few years, GNOME has been releasing regularly on a six-month cycle, which is pretty unusual for an open source software project that’s run by volunteers. The release team decides which applications are ready to be included and what should be included in each release. They make that call, and then they write up release notes that they put out on the web.

Scott: I understand that you can run KDE apps on GNOME and vice-versa. How does that work, if, say, I code a chat client against the GNOME API and you want to use it on KDE? How does KDE support that?

Stormy: When you’re building the client, you get the GNOME libraries, you’ll use the GNOME look and feel, and you’ll use the GNOME tools. When you finish, your application will actually look like it’s in GNOME.

That said, someone running KDE could always download it. They would probably have to download the GNOME libraries, too, and they could run it on KDE without any problem. There’s quite a bit of that that goes on–people using KDE with GNOME apps and vice versa.

Scott: Talk a little bit about what’s in the GNOME core that application developers depend on. Is it libraries for windowing and that kind of basic stuff? What do they actually code into that’s in GNOME?

Stormy: If you were developing on GNOME, you would have access to all the libraries that control things on the desktop: everything from the desktop panel and the menus on the top to how things come up and shut down. It’s the whole user interface for Linux.

Scott: You mentioned a couple of characteristics that give GNOME its identity. One thing you mentioned was internationalization and accessibility, as well as having a six-month release schedule that’s very deterministic. Can you characterize a few other things that you feel really identify the culture of GNOME?

Stormy: The community has key values. It’s really funny, because there are a lot of GNOME stickers that are covered in hearts and slogans about “GNOME love”. It’s a very close-knit community that really focuses on making sure that what they develop is not only beautiful, but it works well, is easy to use, and can reach out to people everywhere.

There’s been a lot of work put into GNOME by individuals and companies to make sure it’s accessible to people that might not be able to see or hear, as well as to people that just might be getting older and can’t see as well.

There’s been a lot of work to make sure it’s internationalized. We’re actually having our first conference in Asia next weekend. I guess to characterize GNOME, I would say that it’s a very caring, very inclusive community that has a lot of fun, but really cares that their product is easy to use and effective for the world.

Sean: Within that framework, could you lay out for us what you see yourself doing with the GNOME project, and what opportunities and challenges lie ahead?

Stormy: My first opportunity and challenge is figuring out exactly where to spend my time. I think of my job as having five parts. One is to be the eyes and ears for GNOME.

Part of it is just to be the person that people can come to, not so much as a representative of the community, but as the interface for the community–a single point of contact as well as someone who attends conferences and does interviews like this one to promote awareness of GNOME.

Another is our sponsors. The GNOME Foundation is funded by donations from volunteers as well as large donations from our corporate sponsors. Part of my job is finding new ones and working closely with existing ones to make sure that their relationship with GNOME is a good one.

Along with that comes marketing. We have a marketing team, but part of my job is to figure out what we want to do with GNOME marketing and help set up the infrastructure so that volunteers can help work on that.

Lastly, it’s just making sure the day-to-day stuff happens. I officially report to the Board of Directors, which is made up of seven volunteers who are elected once a year. These guys are giving their time, and they do a lot of work.

They get frustrated that they don’t have more time in the day, and that once in a while something drops. One of the reasons they hired me is to help get all those things done that they don’t quite have time for, but that they really think are important.

Sean: How do you effectively market GNOME? What are you trying to achieve to give it more visibility? Is there a role for trying to take market share away from the competitor, which in this case I suppose would be KDE, in a sense?

Are you trying to bring more people to the project, or more sub-projects? What do you see as the end goal of marketing more effectively?

Stormy: That’s a really good question, because I think it’s easy to get caught up in how to do it, rather than why we’re doing it. Our goal definitely is not to take market share from KDE.

If we wanted to take market share away from anyone, it would probably be Windows. The community has a really strong value around open source software. They should have free and open source solutions.

Our end goal is to get more people using GNOME and to get more people contributing to GNOME. Probably more than a thousand people work on GNOME. We have 400 individuals who have been officially recognized as contributors and have applied for membership to the GNOME Foundation, but we still have way more than we can do.

I think our end goal for marketing is to have more people see the value of GNOME and use it, and to get more people to contribute.

Sean: In a certain subset of users, it seems that Ubuntu has tried to position themselves as the de facto choice for people who may be considering using Linux. At the same time, they have a tendency to fuzz the bounding box between their efforts and upstream projects. I’m not trying to advance the standard argument that “They don’t contribute upstream.” While I think that’s an interesting argument, I also have the sense that it’s not as legitimate as people sometimes make it out to be.

I am thinking more about the way they package and present their offering, which can tend to take things from other projects and almost leave the impression that it really was their work output. I’m not suggesting that it’s malicious, but I do think that some users get a mistaken impression about the boundaries of the project.

Do you feel that they’re doing a service to GNOME, since they’ve standardized on it? Is there a tension between whether it’s maybe a good short term plan for them to fuzz those distinctions, but maybe not a good one in the long term? Or do you think it’s just good overall?

Stormy: I think that whether it’s good or bad, how they are doing it is often controversial. That said, Canonical is a sponsor of the GNOME Foundation and they have members that are present in the community. I think they are trying to give credit to the community, and in fact, they have asked me for a quote for their next press release.

They have other projects within the distribution. That said, I think they’re definitely trying to present it as one thing to the end user, I imagine probably with the idea that it’s really confusing to end users to get 39 projects instead of one.

I think it’s a balancing act that they’re working out between building a composite offering–which by all means, open source projects want them to do–and giving credit to the community and participating upstream.

Sean: I agree. I think it’s a delicate balancing act. After all, Windows doesn’t ask you to think about which volume shadow copy version is embedded in it, or make it part of the decision making process to actually purchase the product.

Stormy: One of the issues that we have is that all of the distributions take GNOME and then they brand it. They brand their distribution with their logos and their brand. We’re totally OK with that, but in the process it removes all of the GNOME logos.

We have considered rebranding guidelines that are sort of analogous to the “Intel Inside” sticker. For example, we might say that the distros can replace all of the logos and all of the splash screens except for one or two specific ones, which we would ask that they leave in place to retain some of the GNOME identity.

Sean: That’s an interesting problem–how do you retain your marketing and your branding with an open source project? It’s an extension of the standard discussion of how an open source project should avoid getting cannibalized, so to speak, by somebody taking your efforts and moving ahead on them without you.

Your idea of the Intel Inside sticker makes excellent sense. It’s not in the way, but yet you always know it’s an Intel laptop because of it.

Stormy: Right, and maybe the top menu has the little GNOME foot logo.

Sean: Exactly. Do you think that the community will be open to those efforts over time to help grow strong projects, or do you feel that there’s always going to be a tension in that area?

Stormy: Well, I think we’ll work it out. I haven’t floated the idea of leaving the GNOME logos in certain places past the distributions officially, so I don’t know what their response will be yet. I think they would be open to it, as long as it didn’t detract from the overall user experience. Of course, we don’t want to detract from the overall user experience either.

Sean: Right. It’s not like you’re going to have a ten-second Flash animation come up saying “You’re using GNOME!” [laughs]

Notwithstanding the fact that I don’t want to try to position you as competitors, I think it’s obligatory that we ask you what you see as the particular strengths of GNOME over KDE.

Stormy: First, it’s important to affirm that we’re collaborating with KDE, rather than competing with them. For example, they have an annual conference called the Akademy, and we have an annual conference called GUADEC. We’re actually going to co-locate with them next year so our developers can meet each other and collaborate in areas that make sense so that we’re building on open source and not reinventing the wheel every time.

We do have different focuses and different looks and feels, and our communities have slightly different values. We both value free and open source software, but the focus is on different areas, which enables us to reach different people. We are certainly looking to collaborate more with them, not compete more with them.

Scott: What do you feel GNOME’s strengths are compared to Windows?

Stormy: That’s a good question. Our obvious weakness is that everyone knows how to use Windows, and when you’re used to something, it’s hard to switch.

That said, I think GNOME has a lot of strengths in the ability to customize your own look and feel. You have lots of skins and lots of different applications you can apply. A lot of developers play with things they think are really cool. When I use GNOME, it just feels more friendly, if that makes sense.

You also get updates much more frequently, which some would say is bad, but when there’s something wrong, somebody’s working to fix it.

Scott: Looking at it from the outside, it’s obviously a great benefit to people who are packaging distros that they’re able to customize it and use GNOME as this great foundation for a broader user experience.

Another thing that I hear a lot is that if a user has a problem or a question, they can get on a mailing list and potentially be talking to the person who wrote the code. You don’t typically have access to that type of scenario with closed-source proprietary software.

Stormy: It also comes with a lot more than Windows does in terms of applications, like music players, CD players, instant messaging, and financial calculation programs that save you cash. You don’t have to pay extra for Outlook, for example, because it comes with Evolution.

Scott: Talk a little bit about the interaction between users and the people working on the project. Many open source projects are “by developers, for developers,” or in the case of projects like Apache, many are by developers for very technical people.

With GNOME, on the other hand, you have a far less technical user base, especially as Linux-based devices like netbooks proliferate. It’s potentially hard for a really technical developer to put themselves in the mindset of a non-technical user.

What do you see as helping to raise the awareness on both sides and to facilitate communication between those highly technical folks and non-technical users?

Stormy: I think the first thing is that the GNOME community really wants to hear from users. In the annual conference we have called GUADEC (the GNOME Users and Developers Conference), we try to attract end users, although that’s always a challenge.

We’ve also been talking about doing usability studies. We’ve done some in the past, and we’re hoping that our corporate sponsors can help us figure out how to do more. We’ve also talked about doing case studies.

The City of Largo uses GNOME. Dave Richards, who’s an administrator, came to our user experience test this week and talked about why they like GNOME and what they have troubles with. He shared his experience of using GNOME in a work environment full of not necessarily technical people.

I’ve heard his talks quoted from several people already. People really paid attention to what he had to say, and the GNOME developers as a whole are very interested in talking to end users.

One obstacle is simply the tools that users versus developers use. When users have questions, they usually end up on forums. When developers look for feedback or ask questions, they go to mailing lists. It’s really a question of how best to find each other and to talk to each other.

Scott: Right–users aren’t necessarily versed in Bugzilla. Let me flip it around, then. GNOME obviously interacts directly with the users, but the other side of that is interacting with the distros. In a perfect world, how does the relationship between a project like GNOME and a distro work?

Obviously, they need to cater to their users, and they want a certain experience. They’ve got a certain culture, and they’re probably doing certain customizations, which might involve the stuff that you can just configure in GNOME, or maybe it even involves forking it in a small way.

How does that relationship work in a perfect world between the distros who are getting questions posted on their forums and an upstream project like GNOME, and what are some of the places where occasionally the wheels come off?

Stormy: GNOME’s actually a lot closer to the distributions than they are to our end users, because the distributions are the ones that take GNOME to the end users.

Almost all of the major distributions are sponsors of GNOME, and they sit on the GNOME Foundation Advisory Board, so we get their feedback at the foundation level.

Also, many of the distributions employ GNOME developers. A lot of the GNOME contributors get paychecks from the distributions and are involved in their plans at work.

There’s a strong relationship, but there is also room for improvement in the way things work. For example, a lot of times GNOME releases and then the distributions pick it up. They had some of their developers working on GNOME before it released, actually contributing stuff, but some distributions make changes downstream, and we would like to see more of that work happen upstream.

On the other hand, that’s often really just a function of their business model. If they have a customer that asks for something or bugs that need to be fixed, they fix it first and then eventually it works its way upstream.

Scott: That’s similar to what we heard from Shuttleworth over at Canonical, who basically said they were going to work harder at doing more stuff upstream.

Stormy: They just hired a team of designers, and I’m hopeful that those designers will work with our upstream developers so that the developers get design feedback before they actually write a lot of code.

Sean: Many discussions happen around good design and good usability, and a lot of people say that exceptional design typically comes from a flash of inspiration from an individual, like Steve Jobs saying, “I shall birth the iPhone.”

What do you say about this notion that inspiration is led by a benevolent dictator, given that your user interface is essentially driven by a community? How do you get a mass community to come up with those flashes of inspiration, given that a certain segment of people says the only time that ever happens is when one person leads the charge?

Stormy: I’m still learning how it all works. GNOME has human interface guidelines. We have a designer’s mailing list and an IRC chatroom where developers can go ask questions, and hopefully there are designers online who will give them feedback right away.

We also put a lot of focus on the design, like the user experience hackfest that’s going on this week. We’ve gotten together a whole group of GNOME developers in the same place to talk about the user experience, how we can make it better, what areas are issues, and what our vision should be for the next couple of years.

I think it happens not so much by one person working by themselves, but more from a lot of conversations happening between people.

Scott: What’s really interesting from your perspective that we haven’t asked about?

Stormy: Two things. First, I knew before that open source software projects happen by a lot of volunteers doing work, but I didn’t really realize how much work they do. I’m tremendously impressed by how much volunteers contribute to and run the project.

The volunteer Board of Directors keeps the whole organization running. They answer tens of emails a week about funding, strategy, and operations. GUADEC, our annual conference that has between 300 and 500 people every year, is completely run by volunteers.

An amazing amount of good work comes out of these volunteers, many of whom have other full time jobs. I had a sense of that before, but I am much more aware of it now, and I’m even more impressed.

My second addition is that we haven’t talked much about GNOME Mobile. Open source software has been in the mobile industry for a while now, but it’s really starting to pick up, which I think is pretty exciting.

I went to the Maemo Summit a couple of weekends ago in Berlin. There was Linux and open source GNOME running on Nokia tablets. Some of the projects that people showed were just phenomenal. There’s everything from medical stuff, to music stuff, to note taking with a pen in a way that you could search and scroll really easily. Everyone did their presentations from their tablets using all open source software on Maemo.

Everyone was waiting for Linux to be on the desktop, and now with netbooks–with the Eee PC, the Mini and those type of devices–Linux is an important presence there that will only grow. I think the next movement we’ll see is with cell phones and devices, and there’s a lot of interesting work to be done there in the open source community.

Scott: We interviewed a person who was on Mandriva recently who said something that really resonates with what you just said. He suggested that Linux is never going to take over the desktop the way people think of it, in terms of going to Dell and buying a desktop computer with Linux instead of Windows.

He suggested instead that the desktop isn’t really going to be what people have thought of it being. It’s going to be a lot more about this merger between netbooks, mobile devices, and that kind of stuff. We’re talking about different form factors and different use cases, aiming more for where people are going than where they’ve been for the last decade.

If you don’t mind, expand on that a little bit from your own perspective.

Stormy: GNOME is absolutely working on another front. We have the traditional desktop, whether desktop is an overused word or not. But we also have projects like the online desktop, which is trying to merge online applications in the cloud with your desktop. We have the GNOME desktop on Nokia tablets, cell phones, netbooks, and other devices.

GNOME technology is used on a device used in classrooms that can test things like temperature, altitude, or pressure. Instead of having a whole computer, they just have a little test device that gives them everything they need right there.

I think computing is definitely changing. It used to be that you carried your laptop with you everywhere, and now with a lot of the smartphones, I see businesspeople actually leaving their laptops behind completely. They’re working entirely from their tablet, their iPhone, or their cellphone.

Scott: It seems like people are working on stuff that’s going to be really interesting to general consumers a year or two from now. Those of us who are used to carrying a laptop and sitting at a desktop aren’t necessarily thinking of some of those scenarios that are enabled by a mobile phone and high speed network connectivity, whether it’s 3G, 4G, stuff up in the cloud, or whatever.

Walk us through a few stories of the world of the future that you know people are working on.

Stormy: I don’t have any “predicting the future” stories to walk you through, but I can give you my feel for where it will go. When I’m walking around, I get really frustrated when someone mentions something that I want to Google but I can’t.

Or, we’ll be at lunch, and I’ll be trying to tell you about someone I really think you need to meet with, but I can’t just pull it up on my phone. I could if I had my laptop, but I don’t.

I think a lot of people feel like, when they’re walking around not connected, half of them is missing. I think that in the future, it will be much easier to stay connected to all of your knowledge all the time.

Scott: A lot of devices that try to solve these problems have done so in really awful ways, and so people don’t really use them. For instance, a mobile phone manufacturer might want you to record a voice memo for yourself, but that isn’t really what you want.

The iPhone has made some really significant strides in solving those issues. It’s the first time in a long time I looked at technology and thought, “That’s just magic; I don’t know how that works.”

For example, there’s an app that samples a song playing on the radio and gives you the name of the song, the name of the artist, a link to download it from iTunes, and maybe a link to a YouTube video of the song in performance.

Stormy: Right, and I know I wanted that five or ten years ago. I’d be driving around in the car and I’d be like, “What song is that?”

Scott: The things that are the most interesting to me are where two completely independent technologies converge, and all of a sudden whole new things are possible that weren’t really possible before.

The iPhone’s multi-touch screen is relatively large, but still a relatively small form factor for the total device. You don’t feel like you have to carry it under your arm; you can just drop it in your pocket. When you combine that with high-speed connectivity, all of a sudden there’s a lot of stuff that becomes feasible.

What do you make of Android versus Limo, versus whatever some of the other distros are that are trying to scale down to handheld devices?

Stormy: From the GNOME perspective, most of them use a little bit of GNOME technology, and our focus is on working with all of them to provide the most open platform that we can.

We have a set of technologies that we call GNOME Mobile for the whole range of those different devices. I went to the Open Source and Mobile conference in Berlin a couple of weeks ago, and I was really struck by the range of open source knowledge in the people there, from very little knowledge by some people to very high levels in others.

I sat next to three people from a very large provider company. They didn’t know what GNOME was and they weren’t Linux users, but there they were at an open source conference. The carriers and operators are now beginning to get into open source because of movements like Symbian and Android.

Whereas before, they really didn’t have to worry about the software because the device manufacturers took care of it, open source and new initiatives are enlarging the ecosystem to include the carriers, the operators, the device manufacturers, and the software companies.

They’re all playing in the open source software space, and I think that that means there’s a lot of opportunity to enhance and evolve the mobile model.

Scott: So many people are focused on where we’ve been, but what you’re talking about is really where we’re going.

Where do you see the difference? Like you said, it used to be that only the handset manufacturer had to worry about the software, and now it’s the carrier. What are you hearing in terms of, “As a carrier we’re really interested in this” whereas, “As a handset manufacturer we’re really interested in this”?

Are they interested in the same things? Is there a lot of overlap? Or are they really looking at the issue from two completely different perspectives in terms of the software that they would be interested in?

Stormy: I haven’t talked to enough carriers to be sure, but my sense is that they’re not sure exactly what they’re interested in yet.

What I think is interesting is that they’ll both have to focus more on the end user. It used to be that the software came with your phone. You almost picked the brand of your phone based on whether or not you liked their software. I remember switching from one phone to another and being really upset that the menus were different.

Now it’s not just the menus, but there are all sorts of applications I can run on my phone. Those applications aren’t dependent on either the cell phone company or the operator. Now those applications are in a space where they both have to collaborate, and end users can participate as well.

I think that the vast majority of cell phone users will never care about that, but some set of users will care that they’ll have much more choice on what applications they put on their phone. The operators and the cell phone manufacturers will work more closely with end users, and I think we’ll see much better applications. The cell phone companies have to guess at what we want.

Scott: It seems likely that the device manufacturers are going to care mostly about having a basic OS and shell, and some basic apps on the device itself. On the other hand, carriers are going to try to think of a lot of value added services that are only available through them as a carrier.

Whether it’s speech activated search like Sprint has, or streaming television to the phone or whatever, you get it because you’re part of that carrier’s network. They’re pushed into writing end-user application software that ties into their network in a way that they never really had to, or necessarily had the capability to do before.

In that light, it is very empowering to the user if they can just go to the package manager on the phone and say “What can I get?” This is basically a Linux device, and there’s a bunch of stuff that they can just pull down, independent of what the phone manufacturer or the carrier says.

Stormy: Right, and that kind of empowerment of users is a key benefit of open source.

Scott: I really appreciate you taking some time to chat with us.

Stormy: It was interesting as always.

Bookmark this:

Interviewee: Jeroen van Meeuwen

In this interview we talk with Jeroen. In specific, we talk about:

• Getting involved in Fedora–identifying opportunities
• The essential identity of Fedora among it peers
• The relationship between Fedora and Red Hat Enterprise Linux
• The relationship between user adoption and Fedora’s development goals
• New technologies as the driving force behind Fedora’s development

Sean Campbell: To get us started, could you please tell us a bit about yourself and your relationship to Fedora?

Jeroen: Sure. I regard the Fedora project as one of the most innovative Linux distributions. I started using Linux on a workstation about ten years ago–just hacking around to learn how stuff works–and it wasn’t until 2005 or so that I started making actual contributions back.

And now, just a few years later, I am Fedora project ambassador, which means that I have a great position from which to get the message out and talk to people. I am the Vice President of Fedora EMEA, which is a non-profit organization to aid the Fedora project in getting resources, mostly from third-party corporate organizations.

I also develop some applications; one of the most visible ones is Revisor, which is a distribution composer. I have various positions within the Fedora project to coordinate people’s efforts, in order to get things done effectively.

The Fedora project is one of the most amazing challenges I could ever wish for. The community is so open that if you are new to the community and you want to get something done, you just stand up and shout what you are doing, and everyone else will accept that, and they’ll help you get involved and started in making contributions.

Like any free and open source software project, contributors can get lost in getting stuff done, getting involved, and innovating things. That makes for a huge challenge that I find very, very interesting, and that’s how I got involved–that was my big motivation.

Sean: In the last several months, everyone’s retirement account is kind of melting down, although if you take the long view, it should all work out eventually. In times like these where layoffs are a fact of life, do you see that as an opportunity for people to dedicate more of their energy toward contributing to an open source project?

Jeroen: Obviously, getting laid off is very stressful, but it could create that opportunity to contribute back, as well. Speaking for myself, I make a very good living being a senior system engineer, preferably Linux, but I do a little Microsoft and Cisco as well.

Right now, I get a couple of hours a week to invest in whatever I want to do with free and open source software. That might be to promote Fedora at an event, prepare a presentation, or do this interview, for example. I make a very decent living doing what I do, and part of that is the Fedora project.

Whatever your circumstances, I think it is a huge opportunity for everyone who has an interest to continue developing themselves and making sure that they excel in a very unique sense, in the public domain. At any rate, that approach works very well for me, and it has built me a nice CV.

Sean: What motivated you to pick the part of Fedora that you started working on? How does somebody from the outside identify a niche in such a large project?

It seems like cruising the mailing list, looking at the documentation, and downloading the software would get you familiar with the project as a whole, but it might take another step to find a place to apply yourself to development, even after you have decided that you want to get involved.

Some projects have an easier roadmap than others; CentOS says the way to get involved is essentially to show competence. That’s a useful, short answer, but it’s still not a roadmap. What led you to wake up one morning and get involved, and how did you decide on the first step?

Jeroen: For me, it all started with not agreeing with what was happening. I think it was in the Fedora Core 3 timeframe, and I took issue with the way I had to download Fedora. You got a large ISO, and if you downloaded it two or three months after the release, you had to download another 700 megs of updates.

I was looking for some kind of slip-streamed ISO image I could download, so all the updates would have been incorporated in it. I found a sort of sub-project of Fedora that was doing work in that area, Fedora Unity, and they were having problems doing their composes–slip streaming the updates and stuff like that. That was something I decided to sink my teeth in, and that’s were it started.

Scott Swigart: Every distro has a unique ethos and notion of what they feel like their mission is. What do you think makes Fedora Fedora?

Jeroen: We have not even yet released Fedora 10, and like six weeks before the release, we were already planning Fedora 11. We were already able to build Fedora 11 with all kinds of new packages that would have otherwise broken Fedora 10 in many, many ways. But, we created the opportunity for people to get those packages built anyway and commit them to the Fedora 11 branches.

That is kind of what defines Fedora–we always look forward and lead. We focus very strongly on future development, and having Fedora 11 packages being built before Fedora 10 is released is a perfect example.

Scott: Talk about the relationship between Fedora and the upstream projects. Are there certain projects that are most important to Fedora, or how does that work?

Jeroen: One of the big, essential differences between Fedora and other distributions is that we’d rather gain one contributor than a dozen users. In fact, if I could lose 1000 users right now and gain a contributor, I’d do it. It’s not up to me, but if it were, I’d do it.

And that one contributor is gently forced to put whatever he or she contributes upstream. That’s the only way it works–you cannot ship patches to alter whatever GNOME does on your desktop or my desktop without pushing that patch upstream. And if it isn’t accepted upstream, it will not land in Fedora.

We don’t have ways to track what patch is in Fedora and is not upstream, but since we have groups of maintainers doing a single package, every maintainer is checked by the others.

Our policy is to push everything upstream and fix stuff upstream, so Fedora does no work other than making upstream contributions, aside from building a few nice ISOs every six months and building and maintaining the infrastructure to test, develop and distribute to get the software and the releases to whoever is using Fedora.

Scott: That is different than Ubuntu and a lot of other distributions that are more focused on users and making the easiest user experience, which leads them to carrying their own patches and things like that.

Would you say that’s a fair assessment of what makes Fedora different? Not to call Ubuntu out specifically, but does that ring true that other distributions are sort of focused on the experience, whereas you are really focused on the code in the upstream projects?

Jeroen: It is. We aggregate the best of free and open source software and make it work in a general Linux distribution, whether that is Ubuntu or Red Hat or Fedora. We adopt it in an early development stage, if we think it is the next generation.

SELinux is a perfect example–we had it in Fedora Core 2. I have not spoken to a single person who had it enabled in Fedora Core 2, but we shipped it anyway. Since then, we’ve made it into an enterprise product. Everyone wants SELinux, and I think that is one of the most eye-catching technologies that we’ve adopted so far. Now it’s sort of mature, and it has the utilities and the GUI notifications and stuff like that.

To return to the original topic, our approach is very different from other distributions, which basically consume upstream, and when things go wrong, they may or may not send a patch back upstream.

Scott: Talk a little bit about the relationship between Fedora and Red Hat Enterprise Linux.

Jeroen: Red Hat Enterprise Linux could be regarded as a long-term supported, commercially available, hardened Fedora distribution. By hardened, I mean that it takes a lot of people a lot of months to certify a Red Hat Enterprise Linux product on various types of hardware and for various types of applications and to do the necessary amount of QA before it gets out to enterprise customers.

Fundamentally, RHEL is driven by making sure that the support lines at Red Hat don’t have to ring any more than necessary. As I said, Fedora, on the other hand, is very much focused on development, and we do what we think is right. We move fast, we apply fast paced changes, we release early, and we release often, with no regard to whatever is our downstream distribution.

To take RHEL as one example, we do not care at which point in a Fedora release, or in a number of Fedora releases, Red Hat decides to distill whatever packages they want to put in an enterprise Linux distribution, because that is their independent decision.

I think that Red Hat sees it the exact same way, but from the opposite perspective. They let us do what we do best and support us in doing so, so that future versions of RHEL will have those innovative technologies and a number of extra features when they ship.

They also invest a lot of time in issues that no volunteer outside of Red Hat would have picked up. A good example would be Cobbler, which is a PXE provisioning framework that Red Hat customers found lacking in Linux. Red Hat Emerging Technologies has picked that up, and Michael DeHaan has been doing a great job at it.

Right now, we have a brilliant open source software alternative for provisioning. It’s Free Software, and it’s in Fedora and the Fedora Project’s Extra Packages for Enterprise Linux. Maybe it will be in Red Hat Enterprise Linux 6, because it’s one of the most requested features–deployment utilities, but I don’t know.

Scott: You said that if something is not in the upstream project, it’s not in Fedora. Red Hat obviously has a different kind of customer base and set of requirements than Fedora.

Is it your impression that Red Hat carries some of its own patches to some of the projects, if it feels customers need them but it isn’t able to get them upstream in time for one of their enterprise releases?

Jeroen: There is almost no such thing as them not being able to get it upstream in time, because Fedora has already done that work for them. Fedora 10, for example, will just work, and all they need to do is harden it.

But then again, those patches go upstream and may run in Fedora Rawhide, which is where you can test patches and fixes while they are being reviewed by the upstream folks. Rawhide is incredibly fast paced and fast changing.

There are a number of people with their workstations running Rawhide, and they complain every time it doesn’t boot. I’ve had issues with the Alt-TAB key combination that made me eventually disable it. That’s when you notice how addicted you are to using Alt-TAB, but otherwise, I couldn’t get any work done. That stuff will happen.

When new users have questions about that “Rawhide” thing they hear about every so often, we often tell them it’s the latest and greatest and as such, it will break. The actual words we then use are, “Rawhide eats babies…for breakfast”.

Sean: I thought your remark was fascinating when you said that you would rather get one contributor than 1000 users. At the same time, for Linux to be “successful,” someone has to have a user base, whether it’s Fedora or somebody else.

Ubuntu supposedly has eight million users and Fedora has somewhere between nine and 13, depending on how people were counting it, based on some recent stuff I saw.

Does Linux have an inherent challenge in the sense of being a collection of trains running roughly at the same time on roughly parallel tracks, in terms of the sub projects? From a user perspective, they want to buy/acquire/download a thing. In commercial terms, you’d call that productizing it.

To you, is RHEL essentially that vector that productizes what you do and puts a nice logo sticker on it? Of course, oddly enough, RHEL has CentOS on the other side of the continuum that de-productizes RHEL, which is really funny, if you think about it.

Is your perspective that you’d rather make the contribution and let somebody else deal with the problem of trying to deal with the user base?

Jeroen: Are you referring to the extensive marketing that Ubuntu does, in terms of getting it on grandma’s laptop and stuff like that?

Sean: Right–Ubuntu almost fuzzes the distinction between sub-projects, from an end user’s perspective. Dell ads say, “Buy a Dell Mini with Ubuntu.” It doesn’t say “with this version of GNOME and this version of OpenOffice.”

It’s similar to the fact that Windows marketing doesn’t talk about what version of Volume Shadow Copy is installed. But still, the strength of Linux is all of those contributors being recognized and not neglecting to contribute upstream.

From your perspective, it seems that you don’t really feel you have that problem, because Red Hat handles that process for you.

Jeroen: In the area of exposure, marketing, and getting Linux on grandma’s laptop, Fedora is entirely different than most other distributions. For one thing, there’s a new release every six months, and Grandma doesn’t want to upgrade that often.

One way to look at it is that Fedora prefers to be an engineering release. We have very short release cycles and very short development cycles. There are weeks before freezes when you don’t sleep more than four hours a night in order to get your stuff done on time.

That approach is key to making the Fedora project what it is. We focus on development even if our most recent version isn’t yet released, like I described in the bit about Fedora 11 being developed six weeks before Fedora 10 is out the door.

Occasionally, discussions arise within Fedora where people say we need to create a long term support version of Fedora, but that would divert us from our development focus. If I support something that is more than a year old, one could say that I am wasting my time.

For me, long term support is in opposition to what Fedora is trying to accomplish, and it would lead us to lose the people that say development does it for them.

Sean: If you landed a Linux distribution on grandma’s laptop 18 or 24 months ago, WPA2 wouldn’t have worked, wireless would have been clunky, and certain new wireless chips from Intel wouldn’t have worked. There were workarounds and patches, but Grandma wouldn’t have been pleased.

You guys put all this stuff into a network manager that most of the distributions have picked up, and now even a webcam works right out of the box. That doesn’t really map to the fact that lots of people are saying that there’s really no great demand or market for a Linux desktop.

Then I look at what’s on the roadmap again, and I see that you are tackling multiple monitor support. I personally love the idea, as I struggle with my 24″ Samsung, but what insight can you give about the value of moving those rocks?

These issues don’t really seem related to RHEL on the server side, and you have also said that you’re not trying to target the desktop. If people were asked for a show of hands, they’d probably say well that’s what Ubuntu is trying to do. What’s the motivation behind addressing those issues?

Jeroen: Even though our main focus is not to expand our user base, that doesn’t mean we don’t do work that mostly benefits users, including stuff that may wind up on Grandma’s laptop. [laughs]

The motivation for work like that is mostly the technical challenge associated with it. I did another thing for remixes, which was a Core 7 feature, and for me the motivation was quite the same. I learned a lot while doing it, and I found it very rewarding to have other people talk about it and use it.

If you do anything within a certain distribution and it only ends up in your distribution, the reward is a lot smaller than when it spans over all Linux distributions, such as the work of Network Manager, which is awesome. You’re going to see PolicyKit, ConsoleKit, PackageKit, from a lot of people that are involved with Fedora and other distributions as well.

Those are all new technologies that would make desktop life easier. My original point was how we don’t really focus on expanding the Linux user base. We very much more like to do the work that enhances the computing experience, whether it’s for the desktop, the server, or both.

Scott: This conversation has really opened my eyes about how Fedora approaches things. To summarize that, it seems that the goal is to identify promising open source projects to be included in Fedora, which raises the visibility of those projects and encourages the people working on them to work even harder, because they want their work to be included in a release.

It also seems like the short release cycles help to fuel innovation, since people want it in the next release, and so they work very hard at it. Fedora seems really to be about driving hundreds of individual open source projects to be better and to integrate and work together.

Jeroen: To put it in one phrase, you could say that Fedora is a snapshot of the best of free and open source software, and what is next. The goal is not to provide the best in the sense of what’s most mature, but to provide a sneak preview of new technologies.

Scott: That means things that are on the right track can show up in Fedora quite a while before a lot of other distros.

Jeroen: True–one of those things was PulseAudio in Fedora 8. We adopted and shipped it, and we got a lot of users complaining about their sound not working, which resulted in a lot of bugs getting logged and fixed.

That helped PulseAudio get ready to go over the entire spectrum of most distributions, and that’s what we like to do.

Sean: We’re getting close to the end of our time. Is there something else you wanted to touch on, or any closing thoughts?

Jeroen: Red Hat has been a major contributor to OLPC software development, and that desktop environment is now available in Fedora as well. So, if you want to turn your laptop into an OLPC, you can sort of render your keyboard useless …

[laughter]

Jeroen: There are only icons. A child can work with it, even if the most technical, intelligent people will need to click everything and break everything before they know how to work with it.

I’m amazed by the interface, and it is one cool feature in Fedora 10 you want to try–the OLPC desktop. It’s called Sugar desktop in Fedora.

Scott: I’ve actually got one of those things that my five year old loves.

Jeroen: Have you tried to use it yourself?

Scott: Yeah, and it’s like you said–I don’t really get quite as much out of it as she does, but she’ll just explore and explore and explore the thing. It gives that sort of “it’s safe to just push something and see what it does” kind of experience.

Sean: Well, thanks for taking the time to have this call today.

Jeroen: Thank you.

Bookmark this:

Interviewee: Warren Woodford

In this interview we talk with Warren. In specific, we talk about:

• The origins of SimplyMEPIS
• Ubuntu’s role in the larger community
• Differences among distros from a developer perspective
• Corporate use of free versus for-fee Linux
• The Linux desktop and the future of client-side Linux
• Future directions of note: IPv6 and DNSSEC

Sean Campbell: Warren, could you introduce yourself and tell us about some of the things you have worked on during your career?

Warren Woodford: Sure. I’ve been pushing electrons for a very long time. I grew up with what is now the computer industry, and I was already working at almost the VP level when the first microcomputers came along.

My background includes telecommunications, entertainment, field service, mini computers, micro computers, mainframe computers, PCs before they were called PCs, real time processing systems, software for business, software for home, software for government, and tools that people have heard of if they’ve been around a long time–always on the bleeding edge.

That’s the way I worked until the Internet bubble burst, at which time I kind of withdrew and decided to take it easy while the economy was down, not realizing it was going to be so volatile for so long. It was in 2000 or 2001 that I first started looking at Linux.

Aside from the philosophy and technical foundations of Linux, there was a lot there that I really didn’t like, frankly. Because of my background, I had been a champion of GUI interfaces since the early ’80s, and that aspect in particular was very inadequate at the time.

The bottom line is that, when I first found Linux, it was too rough around the edges for me. That represented the possibility of opportunity, not that I was really looking for work. This will piss off a few people, but there was a certain amateur quality about it.

Around 2001 was the first time I used a version of Linux that felt pretty good, which was SUSE, but it also had some significant bugs. It was pretty mature, but it was stiff–just too rigid in the way it did certain things. Mandrake seemed like it was on the right track, but there were bugs in the installation process and things like that.

Still, I felt that there was promise, so I started using Mandrake around 2001, and as I got familiar with everything, I decided that it was marginally good enough. Then, in 2002, Mandrake stumbled badly with their release in the September/October timeframe. They made some big mistakes, in part because of pure hubris.

It was around that point that I started thinking about building a version of Linux, instead of depending on other people. I jumped into it, deciding to pursue it as a way to learn technology that I didn’t know.

It has always turned out that when I learn a new technology, opportunities arise, whether my original reason for getting involved worked out or not. That’s how I got into Linux and decided to develop MEPIS. It was first for myself, and then I decided to see what would happen if I gave it free reign.

It got picked up by Distrowatch and went to #10 in one month, and that told me something. I started spending almost all of my time on it, but then in 2004 I had an injury that laid me up for a long, long time. During that time, MEPIS made it to #1 at Distrowatch, but I couldn’t really do much to maintain it.

Then it slid. Mark Shuttleworth saw opportunity and forked Ubuntu off of Debian. To be clear, I think that has ultimately been good for Debian, and Ubuntu contributes a lot back to the Debian community, but it is clearly a fork.

Sean: What do you think about Ubuntu’s strategy? They contribute upstream–more as of late, in fact–but at the same they time fuzz the distinction for users that Linux is really a collection of sub projects that are traveling in the same direction at roughly the same velocity.

I actually think what he’s trying to do isn’t all that bad, but I may have the luxury of some detached pragmatism. I see it as a logical, commercially driven decision, but I’m curious what you think about it, because you’ve obviously got a lot more history in this than I do.

Warren: I’m not bothered by anything that Ubuntu does. I think that Ubuntu pushes the boundaries regarding purity, and I don’t think that’s a bad thing, although that gets me in trouble with some people.

Some people call me a whiner about the GPL, while from my point of view they are the whiners. The GPL deserves to be scrutinized closely and to be debated, as does any legal document that restricts people’s rights. Calling a person a whiner because they care enough to challenge, question, or state positions about something is itself whining.

I think it’s good that Ubuntu challenges the boundaries regarding what is and is not proper open source. I think that what Ubuntu contributes back, both upstream and cross-stream to Debian, is good. And I think that the way that they kicked Debian in the collective butt has been good for Debian.

The fact that Mark is out there trying to make commercial deals actually may or may not make a big difference in the long run, but it’s not necessarily a bad thing. For example, I know that a couple of years ago he negotiated with IBM to get Ubuntu approved as a platform for running DB2. That would make it very easy to get that DB2 approval extended to Debian if anybody wanted to, and I think that’s OK.

From the point of view of what’s right and wrong, I don’t think there’s anything at all wrong with their fuzzying things a bit, as long as they don’t do it as much as Xandros did. Xandros at one point was blatantly changing copyright, renaming things and such, to make it appear that they had invented KDE or something. I can’t speak for what was in their mind, but something was going on there.

I don’t see Ubuntu doing that. I see them creating projects of their own to build utilities that represent their philosophy about how such things should be done, like the Adept project for a package manager. However if there’s a good product out there already, then there’s no good reason for them to be reinventing the wheel.

I think KPackage has been kind of so-so. On the other hand, I think Synaptic is awfully darn good. But Synaptic is GTK based, and while that’s not necessarily a bad thing, I wouldn’t program in that world. For personal reasons, it would be too inefficient and take too long to do. I wouldn’t program in Python for the same reason.

I don’t see anything wrong with creating or sponsoring Adept, just as I don’t see anything wrong with the idea of starting Ubuntu in the first place. You can complain or you can say that competition’s a good thing.

Mark, for one, is probably not going to pursue something unless there really is a shortcoming to be addressed.

Sean: What about Ubuntu’s OEM strategy? They have done really solid work in getting Toshiba to push Ubuntu, for example, and of course they also have Dell with five systems. They seem to be doing a really bang up job in getting OEMs to ship, promote, and push the Linux based desktop, and that doesn’t even count the netbooks push that’s happening.

Do you think they may have figured out a strategy that could give them a certain position on desktops that others haven’t quite figured out yet?

Warren: An important consideration is that Mark is more or less a billionaire, and he made that money in the computer industry. He can talk to companies like Dell, IBM, Toshiba, and others with a level of credibility that no one else in the Linux industry that I’m aware of can match.

He can get in the door to propose things, and he can get things agreed to that nobody else can. That gives him an advantage when you consider one distro over another, but that’s just how things are.

That’s good for Linux as a whole, and it means that companies like Dell and Toshiba are starting to think about compatibility more than they were before. And that’s a good thing. There are people who have told me, “Hey, this is really great. I bought a Dell machine with Ubuntu and then put SimplyMEPIS on it and it all worked.”

Sean: You’ve got Intel producing video drivers and wireless drivers and you’ve got network managers, so this is where the Ubuntu fuzzing works both for and against you. On the one hand, it makes the user feel like the network manager is Ubuntu’s network manager, even though it isn’t, really. On the other hand, they can just go stick a different distro on it.

Warren: Yes. In that regard, they’re having a positive impact on Linux compatibility with mainstream hardware, by getting the mainstream hardware companies to think a little bit about compatibility.

Scott Swigart: How much difference is there, from an application developer’s perspective, between different Linux distros? And how much work is that to take into account, and how much does something like the Linux Standard Base help with that?

To put it another way, for an application developer, how much effort is it to support and test and ensure that you’re compatible with lots of different Linux distributions? And how much of that work is done by the app developer versus the distros themselves?

Warren: That is a very big question. About three years ago, some of the biggest companies in the computer industry brought together some Debian affiliated companies like mine regarding this very question. They wanted to explore having Debian-based Linux as a competitor for Red Hat and Novell, and it gave rise to the ill-fated Debian Common Core Consortium.

Those conversations arose from this very issue. You couldn’t have commercial applications or commercial support for a particular Linux distro without a known, stable base. That plays out at different levels, because it depends on what kind of application it is. If it’s something for server, then probably, you only care about a core set of packages.

You can have a core set of packages, and you can have standards around that. If you do, then at the very lowest foundational level, companies that are considering something commercial related to Linux have a common base that they can rely on. But what’s the real core if you’re running a practical application?

And if we’re not using straight X, then what toolkit are you using, and what version? Consider the case of Acrobat Reader for Linux. How are you going to release Acrobat Reader in a way that runs cross distro, when each distro and each release of each distro may have different versions of key libraries?

Adobe does it by basically bundling those libraries, so they only have to rely on the very minimum number of compatible packages, or libraries, on a particular release.

From the point of view of an application developer, the problem is that every distro and every release of every distro has variations in what versions of core packages are installed. And because each distro has a different philosophy about long term maintainability and about stability of the distro, it’s a moving target forever.

You know, it’s a miracle that Firefox works so wonderfully. Those guys are incredible, and so are the Open Office people. Figuring out how to write code that is compatible with so many different versions of libraries to run with or be compiled against is a huge job. This is where Linux has a really big disadvantage when it comes to building complicated applications that you want to distribute broadly.

Scott: Educate me on the Linux Standard Base. What you basically said is that it’s a moving target forever. How much does the Linux Standard Base do to alleviate that? It feels kind of like POSIX back in the day.

Warren: In my opinion, LSB doesn’t do a lot for that. LSB provides some value, like POSIX provides some value, but it doesn’t resolve all the issues.

New releases of MEPIS are still on top of Debian, which acts like a stable core or foundation on which MEPIS is built. If you take the latest version of OpenOffice and recompile it for Debian Lenny, OpenOffice 3.0 will compile in that environment. But OpenOffice 3.0 will not compile on top of Etch, because so many things have changed. One of the things that happens is that a lot of libraries change over time–names of libraries, APIs, and entire philosophies change.

Underneath it all, Linux represents a very active developer community in various areas with people trying out new ideas or making improvements. Without them being coordinated with or accountable to projects like OpenOffice, or Firefox or whomever, it’s a real challenge to build a stable distro or to build a complex application and have it be compatible.

Scott: There seems to be a lot of pressure as companies become more comfortable with Linux, to move from the paid distributions to the free ones, because they find that they’re not picking up the phone. They are not making a lot of support calls and things like that.

On the other hand, if we are at conference and we walk up to the Red Hat booth and ask them about that, their response is typically that the Linux market overall is growing, and it’s only going to be a small percentage of the total number of Linux users that go from free to fee. They are just happy to see the overall market grow.

Warren: In general, corporate America wants something that is supported officially. You can argue about why that is, and it may be a bit cynical to say so, but my impression is that people in companies can’t afford to take the chance of guaranteeing something themselves.

You are in a company and you are working toward retirement. You have a good job with benefits, and you do not want to say, “Well, let’s use this free version of Linux. I guarantee you it will be fine, and we will take care of any problems that come up.”

People don’t want to do that in the corporate world. They want say, “Well, Gartner says this thing over here is great and will work fine for our purposes. They have an annual support contract, and they’re an established company, so we can go with this.” The company can feel comfortable, and everybody up the food chain can be held blameless if something goes wrong.

I think that is the number one driving factor for free versions of Linux not being used in corporate America, except covertly or in very special circumstances where management is used to taking risks. Some industries are more risk-averse than others, of course.

The commercial versions of Linux also offer extras that the free versions don’t. If you look at Red Hat versus Fedora or Novell versus SUSE, they are offering extra things, like guaranteed update schedules, tiered support, and other kinds of extras. For example, their versions also contain extra utilities that facilitate and manage enterprise deployment of Linux.

Now, a small company can sure start out with Red Hat and switch to Fedora, and in a ten-person company with a good technical person in house, that could work out fine. For the most part, though, that won’t happen in larger companies except where risk taking is the norm.

Scott: Do you see that changing over the next five years or so? Do you see some of those larger, somewhat risk-averse companies realizing that there is money that they could be saving by not writing a big check every year, if they haven’t been having the problems they were worried about?

Warren: They absolutely will consider writing a smaller check to a different company, but I don’t see them going with something that is completely free and open source unless it is not critical to their operations.

I know of a recent scenario where a company had no approved product for doing a translation; if I remember correctly, it was a transform between PDF and TIFF. They couldn’t find a commercial product that exactly met their requirements and was approved by their enterprise architecture organization.

Since they couldn’t get enterprise architecture to suggest a product, they went with something open source. But that’s a minor usage of an open source product. I guarantee that same company won’t use Linux as a platform OS. They hang their hat on IBM big iron and AIX.

In the corporate world, it’s largely not a matter of writing a check or not–it’s more writing a big check versus a smaller one, where the person championing it to upper management isn’t going to get hung if something doesn’t work.

People will be bothered if there’s a big problem with some software, but if the company is big enough and proper due diligence was done, then nobody is going to be fired if it doesn’t work out.

Scott: What about the year of the Linux desktop?

Warren: It’s never going to happen. Sorry.

Scott: Why not?

Warren: There’s a chicken and egg problem with getting it on the desktops, where no matter how much Mark Shuttleworth does, Michael Dell is not going to tell Bill Gates where to go. No one is going to forget that Microsoft’s the big game in town, no matter how much Microsoft stumbles.

Mark Shuttleworth can spend his entire billion dollars on trying to make Ubuntu good enough to shoot down Microsoft on the desktop and that won’t change. It goes back to what I was saying earlier about the fragmentation in the Linux market.

You don’t have one set of products against which you can build commercial software, or do commercial deployment, or even long term enterprise deployment. It’s doable with Ubuntu, but it’s not a no brainer, although Novell and Red Hat are trying to address that part.

Right now, I don’t know of a single major corporation that would go with Linux on the desktop for one reason–no Visio. Until OpenOffice does a Visio clone, you can forget it.

Sean: That reminds me of a story. Wikipedia went to Ubuntu recently for all their servers, but they still have one Windows machine to run QuickBooks, which runs their financials.

There’s the problem, right? And it leads to the question, if you were king for a day, what is a reasonable goal for Linux on the desktop?

Is it netbooks, where because it can be thin and light, it’s kind of a doorway to the Internet? And then if you want to leverage all those other locally installed apps, you have to go over to the Windows thick client? That scenario is a bit like the way Apple is carving out a niche with a certain set of consumers.

Warren: Like I said, Linux desktops cannot win in big business as long as there is no Visio clone. They can’t win in small business because of the very thing you mentioned–QuickBooks. Small companies all seem to use it.

On the other hand, some things that are happening right now, like Nokia buying Trolltech and Google inventing Android, can shed light on where the opportunity lies.

In other words, appliances are a place where Linux, with a GUI, can land and really thrive in my opinion–not the desktop we’d normally think of.

Scott: We talked to someone from Mandriva, and his impression was that by focusing on the Linux desktop, people are missing the boat. He suggests that Linux is going to bypass the desktop altogether, because people are looking for the day when won’t need their laptop on the road, because they have a phone and similar devices that are sufficiently capable.

So Linux is not well suited for the desktop, as we know it, but it might be very well suited for what comes after the desktop. Even if the desktop always remains as a mainline use case, there are going to be other scenarios with handheld devices.

Linux shouldn’t really be aiming at where people have been, it should be aiming at where they’re going, and it might be better suited for that.

Warren: I think that’s exactly where it is going, and it is happening right now, specifically with things like Android and Nokia. Notice that in the handheld arena, you don’t probably want QuickBooks or Visio.

Because these application markets are just getting established, if you make sure that the environments support the major toolkits for developing applications for those environments, you’re not going to fall short.

To touch back on Linux on the conventional desktop, though, I use MEPIS for everything I do. I just spent a year on site in a corporate environment, and I used MEPIS everyday.

Almost everybody there was using Windows, but my Citrix connections were better, and so was my wireless connection within the corporate environment. Through Citrix, I was able to use Microsoft apps. An individual can absolutely use Linux for their desktop and have very little that they’re falling short on.

I also run all kinds of things in Virtual Box. Actually, for MEPIS, I do a recompile of the open source Virtual Box, because since Sun took over, they’ve been a little behind on recompiling the open source edition. If you’re using a 2.6.26 or a 2.6.27 kernel, you can’t run a guest on Virtual Box unless you have Virtual Box Version 2.0.2 or better.

The bottom line is that I can run Windows in Virtual Box, and I can run all kinds of test scenarios in Virtual Box. I can run from whichever 32 or 64 bit version of MEPIS I want. My main development machine is a MacPro, so I can even run OS X when I need to do that. It gives me one heck of an environment as a developer.

Scott: I want to be sensitive to the time, so is there anything you want to add in closing?

Warren: Earlier you asked about what I thought was not given as much attention as perhaps it might be. To follow up on that, there are reasons for DNSSEC and IPv6, for example, to be implemented and used.

The circles that I run in, I haven’t heard much talk about DNSSEC or IPv6, except very recently. I think that with IPv4 running out of IP addresses, IPv6 has to come along really soon. But the problem is infrastructure not OSes; I think most Linuxes can do IPv6 out of the box. MEPIS certainly does.

DNSSEC, though, has been with us for 16 years now as a concept, and it hasn’t been implemented. It should greatly improve security on the Internet regarding spoofing and things like that, and there’s work being done to actually implement it now. Still, though, I don’t hear much being said about it, and I don’t know to what degree people are getting ready for it or considering implementing it sooner rather than later.

The .gov domain is going to start implementing DNSSEC January 1st. There are trials that have been done, I believe, for .com and .org, but if you look to see what’s been done regarding integrating DNSSEC in user applications, there’s practically nothing.

It’s all at the experimental stage. To go to a website and be able to identify immediately that it does not have a valid DNS record would be a great thing. That’s something that I would hope that the Firefox project is going to put in the next release, but I don’t know that they are.

Sean: Those are great points. Thanks for taking the time to talk today.

Warren: Thank you.

Bookmark this:

Interviewee: Adam Williamson

In this interview we talk with Adam. In specific, we talk about:

• What’s new in the latest Mandriva release
• Relating directly to the users instead of the developer community
• Differences between user priorities and developer priorities
• Deciding what should be included in the distribution
• What will become of the “year of the Linux desktop” idea?
• The future of open versus closed development

Sean Campbell: Adam, please give us some background on you and on Mandriva.

Adam Williamson: Sure. I’m the community manager for Mandriva, which is my most important role with the project, but like everyone else here, I do about twenty other things as well.

I’m also the bug tester, I maintain several packages, and I do a lot of public relations community announcement stuff. Mandriva is one of the leading Linux distributions. It’s been around since 1998, so it’s one of the older ones. It primarily has a desktop focus, and we’ve got a strong international community.

Sean: Since you just released the most recent version, walk us through the most significant features you added to that release.

Adam: The biggest new feature is that we are using KDE 4 as the default desktop. We had a sort of testing version of it in previous releases, but it was more for playing around with than really using. For this version, KDE 4 is the default, so when users install, that’s what they will be using, out of the box.

That change brings a lot of great new features with it, like the plasmoids, the new file manager, Phonon, and all that good stuff. We’ve also overhauled our tools and our install. We’ve given them a completely new appearance, so they look nice and it also improves the ergonomics a bit.

Aside from that, there’s a whole laundry list of stuff. We’ve got GNOME 2.24, as well as LXDE, which is good for netbooks. We have really good netbook support in this release–we test things on the Eee PC, and we also have it working on the Aspire and the Wind. We’ve improved the boot speed quite a lot.

Something that I personally am involved in is support for working with mobile devices: Windows Mobile, BlackBerrys, and things like that, and I’m pretty sure we have the best support for that out of any distribution.

Sean: I remember from our previous conversation that you guys have invested a lot of time in mobile device support. Looking back historically, what particular challenges have you overcome in trying to integrate mobile device support?

Adam: That’s actually my stuff. It started with the 2008 Spring release. One of the nice things about Mandriva is that it’s a very flexible system to work with. About two weeks before 2008 Spring came out, my partner–who works for a cell phone store–brought home a Windows mobile phone. I plugged it in and couldn’t do anything with it. I couldn’t even look at what was on there, let alone start synchronizing it, so I decided to make it work.

There’s a framework called SynCE for Windows mobile devices that plugs into a thing called OpenSync, and then you need an OpenSync front end to actually synchronize anything. It’s a very complicated process, and at the time of 2008 Spring it was even messier because lots of this stuff was broken.

It involved getting OpenSync, SynCE, and a front end called Kitchen Sync all packaged up properly, and patched and working together. It was a matter of little bits of talking to various projects and getting patches and working out a couple of things that I had to fix myself.

I also had a BlackBerry lying around, so I got on a bit of a roll and I thought, well, I’ll get BlackBerrys working as well. It’s one of those little inspiration challenges, and the interesting thing is that it’s something that probably real people do more than distribution people. Your average distro guy probably doesn’t use something like a BlackBerry or a Windows Mobile phone; it’s not his work system.

But in my role of community manager, I know there are quite a lot of people out there who would find this quite useful. It’s kind of something that falls between the cracks for a lot of distribution developers, because they just don’t work that way.

Sean: Given that you’re the community manager, what are the features in Mandriva that you feel the development community rallies around–and you don’t really have to work hard to get them to throw their effort in–and what are the features that you have to sometimes say, “Hey, I need you guys to help out over here?”

Adam: In most companies, the community manager talks mostly with the development community, but I actually work mostly with the user community.

Our development is very collaborative between the community and the internal developers. It’s all done in Cooker, which is the open development distro; we don’t do anything behind closed doors. We don’t really have to drive anyone to work on any particular area, because we find that people tend to work on areas that are useful.

We don’t have a community manager in the sense of someone who directs the development community and says please do this, please do that. We’ve been doing community development for a long time, practically since we started, and we’ve got a lot of long-term contributors who understand what needs to be done. It’s really not something that we’ve had a problem with.

Sean: Given that you’re more focused on the user community, what kinds of things has the user community asked for that you realize are going to take longer to implement than the community thinks? This issue seems to come up a lot, when users are calling for a specific feature, but since they’re not software developers, they may not really understand that what seems like a simple request might take much longer than they think.

Adam: I know exactly the situation you mean–they’re asking for the moon on a plate, but they don’t realize it. The classic example is someone coming into the forums after they try out Mandriva, and it’s the first Linux they’ve tried.

They say “well, why doesn’t my wireless card work?” and “why doesn’t this USB Bluetooth adapter work?” and “Linux is just rubbish! Why doesn’t all this stuff just work straight away?”

I find all that a bit funny, because I came in around 2001, when you’d have the exact same tenor of post on mailing lists or Usenet at the time. The difference is that back then, it would be, “Why doesn’t my network card work? Why doesn’t my graphics card work?” We’ve gone from broken graphics cards to broken Bluetooth in five years, which I think is pretty good, actually.

[laughter]

Adam: But the thing is, problems are getting smaller and smaller as we go along. So I sort of view this as an encouraging message of progress more than anything else. They complain about things that don’t work and as you say, it’s not their fault, because you have to have quite a bit of technical knowledge to understand why it is hard to implement certain things.

Scott Swigart: You mentioned that you worked on the mobile stuff because it was interesting to you, but you also suggested that isn’t necessarily the way a lot of the Mandriva developers work. They might not have experienced the same pain as some of the other users. What do you think are some of the typical places where the itch that a user has is different than the itch a distro developer has?

Adam: One straightforward example is anything you can do from a console. I work remotely, and I’m more graphical than most of the technical guys on the distribution because I’m sort of moonlighting–like, I work in gedit. So when I go to the Paris office, I see a developer’s desk and it’s ten xterm instances, and you can’t see anything graphical underneath that.

I have another good personal example, even though it’s been superseded a bit because Kate and GNOME can both do it. But back around two or three distributions ago, I noticed that there wasn’t any easy way to launch a graphical text editor as root.

You could just open a file and edit it. You know developers are not going to notice this, because they just do things from a console. It’s natural for them just to go “su vi” or something.

So I came up with just a little menu entry that used consolehelper to launch gedit as root. A lot of people found this really, really useful. I do various things writing documentation on the Wiki. I had to write down “open a console, type su, enter your root password, and then type kwrite or gedit filename,” and I thought that was a bit silly.

Why can’t I just click on an icon and open a root text editor? A developer wouldn’t even notice a little thing like that or consider it necessary, but to users, it’s quite a big deal.

Scott: I got my start on Unix systems, so I’m used to the ten xterm windows, and I remember kind of feeling like GUIs were garbage to some degree. The command line was fast, and you could do exactly what you wanted.

You could pipe stuff to grep, you could filter it, and it was just way faster than clicking a million text boxes to say what you wanted. But that’s fundamentally different from what users have been conditioned to expect with Macs and Windows and that kind of stuff.

Talk a bit about the evolution of developers coming to terms with understanding the idea that users just aren’t going to be interested in learning a lot of command lines to be productive and happy on Linux.

Adam: That’s an interesting area, and it’s something that Mandriva has been on top of for a long time. If you go back a ways to the Mandrake days around 2000 to 2003, we were the distribution that did this kind of thing.

Mandrake was the big one, because we had the Mandrake Control Center, the Mandriva Control Center, which is a big graphical thing that lets you configure a lot of the system. That has always been one of Mandriva’s strong points, but it’s definitely evolved over time.

It used to be that a lot of new Linux users who were not top-level technical gurus wanted to open up a console and learn how to pipe stuff to grep and so on.

Now, a lot more of the new users are people who genuinely don’t want to learn that stuff, and it’s possible to run a modern distribution–not just Mandriva but a bunch of others as well–without ever having to do that kind of stuff.

That is definitely a development within the last two or three years. I think it’s fair to give Ubuntu a bit of credit for that, because they put out a graphical product that you install and then feel like you can actually use it as a graphical modern everyday desktop.

Prior to 2005, a lot of people tried Linux for a day or two and quickly realized that they weren’t actually going to be able to use it without learning a whole bunch of technical console crap. They saw that, and they rejected it.

Scott: Right. And Shuttleworth has always been happy to don his flame retardant suit and make himself absolute flame bait. He’ll post something on his blog, and the Linux community will absolutely eviscerate him and say, “I don’t even understand whether this guy gets Linux.” But it gets people talking.

Adam: Yeah, I do think he’s good at that. I’ve never met Mark. I know Jono Bacon at Ubuntu, and he is a great guy. He says that Mark’s a great guy and I’m sure he is, but he does have a great naive act where he says, “Well why can’t we just click a button and do this?”

[laughter]

Everyone’s like, “Well that’s insane. There’s never been a button for that!”

Sean: There’s never been a button in there. There shouldn’t be a button there. And he goes, “Well everybody wants a button to be there.” And everybody eventually goes, “Yeah. OK.”

[laughter]

How would you contrast your business model and approach to packaging and distribution for users and for businesses versus Ubuntu?

It seems that some of what you do is very similar to Ubuntu, in terms of the way you position yourselves, the way the distro is built, and the audiences you target. At the same time, another portion seems more similar to Red Hat, in the sense of your enterprise focus.

Adam: Mandriva has a very wide presence; we’ve always been an “absolutely everything including the kitchen sink” type of distribution. I guess it’s kind of an historical thing, because as I say, we started off a long time ago.

The Linux world was much smaller then, and we were at the time a comparatively large company. It was pretty normal and feasible to take 30 guys and have a web server and a distribution that worked as a server, a desktop, a firewall, and just about anything else.

We are trying to make things a little more focused these days. For the 2007 release, which would be about two years ago, we dropped several versions of the distribution that were editions. We dropped the PowerPack Plus and Discovery commercial editions.

Now we only have one commercial edition, and we sort of simplified the rest of the range. The idea is to focus a little more on Mandriva Linux as mainly targeting a desktop user base and then to separate out corporate server and corporate desktop a little more as a separate corporate line.

The Red Hat comparison probably works best as sort of a mini Red Hat idea. We have the Corporate line and then we have the Mandriva Linux line, which is more an end user community based kind of thing, a little bit like Fedora.

Sean: This was the year of the Linux desktop. Do you feel that one of the things that inhibits that is the fact that every piece of the Linux desktop wants to “market itself” as independent? Users get kind of confused because they get the platform, and Vista hasn’t exactly rocked the world either.

People go in and buy a box that says Vista on it, and it doesn’t say “Volume Shadow Copy version 5.8,” which relieves them of the clutter of feeling like they have to understand whether 5.8 is better than 5.7.

I wonder sometimes whether it’s important for the Linux desktop to fuzz the boundaries of the distribution, sort of the way Ubuntu has done, in order to relieve the user base of that kind of adoption complexity.

Adam: This is definitely my personal view and not the view of Mandriva, but I don’t think there’s going to be a year of the Linux desktop. I think desktop computing is going to become mostly irrelevant before that ever happens. Everything is going to go to mobile devices.

There’s going to be some kind of convergence in the space between netbooks and cell phones and smart phones. We’re going to wind up with a science fiction vision of the device in your pocket that does everything. But that’s just my personal view.

In terms of your question, though, I agree that what you describe is a drawback in a sense for the Linux desktop. I kind of gave up focusing on how to win all the users in the world a couple of years ago.

[laughter]

Adam: Instead, I focus on how to keep making the product better, and how to serve the users we have. I think it’s legitimate to view Linux as something for maybe 20 percent of the most technically inclined users who like to tweak and are curious.

That leads one to the view that maybe what we have isn’t perfect for everyone in the world, but it’s a good product in its own right. It does what our users want, and we’re just going to keep refining it and focusing it and improving it and growing that group of people gradually.

Everyone knows that Apple is never going to have more than 10 percent of the desktop market or something, but no one seems to think that’s a problem. That’s what Apple is supposed to be.

Scott: We could probably talk about Apple all day, especially in terms of the Apple enthusiasts that complain about Microsoft not being transparent.

Adam: Right–Apple is 10 times worse. Apple is the black box, and you have no idea what the hell is going on inside. What comes out is usually pretty good, but if you want transparent development, Apple is not the place to go.

Scott: We had a conversation with somebody a while back about whether proprietary software will continue to exist, or whether it will all go open source. Their point of view was that there’s always going to be room for a great product, regardless of how it’s developed. It sounds to me like the goal with Mandriva is really to be a great product, rather than to espouse open source.

Adam: Actually, we do care a lot about open source, and it’s important to stress that. Otherwise we get a lot of hate mail.

[laughter]

Adam: We’ve always had a free distribution. Since 1998, we’ve been releasing Free with every release. Everything we develop is always free software. We do believe in a lot of the arguments that free software development is inherently a better way to do things. If you go too far the other way, though, you end up with GNewSense, which the FSF loves, but no one uses.

Most people want a great product. If you develop a great product then people are going to use it, and we definitely focus on making Mandriva a great product. We love open source, and we try to push open source development. We got things like RadeonHD and Nouveau in the distribution, trying to push for open source components in place of proprietary ones.

Scott: Open source is a collaborative way of developing. You’re less likely to produce features that nobody’s going to use, because if there really were such things, nobody would write them. Users know that they’re unrestricted in how they use the end product. In other words, basically, my personal view is that the ethos of open source delivers value to the end user, even if they don’t buy into the philosophy of open source.

Adam: I have an interesting take on it because I work in this little nexus. As I said, I interact with users directly every day. I maintain stuff for the distribution, but I don’t code. I’ve never been an engineer, even if I’ve learned how to fix tiny little things just by trial and error and using Google.

But I can actually see everyday occasions where open source helps me and helps the users, even though neither of us have a clue how to write a garbage allocator. There are still instances where it’s important and really useful to me to be able to grab the code or do other things that are directly possible because of the open source nature of the environment.

You can talk to authors, you can download a project, and you can look at the documentation. You can email the author, and he’ll probably get back to you within a day. You’re not going to get anywhere trying to do that at Microsoft–you can try to email the author of Windows Media Player, but it’s just not going to happen.

I see positive outcomes from things like that literally every day in my work, where open source helps people directly.

Scott: I think we’ve come to the end of our time. Thanks for talking with us today.

Adam: Thank you–it’s been a pleasure.

Bookmark this:

Interviewee: Dries Buytaert

In this interview we talk with Dries. In specific, we talk about:

• Building specialized commercial support for open source technology
• Making cloud computing more viable for widespread use
• The relationship between Drupal and Linux distros
• Lowering the barriers to the deployment of Drupal
• Integrating multiple functionalities and components into one solution

Scott Swigart: Dries, could you lay out some of your background for us, including your experience with Drupal and the new work you’ve been doing with Acquia?

Dries Buytaert: Sure. I’m based in Belgium, and I am the founder and project lead of the Drupal project.

Drupal is an open source content management system that combines traditional features like workflow and versioning with social media features, like blogging, forums, RSS feeds, tagging, and wikis. That combination, I think, makes Drupal a unique platform.

We estimate that there are more than a quarter million Drupal sites out there today. Some of these are operated by large organizations like Sony, Forbes, Warner Brothers, and Amnesty International, but many are also operated by individuals.

One of the goals of the Drupal project is to make it very easy for anyone to express themselves online, to get conversations going, and to connect with people.

I started Drupal in 2001, so we have been around for about seven years, during which time our growth has been rather organic. Today, we have more than two and a half million downloads of Drupal Core, which is the main version of Drupal. We have a couple thousand people contributing code, and many more people contributing with support, documentation, and things like that.

More than 700 people contributed to Drupal 6, which is our current main version of the core, and in addition, we have two to three thousand modules that have been contributed by people in the community. It’s a large community of active developers.

I’ve always done Drupal as a hobby, but around a year ago, I co-founded Acquia, which is a venture-backed startup whose purpose is to provide commercial-grade support to Drupal users. The company is based in Boston, U.S., and it consists of about 25 people now.

We’re building a couple of things; the first is a Drupal distribution that we call Acquia Drupal, which consists of Drupal Core and a selection of some of the best modules from Drupal. The second thing we’re building is called the Acquia network, which is a set of electronically delivered services that help you with the operation and maintenance of your site, as well as access to our technical support team.

Scott: I’ve heard the direction Acquia is heading compared to what Red Hat did with Linux, in terms of providing commercial support and enterprise capability for Drupal. Is that an accurate representation?

Dries: Yes, it is true that we are trying to be for the Drupal community what Red Hat was for the Linux community in the sense that both companies offer a suite of services to software projects developed by organic open source communities. Another class of commercial open source vendors could be described as non-organic. They are less focused on the community development model, but instead use open source software licensing as a component of their business strategy–to upsell proprietary versions or SaaS offerings, for example.

Scott: What do you think are the strengths and weaknesses of each approach? What is the significance of the distinction between arising out of a long running open source effort the way Ubuntu did with Debian, or Red Hat with Linux, as opposed to just having open source as part of a strategy?

Dries: To make sure the terms are clear, when I say non-organic open source, I am referring to a case where the majority of the software is built by employees of a company, rather than developed externally by community members. This isn’t to say that these vendors don’t have communities–they just don’t depend on the community for their primary software development efforts.

As a result, I think organic communities generally tend to be bigger, largely because of the tendency to remove some of the control around the project. For example, Acquia is not in charge of Drupal–the Drupal community is. I think because of that, the project might have the opportunity to attract more people.

I think the big advantage of an organic open source community like the Drupal community is that it’s very active and extremely passionate. As I mentioned, we have literally thousands of people contributing code changes to the project.

Scott: So that contrasts with something like MySQL, which was very non-organic, with almost all of the code written by MySQL employees. The company basically controlled the direction of it, and while it developed a very good community of users, that community did not control how the code and features should evolve.

The big switch they made when version five came out to add a lot of the "enterprise" features is interesting, and now, there’s a fork into Drizzle. It still remains to be seen how it takes off, but it looks to some degree as if Drizzle represents the community reclaiming MySQL, although obviously, that fork is started by what are now Sun employees.

Still, there’s the notion of opening it up so people can really contribute code and invest in the future direction of the database.

Of course, both models can be successful–both MySQL and Drupal are doing quite well. In terms of having a community that’s really invested in the software and feels like they have control over the direction, though, it seems like it’s a little harder for a project to go off in the wrong direction if thousands of people with different viewpoints are actively contributing to it.

Dries: I agree. As I said, there are thousands of modules in the Drupal community, and each of these contributed modules is like a small open source project on its own. Some are maintained by a single person, while others are maintained by team of maybe five people or more, but each of these projects has its own road map and release cycles. In other words, there are a lot of moving pieces in the Drupal community, and they are all moving at their own pace and in their own direction.

Part of the value that Acquia provides is very similar to Red Hat–we help manage that complexity for our customers. They no longer have to keep track of all the moving parts; we track and improve the stability, security, and scalability of their modules, we do integration testing to make sure they work well with other modules, we provide configuration support, module selection advice, and so on. In other words, we make sure that our customers are set up for long-term success with Drupal.

In a non-organic open source community, that’s often different, because the software is controlled by a single company. They tend to use different models, such as having a community version and an enterprise version, for example, which may have differences in terms of features or stability.

Scott: It seems that what you guys put out is basically a distro where you’ve done the work to determine which version of which modules are ready for general inclusion, in much the same way as a Linux vendor decides what’s going to be packaged in with their distro. They make the decisions about what’s stable and ready to ship across all those thousands of modules.

Dries: Right.

Sean Campbell: Let me ask you a question in another area. What do you think are the lessons learned, and what do you think about Red Hat’s current direction? For example, bringing in Whitehurst as the CEO surprised a lot of people. Notwithstanding the fact that he’s obviously a believer in open source, airline executive to open source company head wasn’t necessarily what people expected.

More generally, what do you think of their recent evolution in different technology areas, such as the way they’re moving to the cloud and going after the small-to-medium business market?

Dries: Honestly, I’m not really following Red Hat that closely, but I do pick up their news statements and so forth, and one of the things they recently launched, and which I’m excited about is a cloud-based version of their offering.

That’s pretty interesting, in the sense that from Drupal’s point of view, we see a lot of people that want to move to the cloud, but it’s still not trivial. You still have to do all the work like set up load balancers, MySQL servers, master/slave modes, and web servers, and you have to get PHP working.

As a next step, I would really like to see them do is to move up a level in the stack, meaning that they would help with actually setting up the MySQL servers, installing the load balancers, and doing all the rest of the hard work. I don’t know if that’s in RedHat’s road map, to be honest, but it’s probably more in Amazon’s roadmap than it is in Red Hat’s. The more vendors move to the cloud, the more likely it might be to happen, which is the exciting part for me.

Sean: We’ve talked to some other cloud providers who have expressed the same thing. The consensus seems to be that what Amazon provides is a fairly bare-metal environment that you can do whatever you want in, but a lot of people are going to want a lot more stuff preconfigured, supported, and backed up–kind of a ready-to-run environment.

I don’t know if it’s on Amazon’s roadmap, but there are certainly a lot of other providers looking at providing that layer of services on top of the cloud.

Dries: That’s true, and bear in mind that cloud computing doesn’t make it easier to scale a Drupal site; all it does is make it easier for people to do capacity. It’s easy to spin up a new instance and to distribute the loads, but they don’t actually remove all the configuration work that it takes to make it scale.

Scott: Right. You would want the cloud providers essentially to centralize the knowledge about how to do those things, so that every customer of the cloud provider doesn’t have to figure those best practices out for themselves. It certainly seems that there is a market opportunity there for people to provide more than just bare-metal services.

Dries: I agree.

Sean: To set aside cloud computing for a moment in favor of something a bit more mundane, can you characterize the relationship between Drupal and the Linux distros a bit?

Dries: The relationship is somewhat different from a simple desktop or server application. Upgrading a Drupal web site is more of a challenge than upgrading a simple application.The reason for that is that every site is unique. People want to have a unique brand for their site, so they have a custom template or theme, custom features, and so forth. Because every site typically has at least some amount of customization, something like apt-get doesn’t usually work well to upgrade Drupal sites.

Although Drupal is in these distributions, it’s not necessarily the best way to do upgrades, although it is ideal for installing the software from scratch. A lot of people just do an install and have every distribution worry about the dependencies and getting the right version of PHP, Apache, MySQL, or whatever database they’re using. Once the site is in production, though, it tends to get a little more tricky.

Or imagine having a Drupal site running on a four-server cluster–upgrades become more complicated.

Scott: Is work on the Linux standard base at a low enough level that it doesn’t really affect what you guys do, or does work on the Linux standard base in any way help ease compatibility problems between one distro and another?

Dries: We’re not affected by them too much, to be honest, because we run on top of the LAMP stack. The LAMP stack runs on pretty much every platform.

Scott: I know that Microsoft made an effort with Windows Server 2008 to have PHP applications run on Windows. Do you see any interest in running Drupal on Windows, or is it pretty much 100% Linux in terms of the install base?

Dries: There are a lot of people who evaluate and install Drupal on Windows machines, and Microsoft’s IIS web server has a pretty big market share. I don’t know the exact numbers, but it’s somewhere between 30 and 40% of all the web sites in the world that are run from Microsoft servers, so making Drupal work on them is certainly something we care about.

Sean: I’m curious about how you deal with user-facing requests in terms of the project’s evolution. To use another Red Hat example, they don’t see a consumer model in the desktop, mostly because they don’t see a way to charge money for it. On the other hand, Ubuntu seems to see opportunity there, at least in terms of leading them into other offerings.

By their very nature, portal products have two sides that are pretty well walled off from one another. To the IT guy who rolls it out, it’s just another way of displaying files and various elements of a company’s information architecture and data. To another set of users, it’s an entirely user-driven model, where they can make changes and do what they want.

What is your process for engaging with a more end-user-facing community, as opposed to projects that are geared to deal with work by developers, for developers?

Dries: I started working on Drupal as a way to explore web technologies back in 1999, before I released the first version, so the first set of users were pretty much techies and developers. For a long time, the Drupal community continued to be very developer focused; and, to some extent, it still is today.

In the last four or five years, though, we have been working to open the platform more to non-developer end users. We have started to put more and more time and effort into enhancing usability, for example, to support our vision to make it easy for people to express themselves.

There is an increasing number of people and organizations contributing to improve Drupal’s usability.

Scott: I want to spin back around to the cloud, since that’s such an intense area of interest at the moment. What do you think are some of the opportunities for Drupal in a cloud environment?

Dries: I think that, for Drupal to succeed, we have to lower the barriers for adoption. We have spoken about usability issues, as well as the support issues that are addressed by Acquia. In addition to those, scalability is very important; scaling Drupal is currently a non-trivial proposition, because of the complexities of scaling all the layers of the LAMP stack.

When I say "all the layers of the LAMP stack," I am including things like load balancing, MySQL, master/slave stuff, PHP opcode caching, as well as things like memcache and, potentially, integration with content-delivery networks.

On the other hand, because Drupal is written in PHP, it is very accessible to people, as compared to a Java-based solution, for example. People can get started with PHP on a $10 hosting account, and in some cases, those sites can become massively successful. When that happens, we often tend to see people who are not necessarily technical gurus suddenly having to operate and maintain very large sites.

Sean: To state that a different way, one of the challenges of building an easy-to-use framework is that it’s an easy-to-use framework. In other words, people can get themselves into the position fairly quickly of having a site that they will have a lot of trouble maintaining.

In contrast, to consider the commercial model, if you built the site on a Windows box running IIS and couldn’t scale it, you might tend to point the finger at Microsoft, rightly or wrongly. With open source solutions, though, it’s more viable to say, "Well, that really wasn’t our component, or our project. You need to go learn how to optimize Apache or MySQL."

At the same time, it might make it harder to get people past that barrier, because you can’t own all the pieces and the conversations around them. That seems to be a tremendous challenge; if a site is amazingly successful, even if it’s not Drupal that creates their scalability problem, that might be the way they see it, which gives you ownership of issues that you can’t necessarily control.

Dries: That’s a fair characterization, I think. Typically, as a site gets bigger, it goes through a number of phases. One of the first things that will arise is that the site will encounter bottlenecks with MySQL.

You could argue that it’s Drupal’s fault because it is doing too many queries [laughs], but ultimately, you need to scale your MySQL server, because Drupal does what it needs to do to run your site. Of course, scaling that MySQL server has nothing to do with Drupal.

While scaling MySQL is well understood–there have been books written about it–site builders don’t necessarily know how to do it upfront.

Scott: Do you think there is an opportunity, then, for someone to produce a Drupal virtual appliance for the cloud that already has a lot of that configuration done, which people could just push into a fairly bare-metal cloud like Amazon EC2?

Dries: I do think that’s a big opportunity, and some companies in the Drupal world are exploring that area.

Sean: A couple of months ago, Shadow Worth said he feels there should be more synchronization between various projects. He is trying, in some ways, to blur the distinction among the individual elements that make up his project, in an effort to make it more consumable by an end user community.

The effect is similar in some respects to the fact that when an organization buys SharePoint, they don’t necessarily think of themselves as buying Volume Shadow Copy and Windows–they buy the solution.

Do you guys feel you are trying to carve out a place in the market for yourselves around portal solutions in the same way that Ubuntu is trying to do it on the desktop and Red Hat did it on the server?

Dries: I wouldn’t say portal solutions, but solutions in general, yes. That’s our goal with our distribution, for example. Instead of having users assemble raw bits from different places, we do the assembling and we productize it as a solution for them.

We bundle it in one physical package, write documentation for it, and pre-configure it so it works right out of the box. We are certainly in the business of building solutions for people to simplify their effort to build web sites.

Scott: You’ve been involved in content management systems for a long time, and you mentioned that one of Drupal’s strengths is that it understands a lot of the social networking blogs and wikis, and that kind of stuff. What really interesting things do you see on the horizon for where content management systems and social networking are headed?

Dries: A lot of different stars aligning, if you will. A lot of scope and standards are being defined or getting traction — things like OpenID, RDFa, Microformats, OpenSocial, OAuth, PoCo, and more.

One of the things I recently blogged about is what it could mean to have semantic technologies in Drupal. For example, major search engines like Google and Yahoo are moving aggressively toward trying to capture more structured data, because that allows them to build better, more specialized search functionality, like search for products that can filter by price, availability, color, shipping costs, etc.

Today, if you build a product with Drupal, we could have definitions of products in our database, but they are being translated to HTML pages without any semantic markup, so a lot of the structure we have available is lost. I think we have an opportunity to admit structured data, which then in turn will enable a lot of new things to happen.

To support that, we basically have to add some semantic technologies into Drupal. Specifically, as we generate HTML, we should mark it up using RDFa, so that search engines like Yahoo! SearchMonkey can pick up that data and do more meaningful things with it.

Scott: We were at the Gartner conference last week, and there was a hilarious presentation that pointed out the ways in which search hasn’t really changed in 14 years. The presenter said it’s the equivalent of walking into a library where the librarian is behind a wall with a little hole in it.

You would walk up to the hole and yell "Penguins!"The librarian doesn’t know who you are, she doesn’t know if you’re a researcher or a five year old. And so, you look through the hole and she just shows books sequentially and says, "This one? This one? This one?"

[laughter]

Scott: That’s basically what search has been. And this presentation says that the direction search has to go to get better is federation and conversation, to give better context. Thus, the user’s request about penguins is shown in light of information clusters that associate the request with products, or services, or research, or children’s stories, or whatever.

Federation is also interesting, and it seems to apply to Drupal as well. It’s this notion that you can have content that’s in Drupal, content that’s in your mail servers, inside of an organization. People want to be able to search, and to search across all of that through a single interface.

So, talk about those two concepts a little bit and how you see them from the perspective of Drupal.

Dries: There used to be a lot of focus on functionality in the Web 2.0 world, a lot of new features, a lot of new ideas of doing things and moving things online to the web.

The next generation will be more about interoperability and data portability. It’s essentially concerned with being able to take your data in and out of your site, and supporting the next level of mashups enabled by semantic technologies and new protocols.

Instead of generating pages with very few semantic clues, I do think we have an opportunity to turn the Internet into what behaves essentially as a very large database that is very easy to access and parse using crawlers, search engines, and so forth. That’s a tremendous opportunity.

I hope Drupal can help make happen.

Sean: Thanks, Dries. I see that we’ve run out of time, and I think that’s a good place to close.

Dries: Thanks. It’s been a pleasure.

Bookmark this:

Interviewee: Cliff Schmidt

In this interview we talk with Cliff. In specific, we talk about:

• Using technology to build literacy worldwide
• Addressing power and infrastructure issues in developing regions
• Minimizing the cost of hardware and software for devices
• Encouraging community involvement through open licensing
• Crossing te gulf between project developers and the user base
• Deciding how best to advance the quality of life in distant places
• Fundraising challenges as the project gains momentum

Scott Swigart: If you wouldn’t mind, would you please take a minute to introduce yourself and your project, for the benefit of people who might not be completely familiar with it?

Cliff Schmidt: Sure. I’m the executive director of Literacy Bridge, which is a nonprofit 501(c)(3) charity that’s also a tech startup. We’re building a digital voice player and recorder that is targeted to cost between $5 and $10, depending on volume.

Its purpose is to allow people in the very poorest parts of the world to get access to knowledge, often about crucial things like how to prevent your child from dying of dehydration or how to improve the yield on your crops, or any kind of knowledge that people are after.

The project approaches this problem of illiteracy from two directions. First, part of the problem of getting knowledge is being able to read, so we focus on allowing people to improve their reading skills.

Second, the project also doesn’t wait for someone to learn to read in order to get knowledge, because it is sort of a substitute for the Internet. In places without electricity and without any sort of network infrastructure, this device allows you to get knowledge that’s important to you, spoken in your own language, delivered through device-to-device copying, among other distribution paths.

Scott: Dig into that just a little bit. In addition to the fact that it’s kind of a recorder and player, how does it help people build their literacy skills?

Cliff: It’s meant to be complementary to some sort of existing education program, such as a formal primary school program or an adult-oriented night education class.

What it does is very simple. It starts with anyone reading an existing piece of text, whether it’s a schoolbook, the alphabet written on a chalkboard, or some existing brochure that passes on information. In many cases, it will be a textbook aimed at improving literacy skills.

After someone has read that text, which could be anyone in the community who’s able to read, then the person who’s learning to read can sit down with this device and practice their reading. They can listen to the device read that same text back, jump around from page to page or line to line or word to word, and have vocabulary words defined for them.

It’s a similar pr