How Software is Built

Interviewers: Scott Swigart and Sean Campbell

Interviewee: Jim Gettys

In this interview we talk with Jim Gettys - VP of Software Engineering on the OLPC project. In specific, we talk about:

• OLPC base systems software
• OLPC user-facing software
• Platform innovation for the developing world
• Future directions for the OLPC platform
• Engaging kids with OLPC and open software

Sean Campbell: Hi, Jim. Could you introduce yourself and your role with OLPC?

Jim Gettys: Sure. I was one of the first people working on OLPC, and have been the vice president of software. We split the software tasks into two pieces–the base systems and the child-facing software. I deal primarily with the base system software.

I’m also one of the original authors of the X Window system, and I was the editor of the HTTP 1.1 specification for a long time.

Sean: Tell us a little bit about the software on OLPC machines.

Jim: We’re running Linux on the machines, which allows us to take a number of approaches that have been very difficult to do on other systems. Many of the children in many parts of the world–in fact, a majority of children in the world–go home to locations that have no electricity whatsoever, and many of them have schools with no electricity as well.

So, we’ve worked very hard not only on cost but also to make low-power systems, because not only is it hard to get the power, but power is very expensive. That makes it a fundamental design requirement, because just using everyday laptops would leave over half the world’s kids out in the cold. At a school Nicholas Negroponte equipped several years ago in rural Thailand with conventional laptops, fuel for the generator became the most expensive operating expense at that school. And, we’ve built the most rugged laptop in the world.

We focus on two goals in our user interface.

First is enabling collaboration. Most kids learn from other kids, and we also need to enable teachers to collaborate with the kids.

Secondly, a big issue is that conventional interfaces–and it doesn’t matter whether it’s Windows or Macintosh or Linux–are all really inappropriate for young kids, because it’s really hard to use a conventional desktop before you can read, and even more difficult if the child’s parents cannot read either, leaving the child in the dark with no help. How many times have your children come to you asking “What does this dialog box mean?”

Sean: Obviously, you also want a machine that can be distributed with a standard configuration, regardless of the locale or language, right?

Jim: The parents may never have seen a computer before, and in one of the trials, during early discussions about giving each of the kids their own laptop, one of the parents asked, “What is the Internet?”

So we can’t presume that the people know what the Internet is or that the parents can read or the kids can read. It’s a very different world out there than most people think it is.

Sean: It reminds me of well-intentioned efforts where, for instance, someone gives a remote village tractors, but they don’t have a fuel-distribution system. So, six months later, they’re all rusting in the fields.

Jim: We have to presume that there isn’t much, if any, infrastructure.

Sean: I understand that you started with Fedora Linux as your base OS. How did the target populations, areas for deployment, and other constraints that you have to deal with affect that decision?

Jim: Other than worrying about power management and suspended resume in a serious way, the main thing was that we needed a very small base system, because we need to fit everything onto the one gigabyte of flash on the machine.

While our friends at Microsoft are putting Windows onto the OLPC, they don’t expect to be able to actually run off of the base one gigabyte flash, so they have to add an SD card to have enough space, which increases the unit cost.

Other than configuring Fedora to be quite small, at the base system level, there wasn’t a huge amount of engineering work.

Scott Swigart: There seems to be some symmetry between your base set of software that’s going to run on the box and a prepackaged OEM image, but you take it another whole step. This software may never get updated–it may sit on that box exactly as is, and the users may not have the capability, for a period of time, to really even understand how they could update it.

Jim: We need the machines to be usable by the time the kids get them, by themselves. While we hope for and deployments plan to provide Internet access, there is no guarantee that this access will be initially available or reliable. But if Internet is available, we expect regular updates, both for security and for new applications (activities in our terminology).

Scott: Since you can’t expect them to buy their own word processor, for example, you must have had to make a lot of tough decisions about what went into that base software build to get the most value out of that limited memory.

Jim: We have a very large spectrum of software to choose from. Last time I looked, Debian had more than 20,000 software packages, and Fedora has, I think, about 10,000. And there are many options for any given piece of software, so the main thing is careful picking and choosing.

One good example is an office suite. Something like Microsoft Office or OpenOffice is completely inappropriate for a seven-year old. What a child needs is a simple, basic text editor that can at least support font changing and basic simple formatting, but doesn’t throw in the kitchen sink.

Rather than starting from scratch, we’re using a derivative of a program called AbiWord, which runs on both Linux and Windows. It’s much simpler than either OpenOffice or Microsoft Office, and in fact, my 13-year old prefers it to either of them, just because it’s much simpler and it’s all she wants or needs.

That’s what matters for kids; and most children (lucky enough to attend school) only get 5-6 years of basic primary education. We are not training these kids to be developed-world office workers who are doing high-powered PowerPoint presentations or OpenOffice Presents presentations for management. Learning to read and write successfully is what they’re in school to do.

We chose AbiWord because, above and beyond the fact that it’s a lot simpler and smaller than OpenOffice, it also supports real-time collaboration. So two or three kids can actually sit down and share a document in real-time, and they can all be editing on it, or the teacher can work with a student that way and actually be able to edit the document with the child, in real-time. That capability is really important to us, so the kids can learn from each other and from their teachers.

Our point of view is not that more complex things shouldn’t be eventually available to the kids. The set of people we aim at is really fundamentally different than where most laptops have been used.

Sean: You mentioned AbiWord’s collaborative capabilities and the fact that there’s some novel stuff in the system.

Jim: There are a couple of really novel things in this software, both at the base technology level and at the user-interface level, and one of the things that drives that is the view that kids and their teachers will be constantly collaborating.

From our perspective, it’s essential that two or three children can sit down underneath a tree and be able to work together on their computer, without having to go through some weird network configuration exercise. A group of seven-year-olds should be able to immediately run things like this collaborative version of AbiWord, very, very easily.

The concept of “presence” and being able to show you who’s around you in your network area has been brought really far forward in our user interface. We made it really easy for the kids to find out who’s out there doing what, and to very easily invite their friends to work with them. From a user-facing point of view, the whole concept of presence is brought all the way to the front.

Otherwise, while things like what has been done with AbiWord have been possible to do before, the configuration of collaboration has been so painful that it has typically not happened. Abiword by itself has had these collaboration features for a while, but they, as in other such programs, have gone nearly unused since they have been too difficult to configure.

We have succeeded at changing that equation, both to make it very easy from a user-interface standpoint for the kids and their teachers to be able to work together, and also to make it a lot easier for the application developer to actually build a collaborative application in the first place.

Our core applications, for example, now do that kind of sharing. The kids can work on a document together, play music together, and if you want to look at a program and edit it into something else, then the kids can share the text editor of the program, and so on and so forth.

Collaboration is fundamental to our view of what the software should be, because the kids learn from each other and from their teachers most of the time. This is not an isolated world where you should only work on the thing you’re supposed to be working on by yourself, right?

Sean: It seems to me that there are a number of novel things in the platform. There’s the power management that you’ve talked about, there’s the mesh network, there’s the display technology. At the systems level, some of the stuff in the system was pretty novel.

Jim: For example, at the power management level, just making it work fast is really important. I don’t know what your laptop is like, but my HP laptop takes over ten seconds to resume.

There’s no reason why our systems should take so long to resume. To give you an idea, when I measured a 200 megahertz StrongARM running Linux, it was able to go from coming out of reset until it was scheduling user-level processes in ten milliseconds. I measured that on an oscilloscope with one of the other people working here, to confirm that I didn’t screw up the measurement. The OLPC XO-1 hardware is believed good to about 50ms; right now, Linux’s USB stack is taking most of our current approximately 1 second resume time.

Scott: That’s pretty darn fast.

Jim: That’s the way systems can work. Why doesn’t your laptop resume quickly?

Sean: It seems to me that you guys did do a fair amount of groundbreaking R&D to build this, some of which will probably percolate out to the wider community.

Jim: That’s why our $188 machine takes about a second to wake up, and my HP laptop that cost $2,000 takes ten seconds.

There was actually quite a saga about getting our hardware to work properly for resume. I blogged about it quite a bit, and you can read my entries about how painful it was getting the hardware to work properly for suspend/resume.

Sean: Some other stuff too, like the mesh network, seems like it would have to have a lot of impact at the base system level.

Jim: Well, yes, and no. The issue is that, since most kids go home at night and don’t have electricity, we can’t presume that there are places where there’s either an access point to plug in or an Internet cable to that access point.

That just doesn’t exist in these places. Yet we’d like, as much as possible, for the kids to be able to work together, and preferably to be able to access stuff in the school as well. The point of the mesh networking is, in large part, to make it possible to extend the reach of the network to a larger area. The mesh does that, and does that quite effectively, although there are still some interesting problems at a technical level.

The main point is that, if you have a kid, and there are kids who live between him and the school, you’d like it to be possible for him to still get connectivity back to the school, or to other kids. That way, the set of people who can work together is much larger than just those who can hear each other directly.

Scott: Can you talk about memory use? The average programmer today is insanely sloppy. I have four gig of memory on my laptop; I have six gig on a quad, on the corner of my desk. Nobody cares any more.

Jim: Those who like software that actually works fast will find that the smaller they make it, the faster it runs. But yes, many programmers have gotten very, very sloppy, and they only make things run fast enough.

One of the nice things to have been watching is that, starting about a year and half ago, the Firefox people realized that they had a memory disaster on their hands, and they’ve been working very seriously on making Firefox’s memory footprint much smaller than it was; recent data shows Firefox 3 will likely beat all comers, which I expect will force everyone’s hands to do better. For example, when I look at my Firefox usage now, it never has a virtual address space of more than about 200 to 250 megabytes, whereas a year ago, it would have easily been 500 or a gigabyte.

Sean: One other thing I’m curious about–why Fedora over something else?

Jim: Red Hat put up a significant number of engineers to help us out, and the choice of which Linux distribution really doesn’t make that much difference.

Sean: What’s an example of something that didn’t make it into the final design, hardware-wise?

Jim: One of the things that wasn’t feasible at the time we started building this generation of hardware–in terms of both memory and power–was 3-D hardware, although we do have alpha blending in both hardware and software. We’d also have loved a touch screen, which was not feasible.

There are a few really wonderful 3-D education applications, and I certainly would have liked to have had that. We will as technology moves on.

Sean: What are some of the other things you feel like are on the road map? It sounds like there is probably a lot of innovation that you can continue to do at a software level on the existing hardware.

Jim: The software will never be done.

Sean: But the hardware is going to continue to evolve, too. On both fronts, what do you see coming?

Jim: Cheaper or lower power. Those are our first two things. Before capability, we prioritize cost and power, so the next generation has to run at significantly less power and cost significantly less. Anything we can get without impacting that, of course, we would say yes to.

The reasoning for both of those two primary goals is pretty obvious: to get machines to more of the kids in the world and to make power less of a problem for them.

Power drives cost in hidden ways that most people don’t think about. As soon as you get off the grid, the cost of power goes way, way up. As noted above, Nicholas’s school in Thailand shows how unforgiving this problem is, and it has been entirely neglected by the computer industry. Ruggedness is also key; our system will withstand much harsher treatment and environments than conventional systems can; we need to push yet further in these areas.

If you use solar panels, which we are using to charge our machines in many locations, the cost is very sensitive to the amount of power consumption, so lower power reduces costs all through the system.

We want to get it below $100, and the system-level power consumption needs to get down to well under a watt. Understand that with our display in gray-scale mode, the display by itself only takes a couple of milliwatts. I’d like to have a system where the system itself is not consuming very much either.

While significant power savings are beginning to show up by others following our lead using LED backlights industry wide, conventional flat-panels still take more power, due to the widespread use of LVDS drivers.

Sean: In your initial deployment, outside of the areas of power and cost, are your initial users encouraging you to go in a certain direction? What is some of the broad feedback that you’re getting?

Jim: It varies a lot. For instance, people who want to deploy it in urban areas in developed parts of the world respond to it as just another laptop, just maybe cheaper and more rugged.

It’s when you get into the more remote areas that you start hearing things like “oh, this thing can actually be repaired by a kid with a single screwdriver,” and “its power consumption is a quarter or tenth of the alternative.” Some aspects of the machine are valued very differently, depending on the environment you’re in.

Sean: I was thinking more of the developing world users, where the parents may never work with a computer. What’s coming out of that that’s helping to drive your thinking?

Jim: The really nice thing is hearing back from the kids themselves. A lot of that is anecdotal, but the easiest numbers to get quickly have to do with truancy and kids coming to school because they want to, for a change.

I think the more encouraging things are what impact the machines have on the kids and their families. In some of the trials, parents have gone from thinking of the laptops as a foreign and potentially bad thing to regarding them as a possible way for their kids to have a better life.

One of the really amazing things was a description I heard when I visited trials in Peru. One of the teachers said that they had a child who was not getting enough to eat and he had been very disruptive. Since having a computer, he was now engaged in the school and learning, and he had become quite an expert at using the computer. Rather than being disruptive, he was now actively helping teach other kids in the classroom. This kid has gone from being disruptive to being an asset in the school.

Sean: I remember having a Commodore 64 as a kid, and that feeling really seems to make sense to me. Some people seem to innately gravitate toward computers, and other people don’t.

Jim: Whatever their passion is, we want to get the kids engaged in learning. For some kids, it happens to be computing itself. In other kids, it may be music. In other kids, it may be reading literature. In other kids, it may be mathematics. The system should enable the child to go where their passions lead them.

Scott: Another thing I wanted to ask you about is, how do individual software packages get into the project?

Jim: There are a small set of things that you can consider to be really vital, core functions of a computer–a web browser, a simple text editor, and that sort of thing.

The other piece of the question is more subtle. One of the problems, from our perspective, that is now plaguing the entire industry is that, due to security concerns and insecure systems, the behavior of system administrators has been to lock down systems such that you can never exchange software.

That flies in the face of having a vibrant community of people–including kids–developing new activities to be able to share. The collaboration I talked about before includes collaboratively building new software together. That’s one of the areas where we want to be able to grab the kids’ passions.

We’ve worked quite a bit on trying to increase the level of security to make it safe for activities to be shared between people. We worry about security as much as we do because that’s the only way to really get an explosion of interesting new software and for the kids and their teachers to be able to build new things and share them among themselves and kids all over the world.

One of the delightful surprises that shows how persistent people can be is Ben Schwartz, a student at Harvard who, regrettably, I initially overlooked in getting machines to for developing software. He wrote the wonderful “acoustic tape measure” program for measuring distance with audio.

You take two XOs and point them at each other, and by timing the difference between radio and the acoustic signature that gets transmitted from the speaker, you can tell how far apart they are. This Harvard student did this on his own time. He wrote this activity and had it working before he had his own machine for testing.

Sean: That’s a great example of a programmer making the physical world do something. It’s a powerful feeling when you first use code to physically move something in the environment, whether it’s the CD tray or whatever.

Ideally, if you build a good platform, whether it’s a software platform or an API or whatever, it’s a platform for experimentation. People do stuff with it that you maybe never thought of. It seems like it’s easy for software.

Jim: The acoustic tape measure is a classic example of that. We didn’t think about that activity in advance. We also have a wonderful activity that exploits the XO-1’s unique audio input port (which enables DC input). Every child has a simple oscilloscope and data logger at their finger-tips, that can use extremely low cost sensors, such as a simple thermistor. Computing is now a basic skill in our culture. For interested children, it is very important that the computer itself be transparent. That our platform is an open-source platform allows a child to investigate all the way down to the first instructions that the machine executes. We want the kids to be able to learn computing at a deep, fundamental level. And that’s hard to do with commercial software, where the software hides how it works.

We have a special key on the keyboard that will show you the source to applications. That is now getting used in more and more of our applications. If you’re in the web browser, for example, it shows you what the HTML is underneath.

The show source key in most of our applications and activities shows you the Python or other source that defines that activity immediately, and it allows for editing and experimentation.

Or if you’re in Audio Editor that’s in the process of coming together right now, when you first hit that key, you’ll see the basic application. If you hit it again, then you can see things like what the MIDI sequence might be for the music you’re playing. If you hit it again, you can look at the Csound program that defines how to synthesize that sound.

So we’re able to show people what goes on behind the curtains.

Sean: That’s great, Jim. Thanks for taking the time to talk today.

Jim: Thank you.