Interviewee: Jim Gettys

In this interview we talk with Jim Gettys – VP of Software Engineering on the OLPC project. In specific, we talk about:

• OLPC base systems software
• OLPC user-facing software
• Platform innovation for the developing world
• Future directions for the OLPC platform
• Engaging kids with OLPC and open software

Sean Campbell: Hi, Jim. Could you introduce yourself and your role with OLPC?

Jim Gettys: Sure. I was one of the first people working on OLPC, and have been the vice president of software. We split the software tasks into two pieces–the base systems and the child-facing software. I deal primarily with the base system software.

I’m also one of the original authors of the X Window system, and I was the editor of the HTTP 1.1 specification for a long time.

Sean: Tell us a little bit about the software on OLPC machines.

Jim: We’re running Linux on the machines, which allows us to take a number of approaches that have been very difficult to do on other systems. Many of the children in many parts of the world–in fact, a majority of children in the world–go home to locations that have no electricity whatsoever, and many of them have schools with no electricity as well.

So, we’ve worked very hard not only on cost but also to make low-power systems, because not only is it hard to get the power, but power is very expensive. That makes it a fundamental design requirement, because just using everyday laptops would leave over half the world’s kids out in the cold. At a school Nicholas Negroponte equipped several years ago in rural Thailand with conventional laptops, fuel for the generator became the most expensive operating expense at that school. And, we’ve built the most rugged laptop in the world.

We focus on two goals in our user interface.

First is enabling collaboration. Most kids learn from other kids, and we also need to enable teachers to collaborate with the kids.

Secondly, a big issue is that conventional interfaces–and it doesn’t matter whether it’s Windows or Macintosh or Linux–are all really inappropriate for young kids, because it’s really hard to use a conventional desktop before you can read, and even more difficult if the child’s parents cannot read either, leaving the child in the dark with no help. How many times have your children come to you asking “What does this dialog box mean?”

Sean: Obviously, you also want a machine that can be distributed with a standard configuration, regardless of the locale or language, right?

Jim: The parents may never have seen a computer before, and in one of the trials, during early discussions about giving each of the kids their own laptop, one of the parents asked, “What is the Internet?”

So we can’t presume that the people know what the Internet is or that the parents can read or the kids can read. It’s a very different world out there than most people think it is.

Sean: It reminds me of well-intentioned efforts where, for instance, someone gives a remote village tractors, but they don’t have a fuel-distribution system. So, six months later, they’re all rusting in the fields.

Jim: We have to presume that there isn’t much, if any, infrastructure.

Sean: I understand that you started with Fedora Linux as your base OS. How did the target populations, areas for deployment, and other constraints that you have to deal with affect that decision?

Jim: Other than worrying about power management and suspended resume in a serious way, the main thing was that we needed a very small base system, because we need to fit everything onto the one gigabyte of flash on the machine.

While our friends at Microsoft are putting Windows onto the OLPC, they don’t expect to be able to actually run off of the base one gigabyte flash, so they have to add an SD card to have enough space, which increases the unit cost.

Other than configuring Fedora to be quite small, at the base system level, there wasn’t a huge amount of engineering work.

Scott Swigart: There seems to be some symmetry between your base set of software that’s going to run on the box and a prepackaged OEM image, but you take it another whole step. This software may never get updated–it may sit on that box exactly as is, and the users may not have the capability, for a period of time, to really even understand how they could update it.

Jim: We need the machines to be usable by the time the kids get them, by themselves. While we hope for and deployments plan to provide Internet access, there is no guarantee that this access will be initially available or reliable. But if Internet is available, we expect regular updates, both for security and for new applications (activities in our terminology).

Scott: Since you can’t expect them to buy their own word processor, for example, you must have had to make a lot of tough decisions about what went into that base software build to get the most value out of that limited memory.

Jim: We have a very large spectrum of software to choose from. Last time I looked, Debian had more than 20,000 software packages, and Fedora has, I think, about 10,000. And there are many options for any given piece of software, so the main thing is careful picking and choosing.

One good example is an office suite. Something like Microsoft Office or OpenOffice is completely inappropriate for a seven-year old. What a child needs is a simple, basic text editor that can at least support font changing and basic simple formatting, but doesn’t throw in the kitchen sink.

Rather than starting from scratch, we’re using a derivative of a program called AbiWord, which runs on both Linux and Windows. It’s much simpler than either OpenOffice or Microsoft Office, and in fact, my 13-year old prefers it to either of them, just because it’s much simpler and it’s all she wants or needs.

That’s what matters for kids; and most children (lucky enough to attend school) only get 5-6 years of basic primary education. We are not training these kids to be developed-world office workers who are doing high-powered PowerPoint presentations or OpenOffice Presents presentations for management. Learning to read and write successfully is what they’re in school to do.

We chose AbiWord because, above and beyond the fact that it’s a lot simpler and smaller than OpenOffice, it also supports real-time collaboration. So two or three kids can actually sit down and share a document in real-time, and they can all be editing on it, or the teacher can work with a student that way and actually be able to edit the document with the child, in real-time. That capability is really important to us, so the kids can learn from each other and from their teachers.

Our point of view is not that more complex things shouldn’t be eventually available to the kids. The set of people we aim at is really fundamentally different than where most laptops have been used.

Sean: You mentioned AbiWord’s collaborative capabilities and the fact that there’s some novel stuff in the system.

Jim: There are a couple of really novel things in this software, both at the base technology level and at the user-interface level, and one of the things that drives that is the view that kids and their teachers will be constantly collaborating.

From our perspective, it’s essential that two or three children can sit down underneath a tree and be able to work together on their computer, without having to go through some weird network configuration exercise. A group of seven-year-olds should be able to immediately run things like this collaborative version of AbiWord, very, very easily.

The concept of “presence” and being able to show you who’s around you in your network area has been brought really far forward in our user interface. We made it really easy for the kids to find out who’s out there doing what, and to very easily invite their friends to work with them. From a user-facing point of view, the whole concept of presence is brought all the way to the front.

Otherwise, while things like what has been done with AbiWord have been possible to do before, the configuration of collaboration has been so painful that it has typically not happened. Abiword by itself has had these collaboration features for a while, but they, as in other such programs, have gone nearly unused since they have been too difficult to configure.

We have succeeded at changing that equation, both to make it very easy from a user-interface standpoint for the kids and their teachers to be able to work together, and also to make it a lot easier for the application developer to actually build a collaborative application in the first place.

Our core applications, for example, now do that kind of sharing. The kids can work on a document together, play music together, and if you want to look at a program and edit it into something else, then the kids can share the text editor of the program, and so on and so forth.

Collaboration is fundamental to our view of what the software should be, because the kids learn from each other and from their teachers most of the time. This is not an isolated world where you should only work on the thing you’re supposed to be working on by yourself, right?

Sean: It seems to me that there are a number of novel things in the platform. There’s the power management that you’ve talked about, there’s the mesh network, there’s the display technology. At the systems level, some of the stuff in the system was pretty novel.

Jim: For example, at the power management level, just making it work fast is really important. I don’t know what your laptop is like, but my HP laptop takes over ten seconds to resume.

There’s no reason why our systems should take so long to resume. To give you an idea, when I measured a 200 megahertz StrongARM running Linux, it was able to go from coming out of reset until it was scheduling user-level processes in ten milliseconds. I measured that on an oscilloscope with one of the other people working here, to confirm that I didn’t screw up the measurement. The OLPC XO-1 hardware is believed good to about 50ms; right now, Linux’s USB stack is taking most of our current approximately 1 second resume time.

Scott: That’s pretty darn fast.

Jim: That’s the way systems can work. Why doesn’t your laptop resume quickly?

Sean: It seems to me that you guys did do a fair amount of groundbreaking R&D to build this, some of which will probably percolate out to the wider community.

Jim: That’s why our $188 machine takes about a second to wake up, and my HP laptop that cost $2,000 takes ten seconds.

There was actually quite a saga about getting our hardware to work properly for resume. I blogged about it quite a bit, and you can read my entries about how painful it was getting the hardware to work properly for suspend/resume.

Sean: Some other stuff too, like the mesh network, seems like it would have to have a lot of impact at the base system level.

Jim: Well, yes, and no. The issue is that, since most kids go home at night and don’t have electricity, we can’t presume that there are places where there’s either an access point to plug in or an Internet cable to that access point.

That just doesn’t exist in these places. Yet we’d like, as much as possible, for the kids to be able to work together, and preferably to be able to access stuff in the school as well. The point of the mesh networking is, in large part, to make it possible to extend the reach of the network to a larger area. The mesh does that, and does that quite effectively, although there are still some interesting problems at a technical level.

The main point is that, if you have a kid, and there are kids who live between him and the school, you’d like it to be possible for him to still get connectivity back to the school, or to other kids. That way, the set of people who can work together is much larger than just those who can hear each other directly.

Scott: Can you talk about memory use? The average programmer today is insanely sloppy. I have four gig of memory on my laptop; I have six gig on a quad, on the corner of my desk. Nobody cares any more.

Jim: Those who like software that actually works fast will find that the smaller they make it, the faster it runs. But yes, many programmers have gotten very, very sloppy, and they only make things run fast enough.

One of the nice things to have been watching is that, starting about a year and half ago, the Firefox people realized that they had a memory disaster on their hands, and they’ve been working very seriously on making Firefox’s memory footprint much smaller than it was; recent data shows Firefox 3 will likely beat all comers, which I expect will force everyone’s hands to do better. For example, when I look at my Firefox usage now, it never has a virtual address space of more than about 200 to 250 megabytes, whereas a year ago, it would have easily been 500 or a gigabyte.

Sean: One other thing I’m curious about–why Fedora over something else?

Jim: Red Hat put up a significant number of engineers to help us out, and the choice of which Linux distribution really doesn’t make that much difference.

Sean: What’s an example of something that didn’t make it into the final design, hardware-wise?

Jim: One of the things that wasn’t feasible at the time we started building this generation of hardware–in terms of both memory and power–was 3-D hardware, although we do have alpha blending in both hardware and software. We’d also have loved a touch screen, which was not feasible.

There are a few really wonderful 3-D education applications, and I certainly would have liked to have had that. We will as technology moves on.

Sean: What are some of the other things you feel like are on the road map? It sounds like there is probably a lot of innovation that you can continue to do at a software level on the existing hardware.

Jim: The software will never be done.

Sean: But the hardware is going to continue to evolve, too. On both fronts, what do you see coming?

Jim: Cheaper or lower power. Those are our first two things. Before capability, we prioritize cost and power, so the next generation has to run at significantly less power and cost significantly less. Anything we can get without impacting that, of course, we would say yes to.

The reasoning for both of those two primary goals is pretty obvious: to get machines to more of the kids in the world and to make power less of a problem for them.

Power drives cost in hidden ways that most people don’t think about. As soon as you get off the grid, the cost of power goes way, way up. As noted above, Nicholas’s school in Thailand shows how unforgiving this problem is, and it has been entirely neglected by the computer industry. Ruggedness is also key; our system will withstand much harsher treatment and environments than conventional systems can; we need to push yet further in these areas.

If you use solar panels, which we are using to charge our machines in many locations, the cost is very sensitive to the amount of power consumption, so lower power reduces costs all through the system.

We want to get it below $100, and the system-level power consumption needs to get down to well under a watt. Understand that with our display in gray-scale mode, the display by itself only takes a couple of milliwatts. I’d like to have a system where the system itself is not consuming very much either.

While significant power savings are beginning to show up by others following our lead using LED backlights industry wide, conventional flat-panels still take more power, due to the widespread use of LVDS drivers.

Sean: In your initial deployment, outside of the areas of power and cost, are your initial users encouraging you to go in a certain direction? What is some of the broad feedback that you’re getting?

Jim: It varies a lot. For instance, people who want to deploy it in urban areas in developed parts of the world respond to it as just another laptop, just maybe cheaper and more rugged.

It’s when you get into the more remote areas that you start hearing things like “oh, this thing can actually be repaired by a kid with a single screwdriver,” and “its power consumption is a quarter or tenth of the alternative.” Some aspects of the machine are valued very differently, depending on the environment you’re in.

Sean: I was thinking more of the developing world users, where the parents may never work with a computer. What’s coming out of that that’s helping to drive your thinking?

Jim: The really nice thing is hearing back from the kids themselves. A lot of that is anecdotal, but the easiest numbers to get quickly have to do with truancy and kids coming to school because they want to, for a change.

I think the more encouraging things are what impact the machines have on the kids and their families. In some of the trials, parents have gone from thinking of the laptops as a foreign and potentially bad thing to regarding them as a possible way for their kids to have a better life.

One of the really amazing things was a description I heard when I visited trials in Peru. One of the teachers said that they had a child who was not getting enough to eat and he had been very disruptive. Since having a computer, he was now engaged in the school and learning, and he had become quite an expert at using the computer. Rather than being disruptive, he was now actively helping teach other kids in the classroom. This kid has gone from being disruptive to being an asset in the school.

Sean: I remember having a Commodore 64 as a kid, and that feeling really seems to make sense to me. Some people seem to innately gravitate toward computers, and other people don’t.

Jim: Whatever their passion is, we want to get the kids engaged in learning. For some kids, it happens to be computing itself. In other kids, it may be music. In other kids, it may be reading literature. In other kids, it may be mathematics. The system should enable the child to go where their passions lead them.

Scott: Another thing I wanted to ask you about is, how do individual software packages get into the project?

Jim: There are a small set of things that you can consider to be really vital, core functions of a computer–a web browser, a simple text editor, and that sort of thing.

The other piece of the question is more subtle. One of the problems, from our perspective, that is now plaguing the entire industry is that, due to security concerns and insecure systems, the behavior of system administrators has been to lock down systems such that you can never exchange software.

That flies in the face of having a vibrant community of people–including kids–developing new activities to be able to share. The collaboration I talked about before includes collaboratively building new software together. That’s one of the areas where we want to be able to grab the kids’ passions.

We’ve worked quite a bit on trying to increase the level of security to make it safe for activities to be shared between people. We worry about security as much as we do because that’s the only way to really get an explosion of interesting new software and for the kids and their teachers to be able to build new things and share them among themselves and kids all over the world.

One of the delightful surprises that shows how persistent people can be is Ben Schwartz, a student at Harvard who, regrettably, I initially overlooked in getting machines to for developing software. He wrote the wonderful “acoustic tape measure” program for measuring distance with audio.

You take two XOs and point them at each other, and by timing the difference between radio and the acoustic signature that gets transmitted from the speaker, you can tell how far apart they are. This Harvard student did this on his own time. He wrote this activity and had it working before he had his own machine for testing.

Sean: That’s a great example of a programmer making the physical world do something. It’s a powerful feeling when you first use code to physically move something in the environment, whether it’s the CD tray or whatever.

Ideally, if you build a good platform, whether it’s a software platform or an API or whatever, it’s a platform for experimentation. People do stuff with it that you maybe never thought of. It seems like it’s easy for software.

Jim: The acoustic tape measure is a classic example of that. We didn’t think about that activity in advance. We also have a wonderful activity that exploits the XO-1’s unique audio input port (which enables DC input). Every child has a simple oscilloscope and data logger at their finger-tips, that can use extremely low cost sensors, such as a simple thermistor. Computing is now a basic skill in our culture. For interested children, it is very important that the computer itself be transparent. That our platform is an open-source platform allows a child to investigate all the way down to the first instructions that the machine executes. We want the kids to be able to learn computing at a deep, fundamental level. And that’s hard to do with commercial software, where the software hides how it works.

We have a special key on the keyboard that will show you the source to applications. That is now getting used in more and more of our applications. If you’re in the web browser, for example, it shows you what the HTML is underneath.

The show source key in most of our applications and activities shows you the Python or other source that defines that activity immediately, and it allows for editing and experimentation.

Or if you’re in Audio Editor that’s in the process of coming together right now, when you first hit that key, you’ll see the basic application. If you hit it again, then you can see things like what the MIDI sequence might be for the music you’re playing. If you hit it again, you can look at the Csound program that defines how to synthesize that sound.

So we’re able to show people what goes on behind the curtains.

Sean: That’s great, Jim. Thanks for taking the time to talk today.

Jim: Thank you.

Bookmark this:

Interviewee: Mitch Bradley

In this interview we talk with Mitch Bradley – who was responsible for developing the firmware for the OLPC project. In specific, we talk about:

• Mitch’s history with Sun, FirmWorks, and elsewhere
• Evolution of One Laptop Per Child
• Novel hardware in OLPC laptops
• Propagating technology from OLPC to the mainstream
• Open-source intersections with the commercial sector
• Open-source firmware and community
• History and future of firmware’s place on computers

Sean Campbell: Mitch, tell us a little bit about your background, both as it relates to One Laptop Per Child and outside of that.

Mitch Bradley: I spent a year-and-a-half out of ‘Uni’ working as an analog circuit designer at Rolm Corporation, which some people even remember. And then I moved to Sun when it was about six months old.

I was at Sun for 10 years; that was a pretty formative part of my career. I started out as a hardware designer doing things like SCSI and Ethernet interfaces.

I ended up being the guy that could debug the system problems when other people couldn’t as a result of my sort of refusing to specialize in either hardware or software and also as a result of this Forth diagnostics environment that I had put together over the years.

I had gotten interested in Forth at some point and started building little diagnostics. Then, about 1987, Andy Bechtolsheim–who was the hardware wizard at Sun–approached me to do some firmware for his new design, which was ultimately going to be the SPARCstation one. He needed the firmware to be smaller than the prevailing firmware size on the Sun product line.

I jumped at the chance because I had always wanted to rewrite the firmware in Forth.

And thus was born OpenBoot, which still ships on SPARC machines and was taken to the IEEE, became an IEEE standard, and renamed OpenFirmware at that time. Apple picked that up for their PowerPC Macintoshes, as did IBM for some of their PowerPC servers.

And I have been doing open firmware not quite ever since, but off and on ever since. I left Sun after 10 years there and started my own company, FirmWorks, in order to provide support services for OpenFirmware.

As an employee of Sun, I couldn’t really support firmware on third-party platforms at the level that they needed to be supported, so I started FirmWorks to do that.

And FirmWorks is still a going concern, although it has been on and off the back burner several times. I took a hiatus and built high-speed cameras to watch people hit golf balls.

Sean: That’s a nice, pleasant diversion.

Mitch: It was kind of like my midlife crisis. I had to do something different so I wouldn’t go crazy. I like to play golf so that was kind of fun.

Sean:What happened after that?

Mitch: That company cratered in the dot-com crash, and then I worked for a company that was doing voice control for cable television, talking to the control and telling it which channel you want to watch or tell it to go find the West Wing or tell it to go find some movie that you like, or show what sports were on and stuff like that.

We got that working pretty well, but we couldn’t get the cable companies to deploy it, because it wasn’t going to get them enough monthly incremental revenue to meet their threshold of interest.

I started out doing Open Firmware for that company. We were building a super computer because we were going to need a lot of compute cycles in the table TV head-in where rack space was at a premium and power was at a premium.

So, we had a MIPS- based 128 processor mesh machine in a small box that could do a lot of speech recognitions per second, but that eventually died on the vine because they couldn’t fabricate the chips properly.

I was doing the firmwork for that, and I did some DSP stuff and various odds and ends. Then I got involved with OLPC about a year and a half ago and have been doing firmware for them. So, that’s the short answer.

Sean: Tell me a little bit about your role with OLPC.

Mitch: I pretty much did the firmware. When I signed at OLPC, the idea was to use LinuxBIOS (which is now called coreboot). LinuxBIOS started out as a firmware where the idea was really not to have any more firmware than was absolutely necessary–the smallest amount of start-up that you could do and jump to a Linux kernel.

It kind of grew so that it acquired various payloads like GRUB and things like that, because in a real production environment it just doesn’t fly to have the firmware not be able to do anything other than jump to the OS.

There are lots of diagnostics and system maintenance and system upgrade issues that really need to be addressed, so you need something there other than just initialize the memory and jump.

The idea at OLPC was to use LinuxBIOS along with a stripped-down Linux kernel as the payload, and that would be the boot loader for the real Linux kernel, which would be loaded off the main larger flash.

We were running into problems with that, because even when we stripped the kernel down as much as we reasonably could and compressed it with industrial strength compressors, it still wouldn’t quite fit into flash with all the stuff we needed.

I thought of putting in Open Firmware, but the problem at the time was that there were some encumbrances on my Open Firmware implementation as a result of the fact that I did it when I was at Sun, and Sun has some copyrights on it.

But it turns out that shortly after I started it at OLPC, Sun open sourced their OpenBoot implementation as part of the Open SPARC initiative. So that removed the encumbrance on the portions of my code base that I didn’t own outright.

FirmWorks then open sourced their part of the deal and so we had a complete OpenFirmware implementation that met OLPC’s openness requirement.

As a result, I engaged in replacing Linux BIOS–or at least the Linux kernel payload portion of Linux BIOS–with OpenFirmware. I eventually replaced everything because I realized that what was left of Linux BIOS could be replaced with basically a register initialization table and about 50 lines of assembly language.

So over the course of the first four, five months that I was at OLPC, we came out with a coherent firmware strategy based on OpenFirmware. That got us down to the point where we could put a lot more functionality in the amount of space that we had, and it kind of went on from there. A lot of the rest of the time I spent using OpenFirmware to diagnose hardware problems.

OLPC has a lot of innovative hardware, which means that there were a lot of things that went wrong and had to be fixed.

Scott Swigart: I did a tiny bit of firmware work way back in the day on some Motorola UNIX systems. And I remember that the firmware was really beneficial because if the answer was, “Well, boot the OS and run these diagnostics,” the next problem tended to be, “Well, if I could boot the OS, I wouldn’t have to run those diagnostics.”

I am guessing that was the nature of some the problems, things that might be preventing the OS from booting or preventing some fairly fundamental things from working once it booted, and the firmware just kind of gets rid of a lot of stuff in the middle.

Mitch: A lot of times, we would have a situation where the OS in fact would boot, but some peripheral would be behaving in a flaky way. And the difficulty with trying to debug it under the OS is, the OS is doing so many things autonomously that it is really hard to zero in on the details.

Sean: There is just a lot less going on when it is just the firmware and the hardware basically.

Mitch: Exactly. So you can narrow the problem down. And Open Firmware has remarkable ability to let you play with things because it has got a live Forth Interpreter. So you can script anything just by typing a few characters on the serial port.

Over the years, I have developed a fairly substantial array of diagnostics and drivers for all kinds of devices. And also as a result of my history as the debugging guy at Sun, I have a pretty good war-chest of strategies for throwing scripts at hardware and seeing what happens.

In principle, you can do that with shell scripts and various things at the Linux level, but in practice, the productivity is probably two orders of magnitude less, because there is just so much happening, and it takes a long time to recompile a kernel and re-download it.

Then, the answers you get are inconclusive, whereas with a firmware you start seeing some effect. You can fairly quickly change your tests to eliminate possibilities and eventually isolate the problem and prove that the hardware is doing ‘X’ when it should be doing ‘Y.’

Sean: Can you talk for a minute about some of the novel hardware that is in the laptop?

Mitch: The display is the most obvious novelty. One aspect of the display is that instead of being only backlit and color or only reflective and monochrome, it will do either, so it is sort of a combination of transmissive and reflective.

It will display color when you are inside and can put light through the back, or it will display monochrome when you are outside and it would be too bright to see the backlight in any case.

It is a very low-cost display due to some novel pixel layout characteristics, and as a result, we needed a special chip to drive it. You can’t just feed it the same kind of video stream with the same pixel orders that comes right out of graphics processors these days.

This display controller chip does pixel swizzling among other things–it was a new chip that was done on a very aggressive schedule, and it had some problems, so we had to work those out.

In addition to doing its pixel swizzling thing, this display controller chip also has one frame worth of memory, so that when the processor doesn’t have anything to do, the processor can tell this display controller to grab the last frame and keep it on the screen, and the process can go to sleep without losing the display.

That lets us do aggressive power management when the child is, for example, just reading a book or something like that and not generating a lot of interaction, or the interaction is happening on second granularity. During intervening times, the processor power can be turned off and the user doesn’t even see it.

The switching from the autonomous display refresh mode to the processor-mediated refresh mode required a lot of debugging.

Sean: Tell us a little bit about the mesh networking.

Mitch: For our mesh networking, we want to be able to maintain the integrity of the mesh while the processor is turned off. So the radio has its own processor. There is a little ARM chip on the radio module that does the core of the mesh stuff, and that processor remains powered while the main X86 CPU is turned off.

This mesh module is on a USB port, and that was actually a lot of trouble too. We had all kinds of problems with things like, when you turn off the power to the processor, does the USB interface connecting it to the ARM processor freak out?

There are all sorts of issues having to do with power management, too. General power management is a very, very difficult problem because modern computers have a lot of different power domains, and they’re communicating over buses that may cross these domains.

Sean: In terms of One Laptop Per Child, how much time needed to be expended compared to more of a typical commercial effort, where Dell is rolling out a new line of laptops and they want to increase battery life by half an hour, but they are not terribly concerned about having massive leaps forward in power management?

How much more effort do you think the firmware development effort took for all these issues around power management? I have to believe it is pretty huge.

Mitch: I would say it probably is, but I can’t really give you direct comparison because I haven’t been involved in the PC industry at the same level that I have been involved in OLPC.

Sean: Well, that is fair. What percentage do you think was of your guys’ efforts then?

Was it the major thing you were dealing with every time you tried to integrate with a new component or you got a new version of a new component?

Mitch: Power management has been a constant battle for the last year, and we are still not done. I started work in earnest on power management around January of last year and did the early startup code necessary to wake up the processor as fast as I possibly could.

I put that code to bed, but then ever since we have been fighting, fighting, fighting with all sorts of issues like when you turn something back on, it glitches the power rail and mostly works, but if you try to suspend and resume 10,000 times, it will crash.

On a conventional PC, if you could do 10,000 suspend/resume cycles, you would declare victory in some cases.

Scott: So, your PC, if it sits idle for five minutes, will essentially suspend, and then, when it resumes, which takes seconds, you reconnect to the network and a bunch of stuff at a hardware level re-initializes.

You are telling the processor to go to sleep the same way my desktop will, if it just sits idle for long enough or if I tell it to suspend, but you are keeping everything else around it alive. Like you said, you are using an ARM processor in a mesh network to keep it alive. You just put a frame in the display so that it essentially just keeps showing that same frame while the processor went to sleep.

You took a novel approach to extend battery life by essentially turning off the processor, which is one of the things that is going to draw the most power.

Mitch: We certainly had an aggressive goal that nobody had ever contemplated seriously or gone after in a really big way. The goal was basically to provide computers to people that are poorest of the poor. And in fact, right now, we are deploying in Peru, in places where they don’t have any electricity, so we have to roll out solar power along with everything.

Sean: To throw out an example from a completely different direction, Linksys has this 802.11g router, the one that is stacked up to the sky when you walk into a Best Buy. It’s a standard model they have been selling for three years.

The reason I bring it up, is that apparently, the firmware that Linksys uses on that particular router actually had an open source lineage. And so, what happened is there is a whole cottage industry in the last three years that they have built up around, essentially people adding functionality to this router.

Given that open source projects like to borrow, consume, get inspired by, or build on top of other things, how much do you think of what you have done is actually transferable?

Mitch: I’m pretty confident that much of the technology that came out of OLPC is going to be part of the landscape for many years to come. It won’t be a slam dunk in many cases to just pick up piece ‘A’ and deploy it in product ‘B,’ but, for example, the basic display technology, that’s going to happen.

It is a great display and there is no reason that it couldn’t be deployed in any number of things. And the company that manufactures the display has some time ago spent a very large amount of money building a factory to produce these displays in enormous quantity, fully expecting that it wasn’t just going to be for OLPC.

802.11s is a happening thing and the firmware improvements that are being made to make the mesh networking robust are going to appear in other products. The firmware that we use for wireless networking is not open sourced actually; it is done under contract to Marvell, which is the company that manufactures the chip.

And we really wish it were open source and there is a small group that is trying to make an open source replacement for it, but that is not very far along, as far as I can tell.

The elements of this system that are not open source, including that mesh firmware, the firmware that runs the touchpad, and the firmware that runs the embedded controller that does the very lowest level of power management, those are not open sourced, and they are some of our biggest headaches.

We have had endless fights trying to solve difficult problems in those areas where we can’t see the code.

So I would say that one of our biggest development challenges has actually been dealing with the bits of firmware that were closed source, because we just didn’t have enough clout to force them to be open.

Sean: You have been working on OpenFirmware for quite some time, and it didn’t necessarily start as OpenFirmware when you were working on it back at Sun.

From your perspective, what are some of the advantages you see in open source software at this really low hardware and firmware level, versus proprietary closed source?

Mitch: I can sort of summarize the advantages of open source things in sort of two statements. Statement one is that if you can see the source, you can debug it. Closed software is great when it works, but when it doesn’t work, you are pretty much stuck.

And the other advantage to open source is that a lawyer is not going to tell you that you can’t use what you did.

Sean: What is your impression about why, at the firmware level, more of it isn’t open source? Is there a feeling that by open sourcing the firmware, it makes it easier to reverse engineer the hardware itself? What do you think are some of the barriers at the firmware level that make a lot of companies resistant?

Mitch: Certainly, hardware vendors are very much in the mode of trying to protect their intellectual property. They have a lot of clearly definable investment in their designs. They particularly want to sell it, and they’re particularly concerned about some factory cloning their device and selling it more cheaply.

I understand that companies have a legitimate reason for wanting to make a profit, and many of the really successful companies in the past have been fairly secretive.

Certainly, IBM was not in the business of giving away all of the source code for their VM operating system, and Microsoft is not in the business of giving away source code for Windows, and Intel is not in the business of giving away masks for their silicon.

The only two games in town as far as graphics chips–Nvidia and ATI, are very, very closed about their software, their drivers, and a lot of their details about how their hardware works.

It’s hard to argue with the guys who are raking in the dough, while everybody else is sort of scrabbling around for the scraps. There are arguments in favor of opening things up. I’m not going to repeat them–they’re pretty easy to find–but unfortunately I don’t see that a lot of the people making those arguments are as wealthy as the other people.

Sean: Dozens of news articles say that Sun acquiring MySQL proves the business model of open source.

I think there’s an understanding that at some point, open source companies have to be able to get similar levels of return, similar levels of revenue and profitability, and things like that, that closed source companies have done in the past.

Sun to me is sort of a fascinating company in a Petri dish. They are taking stuff that five or 10 years ago they would have argued to be intellectual property that they could never expose to the world, and they are just putting the source code out there and it will remain to be seen what happens.

I think they are doing it even down to some of their chip masks, in addition to their software side. And so, there is this argument that really the value is in the experience and the talent pool that knows that code base and knows that architecture so well–it isn’t necessarily in the code itself.

Somebody could fork MySQL and run off and make their own version of it, but for some reason nobody does. And I think it is because it would be really hard to just staff 200 people and just keep moving the product forward and try to get everybody to convert over to your version instead of the other one.

What are your thoughts on that? You are a lot closer to it, especially where hardware meets software, which is an interesting place where this plays out.

Mitch: You mentioned Sun and openness. It is kind of interesting to me how much distrust there is between the Linux community and Sun or vice versa. It seems like the Linux community really distrusts Sun.

Sean: I think that’s a fair assessment.

Mitch: Sun was really a pioneer at making things available. If you go back to NFS, Sun made NFS available and the Linux community would argue that, “Well they didn’t make it available according to our terms of open source,” but that’s rewriting history, because those terms of open source didn’t exist at that time.

And similarly with the NeWS windowing environment, Sun promoted and made any number of technologies available under very, very generous terms by the standards of the time, but they don’t meet the current standards of open source as articulated by the Free Software Foundation, for example.

People are retroactively applying their particular worldview to situations that happened when things were very different.

Sean: What do you think are some interesting directions that someone could take the firmware for One Laptop per Child? Obviously, you can imagine ways that they could take the software stack running on it and interesting ways to do things at a software level.

But given that it is open source, where do you think somebody could take the firmware? I could see scenarios like if it was a deployed in a country, maybe they eventually take on a project themselves to modify the firmware to support some local need or some balance, maybe the power management versus another activity that they consider more important or something like that.

Mitch: That brings up kind of an interesting conundrum that I have always had as a firmware person. If I do my job really, really well, nobody knows that I have done anything.

Sean: Right–if it works well, no one ever enters the BIOS.

But assuming people looked at it as a target to move forward, what do you think the areas for improvement are? If somebody was to say, “Well I love firmware and I want to tackle this and I want to move the One Laptop per Child platform forward in a given direction.”

What would they be able to do or what do you think they should be inspired to do?

Mitch: I really don’t think there is very much opportunity in that area, because the functionality of the OLPC firmware right now is pretty complete. There are a few little things I need to do, like make it a little bit more robust to avoid getting killed if you try to update it at the wrong time.

But in terms of functionality, there is not a lot to be said, particularly in light of the fact that the OLPC security model makes it impossible to access the firmware unless you have a special developer key.

So, by and large, the firmware is kind of under lock and key, because if you can get access to the firmware, you can own the machine in a very profound way, and it pretty much violates every security assertion you can think of.

Now, you can get a developer key just by asking for it, and our security model is not intended to prevent people from doing things, but rather to prevent bad people from stealing the machines. Nevertheless, most people are not going to bother.

Most people are going to be working at a much higher level and doing Python scripting and this and that and the other.

I think it is more interesting to contemplate where OpenFirmware could go in other platforms. One of the things that I worked on recently was the ability to boot Windows XP from OpenFirmware. That’s kind of working, so that maybe makes it plausible to use OpenFirmware in some embedded platforms based on X86 processors, where you previously could only really consider a conventional BIOS, because you were unwilling to give up the possibility of maybe sometime running Windows.

Sean: I am curious about what that entails at a firmware level to boot XP versus boot Linux or something like that?

Mitch: Well, to boot XP you have to emulate a list of BIOS services that are packaged as real mode INTs of a traditional kind. This is maybe changing with EFI, but as of right now, the historical legacy real mode BIOS INTs are the name of the game.

Basically, what I had to do was go through and build a call gateway, so that I could bounce a real mode INT into OpenFirmware, and then I could just write Forth scripts to implement the functionality.

Sean: How much of a community is there around OpenFirmware? Are there dozens of participants, or hundreds, or what?

Mitch: There are probably dozens, but not very many dozens. We have given OpenFirmware classes to any number of people inside IBM, and there are quite a few people inside Sun that know it, but the difficulty is that most of the people that know it are inside some company.

Sean: So you are one of the few independent people who knows OpenFirmware?

Mitch: Yeah. I have a few colleagues that I call on from time to time when there is work to be done, and there are actually quite a few people that have done OpenFirmware in the past that would very much like to be doing it now, but since the industry, has kind of collapsed into PCs, they just really haven’t had an opportunity.

So if the OpenFirmware things were to take off again–and I am not really expecting it to in any big way–I know some people I could probably reengage.

Sean: This has been great. We usually turn the microphone back to the person we are interviewing and say, “Hey, did we not cover something you wish we brought up, is there something unique about the subject you’d want to just get on the record or just kind of a closing thought or two from your end?”

Mitch: Yeah, actually, there is one observation I would like to make. In the time that I have been in the industry, the speed and capacity of most aspects of the computer have changed by a factor of several thousand, but the space available for firmware has changed by a factor of at most 10.

So I have sort of maintained this frugality mindset over the years that seems to be going away in a big way and people don’t think anything about throwing 10 megabytes of code at something.

Sean: Well, it is funny you mention that, because I think eventually we will be on the ninth generation language and it won’t even look like a computer language. Our grandkids or even our kids will have a concept of programming that won’t be anything like what we have.

It seems like the present time will be kind of the dark hallway of computing, and you will never the same largesse, that people who hang out in the OS actually get, unless something changes, but it doesn’t sound like from your perspective it is really going to change in the near future.

I mean, maybe there will be some growth vectors, but firmware will always be kind of tied into this really tight box while still trying to do a lot of things.

Mitch: Never say never. You don’t know what is going to happen with hardware, but it is certainly the case that over these 25, 30 years, they are still putting in ROMs that are about a megabyte.

When I was first starting up in firmware at Sun, we had 128K ROM that quickly bumped up to 256K and now we have grown by a factor of four, but in the meantime disk sizes have changed by a factor of 1000.

Sean: And you have also got the multitude of devices that are in the computer, just in terms of the different ways to connect, although I would assume that kind of collapses and expands. At one point, your wireless controller is separate from your Ethernet controller, which is separate from your EVDO controller, and eventually they are all in one device that talks on a standard that can communicate to all of it I guess.

Mitch: Well, they certainly collapse in terms of what the hardware looks like, but the software doesn’t collapse at all, because they still retain their old programming models.

Sean: That’s true, because you still have to interface with them, so even if it is all sandwiched on one little mini-PCI card, from your perspective, while they saved on space, they didn’t really save on the amount of stuff that you have to deal with to make the device perform the actions that it is designed to do.

Mitch: In fact, in many cases it gets worse, because in order to preserve the old programming model, but also acquire the new functionality that they want to add, they hide things behind a bunch of layers.

So especially at the firmware level, you have to deal not only with the original programming model, but with all of the layers of modalities that lets you put it into that model.

Sean: We both appreciate you taking the time to do this. Thank you.

Mitch: Thank you.

Bookmark this: