Interview with Jeroen van Meeuwen – Fedora Project Vice President Fedora EMEA

Interviewers: Scott Swigart and Sean Campbell

In this interview we talk with Jeroen. In specific, we talk about:

Sean Campbell: To get us started, could you please tell us a bit about yourself and your relationship to Fedora?

Jeroen: Sure. I regard the Fedora project as one of the most innovative Linux distributions. I started using Linux on a workstation about ten years ago–just hacking around to learn how stuff works–and it wasn’t until 2005 or so that I started making actual contributions back.

And now, just a few years later, I am Fedora project ambassador, which means that I have a great position from which to get the message out and talk to people. I am the Vice President of Fedora EMEA, which is a non-profit organization to aid the Fedora project in getting resources, mostly from third-party corporate organizations.

I also develop some applications; one of the most visible ones is Revisor, which is a distribution composer. I have various positions within the Fedora project to coordinate people’s efforts, in order to get things done effectively.

The Fedora project is one of the most amazing challenges I could ever wish for. The community is so open that if you are new to the community and you want to get something done, you just stand up and shout what you are doing, and everyone else will accept that, and they’ll help you get involved and started in making contributions.

Like any free and open source software project, contributors can get lost in getting stuff done, getting involved, and innovating things. That makes for a huge challenge that I find very, very interesting, and that’s how I got involved–that was my big motivation.

Sean: In the last several months, everyone’s retirement account is kind of melting down, although if you take the long view, it should all work out eventually. In times like these where layoffs are a fact of life, do you see that as an opportunity for people to dedicate more of their energy toward contributing to an open source project?

Jeroen: Obviously, getting laid off is very stressful, but it could create that opportunity to contribute back, as well. Speaking for myself, I make a very good living being a senior system engineer, preferably Linux, but I do a little Microsoft and Cisco as well.

Right now, I get a couple of hours a week to invest in whatever I want to do with free and open source software. That might be to promote Fedora at an event, prepare a presentation, or do this interview, for example. I make a very decent living doing what I do, and part of that is the Fedora project.

Whatever your circumstances, I think it is a huge opportunity for everyone who has an interest to continue developing themselves and making sure that they excel in a very unique sense, in the public domain. At any rate, that approach works very well for me, and it has built me a nice CV.

Sean: What motivated you to pick the part of Fedora that you started working on? How does somebody from the outside identify a niche in such a large project?

It seems like cruising the mailing list, looking at the documentation, and downloading the software would get you familiar with the project as a whole, but it might take another step to find a place to apply yourself to development, even after you have decided that you want to get involved.

Some projects have an easier roadmap than others; CentOS says the way to get involved is essentially to show competence. That’s a useful, short answer, but it’s still not a roadmap. What led you to wake up one morning and get involved, and how did you decide on the first step?

Jeroen: For me, it all started with not agreeing with what was happening. I think it was in the Fedora Core 3 timeframe, and I took issue with the way I had to download Fedora. You got a large ISO, and if you downloaded it two or three months after the release, you had to download another 700 megs of updates.

I was looking for some kind of slip-streamed ISO image I could download, so all the updates would have been incorporated in it. I found a sort of sub-project of Fedora that was doing work in that area, Fedora Unity, and they were having problems doing their composes–slip streaming the updates and stuff like that. That was something I decided to sink my teeth in, and that’s were it started.

Scott Swigart: Every distro has a unique ethos and notion of what they feel like their mission is. What do you think makes Fedora Fedora?

Jeroen: We have not even yet released Fedora 10, and like six weeks before the release, we were already planning Fedora 11. We were already able to build Fedora 11 with all kinds of new packages that would have otherwise broken Fedora 10 in many, many ways. But, we created the opportunity for people to get those packages built anyway and commit them to the Fedora 11 branches.

That is kind of what defines Fedora–we always look forward and lead. We focus very strongly on future development, and having Fedora 11 packages being built before Fedora 10 is released is a perfect example.

Scott: Talk about the relationship between Fedora and the upstream projects. Are there certain projects that are most important to Fedora, or how does that work?

Jeroen: One of the big, essential differences between Fedora and other distributions is that we’d rather gain one contributor than a dozen users. In fact, if I could lose 1000 users right now and gain a contributor, I’d do it. It’s not up to me, but if it were, I’d do it.

And that one contributor is gently forced to put whatever he or she contributes upstream. That’s the only way it works–you cannot ship patches to alter whatever GNOME does on your desktop or my desktop without pushing that patch upstream. And if it isn’t accepted upstream, it will not land in Fedora.

We don’t have ways to track what patch is in Fedora and is not upstream, but since we have groups of maintainers doing a single package, every maintainer is checked by the others.

Our policy is to push everything upstream and fix stuff upstream, so Fedora does no work other than making upstream contributions, aside from building a few nice ISOs every six months and building and maintaining the infrastructure to test, develop and distribute to get the software and the releases to whoever is using Fedora.

Scott: That is different than Ubuntu and a lot of other distributions that are more focused on users and making the easiest user experience, which leads them to carrying their own patches and things like that.

Would you say that’s a fair assessment of what makes Fedora different? Not to call Ubuntu out specifically, but does that ring true that other distributions are sort of focused on the experience, whereas you are really focused on the code in the upstream projects?

Jeroen: It is. We aggregate the best of free and open source software and make it work in a general Linux distribution, whether that is Ubuntu or Red Hat or Fedora. We adopt it in an early development stage, if we think it is the next generation.

SELinux is a perfect example–we had it in Fedora Core 2. I have not spoken to a single person who had it enabled in Fedora Core 2, but we shipped it anyway. Since then, we’ve made it into an enterprise product. Everyone wants SELinux, and I think that is one of the most eye-catching technologies that we’ve adopted so far. Now it’s sort of mature, and it has the utilities and the GUI notifications and stuff like that.

To return to the original topic, our approach is very different from other distributions, which basically consume upstream, and when things go wrong, they may or may not send a patch back upstream.

Scott: Talk a little bit about the relationship between Fedora and Red Hat Enterprise Linux.

Jeroen: Red Hat Enterprise Linux could be regarded as a long-term supported, commercially available, hardened Fedora distribution. By hardened, I mean that it takes a lot of people a lot of months to certify a Red Hat Enterprise Linux product on various types of hardware and for various types of applications and to do the necessary amount of QA before it gets out to enterprise customers.

Fundamentally, RHEL is driven by making sure that the support lines at Red Hat don’t have to ring any more than necessary. As I said, Fedora, on the other hand, is very much focused on development, and we do what we think is right. We move fast, we apply fast paced changes, we release early, and we release often, with no regard to whatever is our downstream distribution.

To take RHEL as one example, we do not care at which point in a Fedora release, or in a number of Fedora releases, Red Hat decides to distill whatever packages they want to put in an enterprise Linux distribution, because that is their independent decision.

I think that Red Hat sees it the exact same way, but from the opposite perspective. They let us do what we do best and support us in doing so, so that future versions of RHEL will have those innovative technologies and a number of extra features when they ship.

They also invest a lot of time in issues that no volunteer outside of Red Hat would have picked up. A good example would be Cobbler, which is a PXE provisioning framework that Red Hat customers found lacking in Linux. Red Hat Emerging Technologies has picked that up, and Michael DeHaan has been doing a great job at it.

Right now, we have a brilliant open source software alternative for provisioning. It’s Free Software, and it’s in Fedora and the Fedora Project’s Extra Packages for Enterprise Linux. Maybe it will be in Red Hat Enterprise Linux 6, because it’s one of the most requested features–deployment utilities, but I don’t know.

Scott: You said that if something is not in the upstream project, it’s not in Fedora. Red Hat obviously has a different kind of customer base and set of requirements than Fedora.

Is it your impression that Red Hat carries some of its own patches to some of the projects, if it feels customers need them but it isn’t able to get them upstream in time for one of their enterprise releases?

Jeroen: There is almost no such thing as them not being able to get it upstream in time, because Fedora has already done that work for them. Fedora 10, for example, will just work, and all they need to do is harden it.

But then again, those patches go upstream and may run in Fedora Rawhide, which is where you can test patches and fixes while they are being reviewed by the upstream folks. Rawhide is incredibly fast paced and fast changing.

There are a number of people with their workstations running Rawhide, and they complain every time it doesn’t boot. I’ve had issues with the Alt-TAB key combination that made me eventually disable it. That’s when you notice how addicted you are to using Alt-TAB, but otherwise, I couldn’t get any work done. That stuff will happen.

When new users have questions about that “Rawhide” thing they hear about every so often, we often tell them it’s the latest and greatest and as such, it will break. The actual words we then use are, “Rawhide eats babies…for breakfast”.

Sean: I thought your remark was fascinating when you said that you would rather get one contributor than 1000 users. At the same time, for Linux to be “successful,” someone has to have a user base, whether it’s Fedora or somebody else.

Ubuntu supposedly has eight million users and Fedora has somewhere between nine and 13, depending on how people were counting it, based on some recent stuff I saw.

Does Linux have an inherent challenge in the sense of being a collection of trains running roughly at the same time on roughly parallel tracks, in terms of the sub projects? From a user perspective, they want to buy/acquire/download a thing. In commercial terms, you’d call that productizing it.

To you, is RHEL essentially that vector that productizes what you do and puts a nice logo sticker on it? Of course, oddly enough, RHEL has CentOS on the other side of the continuum that de-productizes RHEL, which is really funny, if you think about it.

Is your perspective that you’d rather make the contribution and let somebody else deal with the problem of trying to deal with the user base?

Jeroen: Are you referring to the extensive marketing that Ubuntu does, in terms of getting it on grandma’s laptop and stuff like that?

Sean: Right–Ubuntu almost fuzzes the distinction between sub-projects, from an end user’s perspective. Dell ads say, “Buy a Dell Mini with Ubuntu.” It doesn’t say “with this version of GNOME and this version of OpenOffice.”

It’s similar to the fact that Windows marketing doesn’t talk about what version of Volume Shadow Copy is installed. But still, the strength of Linux is all of those contributors being recognized and not neglecting to contribute upstream.

From your perspective, it seems that you don’t really feel you have that problem, because Red Hat handles that process for you.

Jeroen: In the area of exposure, marketing, and getting Linux on grandma’s laptop, Fedora is entirely different than most other distributions. For one thing, there’s a new release every six months, and Grandma doesn’t want to upgrade that often.

One way to look at it is that Fedora prefers to be an engineering release. We have very short release cycles and very short development cycles. There are weeks before freezes when you don’t sleep more than four hours a night in order to get your stuff done on time.

That approach is key to making the Fedora project what it is. We focus on development even if our most recent version isn’t yet released, like I described in the bit about Fedora 11 being developed six weeks before Fedora 10 is out the door.

Occasionally, discussions arise within Fedora where people say we need to create a long term support version of Fedora, but that would divert us from our development focus. If I support something that is more than a year old, one could say that I am wasting my time.

For me, long term support is in opposition to what Fedora is trying to accomplish, and it would lead us to lose the people that say development does it for them.

Sean: If you landed a Linux distribution on grandma’s laptop 18 or 24 months ago, WPA2 wouldn’t have worked, wireless would have been clunky, and certain new wireless chips from Intel wouldn’t have worked. There were workarounds and patches, but Grandma wouldn’t have been pleased.

You guys put all this stuff into a network manager that most of the distributions have picked up, and now even a webcam works right out of the box. That doesn’t really map to the fact that lots of people are saying that there’s really no great demand or market for a Linux desktop.

Then I look at what’s on the roadmap again, and I see that you are tackling multiple monitor support. I personally love the idea, as I struggle with my 24″ Samsung, but what insight can you give about the value of moving those rocks?

These issues don’t really seem related to RHEL on the server side, and you have also said that you’re not trying to target the desktop. If people were asked for a show of hands, they’d probably say well that’s what Ubuntu is trying to do. What’s the motivation behind addressing those issues?

Jeroen: Even though our main focus is not to expand our user base, that doesn’t mean we don’t do work that mostly benefits users, including stuff that may wind up on Grandma’s laptop. [laughs]

The motivation for work like that is mostly the technical challenge associated with it. I did another thing for remixes, which was a Core 7 feature, and for me the motivation was quite the same. I learned a lot while doing it, and I found it very rewarding to have other people talk about it and use it.

If you do anything within a certain distribution and it only ends up in your distribution, the reward is a lot smaller than when it spans over all Linux distributions, such as the work of Network Manager, which is awesome. You’re going to see PolicyKit, ConsoleKit, PackageKit, from a lot of people that are involved with Fedora and other distributions as well.

Those are all new technologies that would make desktop life easier. My original point was how we don’t really focus on expanding the Linux user base. We very much more like to do the work that enhances the computing experience, whether it’s for the desktop, the server, or both.

Scott: This conversation has really opened my eyes about how Fedora approaches things. To summarize that, it seems that the goal is to identify promising open source projects to be included in Fedora, which raises the visibility of those projects and encourages the people working on them to work even harder, because they want their work to be included in a release.

It also seems like the short release cycles help to fuel innovation, since people want it in the next release, and so they work very hard at it. Fedora seems really to be about driving hundreds of individual open source projects to be better and to integrate and work together.

Jeroen: To put it in one phrase, you could say that Fedora is a snapshot of the best of free and open source software, and what is next. The goal is not to provide the best in the sense of what’s most mature, but to provide a sneak preview of new technologies.

Scott: That means things that are on the right track can show up in Fedora quite a while before a lot of other distros.

Jeroen: True–one of those things was PulseAudio in Fedora 8. We adopted and shipped it, and we got a lot of users complaining about their sound not working, which resulted in a lot of bugs getting logged and fixed.

That helped PulseAudio get ready to go over the entire spectrum of most distributions, and that’s what we like to do.

Sean: We’re getting close to the end of our time. Is there something else you wanted to touch on, or any closing thoughts?

Jeroen: Red Hat has been a major contributor to OLPC software development, and that desktop environment is now available in Fedora as well. So, if you want to turn your laptop into an OLPC, you can sort of render your keyboard useless …

[laughter]

Jeroen: There are only icons. A child can work with it, even if the most technical, intelligent people will need to click everything and break everything before they know how to work with it.

I’m amazed by the interface, and it is one cool feature in Fedora 10 you want to try–the OLPC desktop. It’s called Sugar desktop in Fedora.

Scott: I’ve actually got one of those things that my five year old loves.

Jeroen: Have you tried to use it yourself?

Scott: Yeah, and it’s like you said–I don’t really get quite as much out of it as she does, but she’ll just explore and explore and explore the thing. It gives that sort of “it’s safe to just push something and see what it does” kind of experience.

Sean: Well, thanks for taking the time to have this call today.

Jeroen: Thank you.

Bookmark this: