How Software is Built

Interviewers: Scott Swigart and Sean Campbell

Interviewee: Alexandre Julliard.

In this interview we talk with Alexandre. In specific, we talk about:

• Developing an open source project in cadence with Windows
• Philosophies about how Linux should reach out to Windows users
• The leadership model used in the Wine project and deciding what goes in
• Deciding what applications to support and what to leave out
• The difficulties of working with the closed Windows API and Wine’s relationship with Microsoft

Sean Campbell: To start, please tell us a little bit about your background as it relates to Wine, your role as CTO with Crossover, and that sort of thing.

Alexandre Julliard: I’ve been working on Wine since the beginning of the project, 15 years ago, when I was still a student. For a number of years, I worked on Wine in my spare time, and then in 1999, I joined CodeWeavers to do Wine development full time. I’ve since become CTO of CodeWeavers where, essentially, all we do is build around Wine.

Sean: With a project of that longevity, what interesting evolutions have you seen, in areas like a community evolution or a usability evolution?

At the same time, as the Windows platform and the types of applications on it evolves, does that create a race condition of sorts, where you’re trying to catch up to everything that’s changing, and at the same time, adding new features to Wine?

Alexandre: Yes, mostly the evolution is driven by Microsoft and how Windows is evolving. And clearly, in that time frame, Windows has become much more complex. One of the major transitions we had to make was from 16 bit to 32 bit, which was very hard for Microsoft, and very hard for us, too, as we had, of course, the same problems.

The sheer size of the Windows API is, I would say, the major factor in the evolution of Wine. The first versions were small, self-contained binary code that just provided all the APIs. Then we had to make the code much more modular, to split things up, to define interfaces between the different areas of the code. We also had to start being very serious about regression tests. At the beginning, we were just hacking things until it worked. But, it became clear that it wasn’t scaling, and that we needed regression tests to be able to nail down the behaviors of the APIs, and make sure that when we add more stuff, we don’t break the old APIs.

That was a major factor in the evolution of Wine, and the fact that it’s really becoming stable is because of the huge test suite that we now have.

Sean: Given that you have to deal with the potential of really any Windows application running on Wine, you’ve got a massive set of applications, and I’m sure, lots of legacy applications that your users hope will still run, in cases where maybe the legacy application isn’t even really fully supported by Windows anymore.

But before I ask about that, I’m curious about what you are doing around 64 bit, given that you had a big evolution from 16 to 32. You take somebody who puts 64 bit Vista on their machine, and inevitably, iTunes will only kind of sort of work until they get it working, or there will be some kind of issue with their TiVo software package

Of course, those examples are just consumer grade stuff, but, there are also issues with drivers, with smart cards, and on and on. Have you started to see those issues creep up on the roadmap as something you’ll have to address? Or, is that just far enough out from your perspective that it’s still predominantly a 32 bit world?

Alexandre: Well, it’s clearly on the roadmap, but it’s not something we’ve had a problem with yet. There are simply no 64 bit applications that anybody wants to run that don’t have a 32 bit version. We don’t run drivers, so we don’t have the problem of the kernel being 64 bit. That’s not an issue for us; it’s just the applications.

Sean: That makes sense. Let’s talk a little bit about the QA process, then. Can you give us a sense of how many people are on the QA team? What’s the method of doing QA? How much user involvement versus core members from Crossover are involved in supporting Wine? Your relationship to Crossover and Wine seems a little bit analogous to the way Canonical is to Ubuntu, but maybe that’s a mischaracterization.

Alexandre: The relationship of CodeWeavers to Wine is maybe more like Red Hat to Linux, in the sense that we’re one of the contributors, but we don’t direct anything in the project. Obviously, I’m personally directing things in the project, but CodeWeavers doesn’t have a word in my choices with Wine maintenance.

Sean: That’s a good clarification.

Alexandre: As far as QA, we have several approaches. At the start, in Wine itself, we have a regression test suite, which basically performs unit tests of very specific Windows APIs. We test them one by one, call them with all sorts of parameters and test the return values and all the parts of the behavior of a single API. It is purely automated, so it is run after each change.

We also have higher level application level tests, and we do some of that at CodeWeavers. We have a team of three or four people who test the applications. We also have automated mechanisms, based on a tool called CXTest, which has been written by CodeWeavers specifically for Wine testing. For instance, it lets us run application installations in a fully automated way.

And then, we also rely simply on bug reports from the community. We have an application database on the Wine website where people can report what works, what doesn’t, and a public bug tracking system. We try to leverage the power of open source where all of these people are willing to help and everybody tests a different application and reports problems.

Scott Swigart: Linux is obviously becoming more popular, specifically on the desktop, which certainly must be good for you guys because of things like the ASUS netbook. It seems like there’s more demand for Linux desktops than ever. Certainly the vast majority of people are running Windows, but Linux is starting to penetrate into mainstream thinking.

I encounter two predominate thoughts. One is that we should make Linux as much like Windows as we can, because that’ll make it very easy for people to move from Windows over to Linux, and that’s the ultimate goal.

It seems like you guys are aligned with that school of thought. Wine and CodeWeavers exist because they recognize that some people are just going to have to run Windows apps as part of what they do, either at work or at home. Office is probably the most the common one that people want to run.

But there’s another camp that says, “The problem is Windows. We’re not going to solve it by making Linux more like Windows, and we shouldn’t we even go down that path.”

Do you ever encounter that point of view? What is your response to the idea that making Linux like Windows is a problem, rather than a solution?

Alexandre: It’s certainly something we hear a lot. Many people don’t like Wine because of that, because they think that Linux should get its own applications rather than using the Windows ones. I think that, with the number of Windows applications out there, it’s simply not realistic to hope that all of them could be brought to Linux right away.

The motivation for Wine has always been to make a transition possible. And once enough people get on the Linux desktop, then you create the demand for real Linux applications. I certainly agree that the goal shouldn’t be to make Linux just like Windows; there’s no point. Linux is different and should remain different, but there needs to be a way to enable people to switch, and that’s where we come in.

Sean: In a different area, you’ve been leading the Wine project for a long time–what has that position of leadership been like? In a lot of open projects, you see a certain amount of churn at the leadership level, or you see the project evolve into something else, which causes the leadership team to change. But you’ve had a very steady focus for a long time leading this community.

What kind of leadership style have you taken with the Wine community? Is it the benevolent dictator? Is it a very community-oriented decision making process? Give us a sense of what decision making looks like in the Wine community. Is it top down or bottom up driven? Because every community we’ve talked to about open search projects seems to be just a little bit different.

Alexandre: Well, it is clearly the dictator model.

Sean: [laughs]

Alexandre: I’m really making the decisions, and I think part of the reason people accept my decisions is because I’ve been around for so long. And I’m the only one who’s been around since the beginning, so I have an overview of the project that nobody else has.

I’m a big fan of the dictator model, not just because I’m the dictator in this case, but because I think it’s really the most efficient. It cuts down on discussions, on politics, on all that stuff that can cause a project to lose a lot of time and cause trouble between developers. I think it’s good to have a very clear way of making decisions and not to lose time over decisions, even if they may not always be correct.

Sean: That totally resonates with my entrepreneurial experience. Sometimes it’s better to make the decision that’s 85% right if you can evolve it later. You’ll at least have made a decision as opposed to just getting deadlocked and not making anything at all, especially if you can do 85% goodness while you’re waiting to fix the 15%.

On a different topic, what was the rationale behind the versioning scheme for Wine? Obviously, the project was in use for an extended period of time before it hit “version 1.0.” But yet, after some years of development, you felt it was time to go to 1.0.

Alexandre: The idea was to provide something that people can really use. Given the size of the Windows API, it just doesn’t make sense to release something like this only half implemented, because nothing works. Most applications really require large percentages of APIs to actually be working, so that’s why we didn’t want to create false hopes and release something that wouldn’t support anything beyond Solitaire.

Sean: So what led you to finally decide you’d reached the 1.0 milestone? Was is just that you felt you had covered enough of the Windows API, or was there some other instigator?

Alexandre: One part of it was support for enough applications that people find interesting today, which are not the same that people found interesting five years ago. There were also some questions of internal design and a number of things we wanted to clean up to make sure that we can grow from here without needing to re-architect things and break compatibility and things like that. So, we wanted to make sure that 1.0 was a good foundation for the future.

Sean: Tell me a little bit about the process of getting an application “supported,” although I realize that’s kind of a gray issue because certain things will work and certain things won’t? What’s the process around filtering out, “Well, that’s somebody’s pet application, but we don’t really feel a lot of people use it?”

Is that a fairly subjective judgment on your part? There are things like Office that are fairly easy to establish that everybody uses. But, there’s probably that second tier or third tier of applications that I imagine it’s a little hard to decide about, especially if it’s a vocal sub group that wants it supported.

Where do you draw the line between the core things you’re going to support and the things that you push off to the community?

Alexandre: Clearly, we need to go where the market goes and try to figure out what applications people want. But in general, it’s very hard to say, “I have this application and I want to make it work.” What we usually end up doing is to fix bugs all over the place, and then we look at 10 applications and we find out that three of them work and we decide that we aren’t going to support the others for now.

It’s really more reasoned by how well an application is already working than by a conscious decision of concentrating on one application instead of another one.

Sean: It sounds like sort of a mixture where there are certain things that you obviously want to work on, as well as other things where the decision is made by the amount of friction it takes to get it working and the amount of work you have to go through.

Alexandre: Yes. And the process of getting an application to work is fairly slow, so we depend a lot on the community and on people spending the time to make their favorite application work. Once they do that, it becomes an application that we support. And it is sometimes the case that one developer is the only one in the world who cares about this application, but that’s really enough to make it work and support it.

Scott: Talk a little bit about what some of the most important applications are. From my perspective, on the outside, it seems like Photoshop is something that comes up a lot, and both the older version and the current version of Office seem to be really high priorities for people. What do you see as the top five applications that everybody wants to run?

Alexandre: Office is clearly one of the major ones, though it has changed a bit. A few years ago, there was no discussion, everybody wanted Microsoft Office, and today, it’s a bit less so. People are happy with OpenOffice, and there is less pressure on Wine to support Microsoft Office. Photoshop is in high demand, as is Outlook (more so than Office as a whole), and games are fairly big too.

Scott: I noticed that there is a Crossover product specifically for games.

Alexandre: Yes, that’s fairly recent actually. We noticed that running their favorite games was something that people really wanted to do with Wine, and it turns out that most games are fairly easy to support, compared to something like Office. The range of APIs that games use is much smaller.

Scott: Outside of the 16 to 32 bit change, what was one of the hardest changes you had to go through?

Have you had the experience of waking up one morning to realize that Microsoft has released some update to “X” product, or a Service Pack, or a particular OS change that was particularly more painful? Can you describe for us one of the more challenging engineering things that you had to deal with as Windows was evolving?

Alexandre: The truth is that Microsoft cannot change things easily, because they are a slave to compatibility just as we are. We typically don’t suddenly find out that we need to support something that we hadn’t planned for.

The most problematic thing for us recently is that applications are starting to depend on features that are only present in recent Windows versions. For a long time, you couldn’t release a Windows application if it didn’t run on Windows 95. So we had Windows 95 compatibility and everything was fine. Now there are more and more applications that don’t bother with 95 compatibility, so we find that a lot of the architecture that was good enough for 95 support is not good enough for XP.

Sean: What do you think about the VMware Fusion and Parallels coherence and unity features, since this is starting to blur that line of what OS you’re running? People are starting to see the option, for example, of running XP on a virtual machine under Linux, and that’s where you load your Windows only apps.

What’s your opinion about actually running the app on Windows through something like Wine or Crossover versus using a virtual machine where your Windows apps go, even though you’re living in your Linux partition most of the time?

Alexandre: There are different aspects to that scenario. Clearly, no matter how much effort VMware and the others make, they cannot get the level of integration that we can achieve with Wine, where the applications can integrate with the Linux applications in all sorts of ways.

The other thing is that with Wine, you don’t need a Windows license, and you can be completely independent of Microsoft if that’s what you want. That’s something you will never get with VMware.

Scott: At the technical level, as you’re working to support an application, I’m guessing that you guys have implemented a large set of Win 32 APIs, so you start running the applications and see what does and doesn’t work, right?

How do you trace that, as an application’s running? What are some of the tools or diagnostics or whatever you use to figure out where it broke? Maybe it’s because the application hit an API that you haven’t implemented, or it hit an API that you’ve implemented, but it’s not 100% compatible. Talk a little bit about how that works.

Alexandre: The main difficulty we face when we run an application that doesn’t work is to find out why. We have some mechanisms, we have a lot of traces in the code. In particular, we have a way of tracing every single call that the application makes to the Windows APIs. We have an integrated Wine debugger in case of a crash and things like that.

Really, what it boils down to is, you look at the traces and you try to spot in the trace something that doesn’t look right. That’s pretty hard. You need a lot of experience to be able to find those needles in the haystack.

Once you have found something that doesn’t behave quite right, usually it’s trivial to fix. You just write a test case, see how it’s supposed to behave under Windows, and change the implementation. That’s really the easy part.

Scott: I guess that since you’ve always got the Windows API as a reference, it’s no mystery.

Alexandre: Actually, the Windows API is a mystery and that’s why you have to run all of these tests.

Scott: [laughs] Right.

Alexandre: And it’s the same way with documentation, it’s very incomplete, so the tests are the only way of finding out how it’s supposed to behave.

Scott: What kinds of interesting or surprising things have you learned about the Windows API? In some ways, you looked closer at the Windows APIs than just about any programmer who builds applications for Windows. As you have looked at it from your unique perspective, what’s been interesting or surprising about the Windows APIs themselves?

Alexandre: Well, I don’t know if it’s surprising, but the bad quality of the API is clearly the thing we take from that. Most APIs are buggy or don’t behave as recommended or just have some have corner cases that get triggered by some applications that nobody knows about, probably not even Microsoft.

People just write code and try it on Windows, and it works, and they don’t know why. Likewise, nobody knows why it doesn’t work on Wine, so we have to figure out why the code works on Windows. We actually find a lot of bugs in applications. Wine is a great debugging tool for applications.

Scott: [laughs]

Alexandre: The thing is that the Windows API actually goes to a lot of effort to hide the bugs. Whenever the application does something incorrectly, the Windows API tries to hide it instead of crashing or showing an error.

It’s good in a way that it doesn’t crash, of course, but it also prevents people from debugging their applications. So, we find a lot of bugs that if the Windows API were a bit more straightforward in what it accepts, the bugs wouldn’t have made it into the applications.

Sean: Have you seen any kind of evolutions over time, whether positive or negative? Have you seen evolutions in the API in terms of quality, or do you feel like you could sometimes get a sense of which way the wind is blowing with the Microsoft development process internally just by where you see them put emphasis on the API from release to release or service pack to service pack?

Alexandre: Yes. In the recent API, they have clearly made an effort to try to avoid security issues like buffer overflow and things like that. That’s clearly something they are now adding to the new API that they never cared about before.

Lately it seems they are more trying to spot bugs early and to crash when something is incorrect, instead of just hiding the bug. That’s definitely a positive evolution.

Scott: Have you seen Microsoft’s relationship to your project change at all over the years? I don’t think it’s any secret that Microsoft views Linux as a competitor, so they’re not really out to make stuff run better on Linux. What’s been your relationship, if any, to Microsoft, either as a company or individually–has anybody been at all supportive of the work that you’re doing?

Alexandre: Actually, we’ve not had any contact with Microsoft at all. We know that they are very aware of Wine, and they watch what we’re doing very closely, but we’ve never had any kind of official contact, either support or threats or anything. They just don’t talk to us.

Scott: They have just completely left you alone? They never sent the lawyers, but they never sent tech people to help you out either?

Alexandre: No.

Sean: We’ve reached the end of our time, but I wanted to thank you for taking the time to talk with us today.

Alexandre: OK. Thanks.