Interviewers: Scott Swigart and Sean Campbell

Interviewee: Jim Grisanzio.

In this interview we talk with Jim. In specific, we talk about:

• The process of opening a huge codebase to the public
• The unique place of OpenSolaris in the open source community
• Lessons learned from earlier open source projects at Sun and elsewhere
• Trading lessons learned back and forth with the Linux community
• Regional differences in the role of open source technology in Asia
• The reasoning behind Sun’s products and technologies that compete with each other

Scott Swigart: Jim, tell us about yourself and your role at Sun to get us started.

Jim Grisanzio: I’ve been at Sun for about eight years now, in multiple roles, mostly focusing on project management in one way or another. You can read my bio at blogs.sun.com/jimgris/page/bio.

The first three and a half years or so, I was in a communications role at the corporate level, doing lots of outreach to the media and analysts, focusing mostly on Sun’s software businesses, including Java and various open source projects.

I started to drift over to work more with the developers and engineers, because they were closest to the action and some of them are excellent spokespeople. It was very interesting to go into the core engineering operation of the company and get those people in front of the media and the analysts, both for competitive undertakings and just for plain informational efforts.

I enjoyed interacting with those very technical people a lot, and as OpenSolaris project was created, I decided that I really wanted to dedicate myself to it. So then I was faced with the task of moving from marketing into Solaris kernel engineering. That’s a tough move.

Scott: [laughs]

Jim: It took a lot of interviews, tapping those very same engineers I had been working with previously. I eventually came up with the idea of focusing on communications among the community. Communities are large numbers of people, and they’re distributed globally.

To get a project started, there’s an important role to be filled in focusing on that and that alone. You already have plenty of engineers, but you don’t have plenty of community people.

So, that’s how we crafted it, initially, and it worked out very, very well. I came over from the marketing side to the engineering side, I started off in a project manager role, as the community manager. I’ve been on the project ever since.

I was actually on the first team that opened up all of Solaris. We were charged with creating the program itself. Now, four years later, everything is distributed and it’s a very, very different place.

Scott: Linux is on an ascent; you’ve got IBM with AIX, you’ve got HP with HP-UX, you’ve got Sun with Solaris. And Sun made a fairly unique decision to take this proprietary Linux and open it up; in fact, Sun is obviously doing that across the board with all if its products–making Java open, and even making hardware specs open.

How do you even start on that? I’m not sure people realize the complexity associated with open sourcing a product, and you even hear things like, “Microsoft should just open-source Windows.” I think what people don’t necessarily realize is that with proprietary software, you probably license technologies from other companies. There’s a whole bunch of encumbrances in the software that prevent you from just posting the code to the web the next day.

Plus, as you mentioned, you want to build a community around it. You want there to be a way that people can be involved. So, how do you even start down that path of taking something as big as a proprietary UNIX operating system like Solaris and getting it to the point where you are now, four years later?

Jim: Well, you mentioned Microsoft. Even if they wanted to open Windows, it would take years to do it, just as it’s taken years for us. And that’s what you do. You sit down, and you map out a multi-year strategy, knowing full well that things will change along the way.

Now, the story of how we opened Solaris has never really been told, although bits and pieces have been. I’ve been sort of tinkering around the idea of writing a book around this, because I’ve been involved in most of it; certainly not all of it, but I’ve been around long enough to know all the characters.

Multiple things happened at the same time. Some executives and engineers had been wanting to open Solaris for a long time. We opened it in June of 2005, although there was an entire year before that when our team was preparing, so that brings you to about June of 2004.

There were at least two or three years prior to that when there were discussions among various kernel engineers kicking around the idea. So, that’s late Solaris 8, early Solaris nine timeframe. That’s a long time ago now.

There were multiple ways this came together from the top and also from the bottom and from sideways, as well. And, you’re right that we didn’t own all of what’s in Solaris. A lot of the code, like what’s in Solaris 10, is brand new, written by a bunch of 30 year olds, whereas the older stuff has been around for a very, very long time.

We had a very serious challenge in the licensing and development methodology and business models around this. And there were many, many competing ideas on how to do it; how to license it, what the development methodology would be, what the business model would be, and it took years to figure all that out. Quite frankly, in fact, we’re still figuring it out.

We already have some experience at this now, but no company of our size has released all three of its core products at the source level. Sparc, Java, and Solaris are the three core products of the company, and to open those up has been quite a learning experience.

There was no model to look at, really, so it meant spending a couple years going through the code and finding the license agreements. This stuff goes way back, and we spent lots of time digging in old file cabinets.

We also had to find a license that would be workable with the many, many copyrights inside the code base, so we had to be able to mix licenses, essentially. We needed to figure out a development methodology that would work with our base–the old Solaris market–as well as what we wanted to build. And even then, we still had to address the process of opening everything.

When we actually opened in June, 2005, we didn’t open up everything. We opened up most of it–it was mostly the kernels at that point–but Solaris is designed with huge chunks of code called consolidations. The unique kernel is one, storage is another, and install is another, and you get these huge, huge chunks of code that come together inside of a distribution. We couldn’t open it all at the same time. Heck, we were just worried about making sure this thing actually could build on the first round, because we could not open all the bits as source.

There are a few things that we had to keep closed because we didn’t own them. It was kind of minor stuff compared to the scale of the source that we actually owned, and some of the stuff that we had to keep closed initially we’ve subsequently opened.

I would say the entire first year and a half was a constant wave of opening things, after the first release, and very few people realize that. There were 20 or 25 releases of code after the initial kernel, and most of it’s open now. I don’t think we’ve opened up anything recently, except little projects here and there that are continuing to open.

It took us a couple of years to do, after we figured out the licensing scheme and everything. This has left us vulnerable, in some sense, from a business perspective, because we didn’t have the benefit of, say, something like Linux that started very small. Linux started from an individual, then a group of people, and then a large group of people, and then companies got involved, but it started way off in the corner.

Four years ago, Sun was under competitive attack. Everyone was hitting us hard, and from a business perspective, things were challenging. We were going to open up these huge code bases while everybody watched, so every single mistake we made was a headline.

One of the things we did early on was to go to the open source community, and we had multiple conversations with the leadership. We said, “OK. We’re going to do this.” And they didn’t believe us initially, by the way. “How do we do it? What are your ideas? What are your suggestions?” We had multiple conversations, and we had lots of agreements and disagreements.

For about a year, we spoke to many, many people. That culminated in a big meeting at OSCON, in 2004, I guess. We had about 30 open source people in there from OSCON, various leaders, a couple of our core kernel engineers, and the rest of our team. We just talked and fleshed it out: “What are some things that we could do that would be really, really stupid? And what are some things that we could do that would be really, really cool?”

Scott: One of the things that is interesting when I look at OpenSolaris is that the open source community and the Linux community, to some degree, don’t really know exactly what box to put it in. On one hand, they’re happy because it’s open source. On the other hand, some of them had gotten used to thinking of Sun as a proprietary UNIX enemy. They may think it’s a pretty cool distro, but still, it’s not a Linux kernel.

I sense people being a little bit conflicted about OpenSolaris, and at the same time, it’s gotten some pretty good reviews.

What do you think? Maybe from a technical level, what are some of the advantages that OpenSolaris has by not running the Linux kernel? And at the same time, what are some of the challenges that OpenSolaris has because it’s not running the Linux kernel?

Jim: Well, from a technical perspective, you really have to speak with the engineers, but people have definitely been conflicted. There’s no question about that, and there’s no way around it.

Solaris grew from the open source community, though. It grew from BSD. And so, it has a long heritage of open source. And in the interim, while we were closed, then Linux grew, and it grew into a very large community.

One of the reasons people have been conflicted is that we have been confusing. Sun has been a huge contributor to open source over the years, but we as a company are sometimes confusing in our statements, in terms of being supportive or not.

Scott: Well, it’s hard, if you’ve got 10,000 people, to just have everybody suddenly kind of see the light and be on the same page. I mean, turning the company means individually turning everybody in it. And that doesn’t always happen simultaneously or smoothly.

Jim: Absolutely, and Sun’s culture is such that things are not just all of a sudden ordered in one way and everybody moves. This is an engineering company, and it’s always been very innovative in that respect. People have a lot of freedom to talk, to flesh things out, to argue, and to debate. Whenever you go into a meeting in this place, there are always multiple competing views, and that’s very, very cool.

Having said that, though, we have had the legitimate problem that we’ve had to open it up in phases, so you’ve only seen little snippets at a time, and it’s been growing over the years. When we first launched, we didn’t have a distribution. It was just source code. Now we have the OpenSolaris binary distribution, but that’s very recent–only six months ago.

Before then, it was just source code. We had a binary in the sense that we offered Sun’s Solaris Developer Edition and Solaris 10 itself, but those were not branded OpenSolaris. Our offering has come together in stages, so people have never seen the whole thing at once, and so it’s been confusing.

The situation is becoming clearer now, though, because of the distribution. People are noticing that it’s something they can download, use, and contribute to, and we had to do a few years of work in order to get here.

From the perspective of technical advantages, we got a lot of engineering credibility for Solaris 10. There was a big upgrade between 9 and 10, with things like DTrace, Zones, and ZFS. If you ask engineers and other technical people, they would point those things out as being competitive with Linux. In terms of OpenSolaris, those things get most of the attention.

ZFS probably is the most important new technology on the whole project, from an open source perspective. It’s a new file system, and it has also been ported over to Mac OS. I also just read recently that there’s a port started for DTrace, over to Linux.

These are the bits of technology where many people would say we have competitive advantage over Linux, although some engineers will argue against that. The Linux community has a vast community and developer base, and that’s what we’re trying to create. That’s where they’re mature and we’re young.

I also I like to have conversations with other communities. I’m always hanging out with the Linux community here in Tokyo. It’s an international community here, and they’re quite open to having conversations from the technical perspective and the community perspective; the same thing’s happening in China.

The OpenSolaris community and the Linux community are getting together over there and having conversations. They’re talking about where we can collaborate with ideas and things like that. So, it’s a little bit competitive and it’s a little bit of cooperative.

Sean Campbell: Let me ask you a question in a different area. Leaving aside any product comparisons about whether you’d rather run VirtualBox versus Workstation versus something from Parallels, did those guys learn anything from the OpenSolaris team since the acquisition? Have you guys tried to communicate with each other?

Obviously, you have a long history with getting things out there under open source, and lots of lessons learned. As Sun brings in groups like MySQL or VirtualBox, what kind of cross-pollination goes on there?

Jim: A lot. I’m not familiar with the guys at VirtualBox necessarily, but for any acquisition, there are many, many meetings at the executive, engineering, and marketing levels as we integrate the new company and the new technology.

We’ve had some of the MySQL guys come here to Tokyo, where I’m based right now. I got them in a conference room for an hour, and got them to talk to me about what they do, how they do it, and why they do it, from a community building perspective.

Sean: What were some of the highlights of that conversation? They have been successful at building a pretty strong community. What were some of the takeaways from that conversation?

Jim: Specifically with MySQL, they are essentially building a user community, and we are just actually creating our user community now. It’s brand new for us, and Solaris has been an enterprise operation. Really, really smart people use Solaris.

Well, I’m not really, really smart. I’m just a user who wants to be able to use Solaris, and it’s important for the team to be able to have conversations with people like me as well. That’s the new area we’re getting into, and MySQL has significant experience there. Obviously, they are technical users, but they’re somewhat different from the traditional Solaris focus on developers, kernel developers, and other highly technical people.

One of the things I found very interesting with MySQL is that their community leader is a VP reporting right to the CEO, instead of to marketing or engineering. When I came onto this project, I came from marketing to engineering. The argument that I used at the time is that community building is an engineering process.

I still largely believe that, because ultimately you have large groups of people hunkered around code, and that code is technical, rather than a marketing thing or a project management thing. The MySQL guys agree to some extent, but they say it’s not a complete view. They suggest that there are three separate buckets: marketing, community, and engineering.

Their argument is that the community guy should interact with marketing and engineering as equals. My focus was always to get the engineer out across the firewall working with people who are outside the company. That is the first principle, since without that you cannot build a community.

But, from a user perspective, if you could build a large user community, the engineers aren’t necessarily involved in 100% of that. They are far too technical, and that focus is going to make the community much, much smaller.

Now, the user community is gigantic. I’m interacting with them and people like me, but the actual core engineers are interacting with a much smaller group of people because they’re far more technical. So, that’s one thing I learned from them.

Through meeting various MySQL people, either in California or here in Japan, I do see a lot of cross-pollination of ideas. I don’t know that we have necessarily changed our strategic models, but we have integrated a lot of ideas that we see working for one another.

This whole concept of user identity is what’s changing right now in the OpenSolaris project, as we move from purely a development project into a multi-level community of users, developers, application programmers, and others. The timing is perfect for MySQL to come in, because this concept of users is ingrained in them.

Scott: Given that you guys just rolled out a distribution, are there other Linux projects that you look at and say “I really like what Ubuntu is doing around users,” or “I really like the way Red Hat or Novell positions themselves in the enterprise,” or “I like what Citrix is doing with their open source virtualization stack.”?

What stands out to you as you march ahead with a full distribution? Are there things out there that you’re looking at saying, “That would be cool if we could do that.”?

Jim: We have been impressed with the Ubuntu model, for sure. In fact, the whole Linux community impresses me, and I believe that there is a lot for us to learn there. I’ve always been a supporter of the Linux community–even before we opened Solaris–just because they have built a very large and thriving community, and it’s something that you can learn from.

With our new package manager, it now takes half an hour or 40 minutes to install Solaris. When I started in this project, Solaris 10 took three hours and I couldn’t do it.

[laughter]

I couldn’t do it, because it was not really meant at that time for people like me. It was for system admins and high-end developers. At that time, when we launched Solaris 10, we had just finished the port over to x86. It’s come a long way, and when you look at the distribution now, it’s far more focused.

I just upgraded last night to the latest build, and the interface is very nice. It installs cleanly, it finds all of your hardware, and general users can actually use it.

Previously, we had Sun’s Developer Express distribution, but the OpenSolaris distribution is a different model. It’s a much smaller footprint. You go out over the network, get your applications, and you upgrade. It’s not all in one big chunk. It’s also just a CD now instead of five or six DVDs.

There’s been a lot of learning, I think, from the Linux community, and I think there are a lot of things we can teach as well. Brian Cantrill is one of the creators of DTrace, and he offered in his blog to help the community with the port of DTrace over to Linux.

The other thing that’s happening now, is that as opposed to the first couple of years where we focused on opening the source and building a little niche community, we now have the early stages of a community, and we’re in the process of opening the entire Solaris development infrastructure. We want to take contributions into the distribution in the form of packages and things like that, just like you have with the Linux community.

We also want to open up the core infrastructure for kernel development, so now you’re talking about all the tools that we use inside: all the build tools and the SCM system. We have actually moved off of our old SCM system onto a new one, which we just completed this week, and that is a huge milestone. From there, we are looking ahead to being able to actually open up the whole thing outside.

The outside community people will have access to the same tools and processes that we have inside, which will help the distribution and the community building. The Linux community has an open model, and we want to have an open model as well, from a development perspective.

There are multiple levels of development, though: the kernel, applications and on up the stack. Of course, there are unique OpenSolaris features brought in here. From a user perspective, it’s pretty similar, but from a development perspective, you’re relying on the Solaris technology, which is built to run huge, highly scalable systems on.

Sean: Let me ask you a question in a completely different area. What do you think are some of the key differences in the way OpenSolaris and Linux are viewed, implemented, and embraced in Japan and China, compared to what you have seen in the US?

Jim: It’s a different world here, which is both negative and positive. I am speaking mostly for East Asia–that is, Korea, China, and Japan–as opposed to India.

Open source is very, very new here. There is lots of Linux here in Japan, but there is more Microsoft here than Linux. Microsoft is huge all over East Asia. When I came here, I thought I would see more Linux in terms of community people, but it is pretty small. One of the cool things about it, though is that it’s very international, and I can actually access the community because it is not only Japanese. It’s a very cool group of people too, by the way.

It’s pretty similar in China, as well, although maybe just a little bit smaller. I am going to be going over to China in a few weeks to have some conversations with people from the OpenSolaris and Linux communities.

Open source in general doesn’t have as big a profile here as it has in the West, and it’s not as well understood. There are huge language barriers, which is a tough issue, because there’s just not a lot of English here. When you go to places like India, there is a boatload of English, so the communication between India and the West is much more robust.

I was in Bangalore last year for FOSS India. The cool thing about that conference was that it focused on contributions. How are we Indians going to contribute to the Linux community? It was focused mostly on Linux and how India could be a net contributor into the community. You don’t hear that language a lot in East Asia. It is just not there yet.

I view that as a huge opportunity, and I am very excited about it. When you go to China, particularly recently, they are asking lots of very probing questions. The second thing that is happening in China is that the level of English skills is significantly increasing, very quickly.

I might be skewed in that view in the sense that I am interacting mostly with college kids, but they are really hungry, and they are assertive and smart.

I was there about six months ago, doing a talk at a university. There were 300 people in the room, which they think is small. I was going through my community stuff and they were asking lots of questions, about licensing, and the kernel, and all kinds of things.

Even though I insisted that I am not an engineer, and that I was not going to sit there and talk about bits and ones and zeros and stuff, they asked the questions anyway. I wound up promising to follow up with a lot of them; I’d get their card and say, “I’ll have to get a kernel guy to talk to you.” Before I got back to my hotel, I’d get an email reminder from them: “Did you follow up yet?”

Scott: [laughs]

Jim: I mean, this was really, really cool. I’ve established some relationships with these kids, because they’re very, very interested, and I’ve made some connections for them. My pitch to them is to get them involved in the main project. I’m not as interested as getting them involved only in China. That’s good, and it’s a first step, but I want them to get involved in the entire project, globally.

And that’s been challenging in East Asia because of the language barrier When I talk to Linux people in China and here in Japan, I get a similar story. We’re coming in a little bit after Linux, time-wise, but language issues are big, and the concept of really contributing, the way we look at it in the West, is not widely understood yet, but that’s changing, mostly in China right now, which is very cool.

This is a huge, green field opportunity, which is one of the reasons why I’m here. I have an interest in emerging markets, and so I’m going to spend a fair amount of time in the next few years in China. Because when we go to the universities, the interest is overwhelming.

The traffic patterns are changing a bit at our site; a lot of them are coming from East Asia now. Even in Japan, which is a pretty conservative and quiet place, the traffic from Tokyo is huge. It’s just that they express themselves differently here than the West, and it’s the same thing in China. Community building is a different thing here.

Scott: Let me ask you a question at a higher business level. Some people I talk to are a bit confused by Sun and all the competing technologies. You’ve got Solaris containers for virtualization, XVM, and VirtualBox. You’ve got your own hardware line, but you also do x86. And you’ve got your own OS, but you also support Linux. A lot of people just can’t figure all of that out.

It seems that what Sun has decided to do is maybe different than somebody like IBM or Microsoft. IBM tries to get you all the way into their WebSphere stack, and Microsoft wants to get you on Windows Server, and they’ll want you using Exchange, SharePoint, and SQL Server.

It seems like Sun’s just making sure that the door on the conversation is never really all the way closed. If Sun’s talking to somebody, and they say, “Hey, we think Solaris is a fantastic operating system,” and somebody comes back and says, “You know what? I really prefer Linux,” Sun can turn around and say, “You know what? We love Linux too. We just think you should run it on our hardware.”

And then somebody says, “Well, we’re going to go with Dell servers.” Sun can come along and say, “Well, but you’re going to need storage. And have you seen our MySQL database?” and this kind of stuff.

Am I onto something there, strategy-wise?

Jim: I wouldn’t quibble with anything you’ve said, really, although you have to understand that the company is going through a pretty big transition, so there are lots of parts around. People don’t say IBM is confusing, even though they have a whole lot more operating systems than we have.

Scott: [laughs]

Well, they just call it all WebSphere, and then it sounds like it’s all the same.

Jim: Well, yeah. [laughs]

Scott: Still, it seems like Sun intentionally keeps competing products in the portfolio and lets the marketplace kind of figure it out, and as long as there’s demand for both, Sun is going to just keep supporting both.

Jim: Definitely. I’m not on the product side, but it just makes sense to keep products in the line if there’s a market for them. There’s a huge market for SPARC, and we’re growing in other areas with x86. They don’t necessarily conflict, even if some of them overlap.

One point that you hit on is that we want the conversation channels open. Jonathan Schwartz says this externally, but he says it internally as well, and he’s very consistent with the message that when you open your code, and you move your engineers and your infrastructure outside for your development, you’re engaging in a fiercely substantial amount of conversations.

We do this with our blogs. We have something like 5,000 blogs now–some huge, obnoxious number. We have all these communities where thousands of Sun engineers are now outside, and we have hundreds of e-mail lists on OpenSolaris alone, and we’re engaged with tens of thousands of people.

There are multiple levels to this conversation, and it gives our salespeople access to different levels of organizations, and that is the key, from a development perspective.

When you move from development into business, I am interested in this level of conversation: how does that transform into business opportunity for the company? When we started the project, we did a pilot program, which I primarily ran.

For about a year, we opened the code to a small group of people from outside the company who came in under NDA. It was only under NDA because we hadn’t licensed yet. This was my first experience with this issue of multiple conversations within our customer base.

I don’t ordinarily speak to Sun customers on Wall Street. They’re kind of high-end guys, but when we started the pilot program, we went to our customer base and our various communities, and we said, “Who wants to come in to help us start?”

I eventually found some people on Wall Street–sys admins and developers and such–and I brought them in, and then we added some guys in the defense industry and the education industry and whatever market Solaris happened to be in.

That was a very interesting experience for me, and also for Sun, to realize that you can have multiple conversation channels into a customer base, starting from the development side. This is all very, very cool, because information flowed freely, and I think there’s more than enough evidence now that these conversations will lead to more engagements and opportunities for Sun to be able to sell things.

Scott: There are so many different ways now that Sun has to start a conversation and keep a conversation going with the customer, around virtualization and different ways of doing it, and around Solaris, Linux, hardware, storage, and databases.

I suppose that’s why it makes sense for Sun to have created a strategy where the doors never really close.

Jim: Exactly. I was talking to a customer early on, maybe four years ago now, and they said, “You know, you’re not really into virtualization.” Well, now they’ve got some virtualization to pick from.

[laughter]

It’s interesting because I remember that conversation with this guy, and he was very genuine, and we had a good conversation. And I also remember all the conversations internally, and huge plans to get into virtualization. It was already part of Solaris 10; it just wasn’t well known.

A lot of my challenges every day center around electronic communication, because it takes a long time to build a community. It takes a long time to get messages out across cultures, languages, and geographies, especially while you’re transforming the whole company and opening your core product. There’s a lot going on.

Sean: At this point on the conversation, we usually like to turn the microphone over and ask whether there are any closing thoughts that you want to add.

Jim: The closing thought I can offer is that this really has been a multi-stage process, with a lot of figuring it out as we go, and a lot of looking back and figuring out the specifics.

Looking back on it now, after four years, we’re pretty much where we wanted to be. We’re behind on some things, and we’re behind moving some of the core infrastructure outside, but we’re way ahead on other things. We had no idea we would have a community of the size that we have right now. I think we’ve moved faster on the community front than we have actually on some of the infrastructure front.

Governance is going to be a big challenge for us, I think, as we move tools and engineers outside. They bring processes with them, so we have a governance model that we have to integrate into those processes. And we will have to decide whether those are the right processes for the open source community, or whether they are too focused on Sun.

I think that the focal point, to kind of tie these things together, is the OpenSolaris binary distribution, and the fact that it’s surprisingly good. We always felt that we would get good reviews from a technical perspective, because Solaris is so strong technically, but, people seem genuinely happy with it.

Sean: Thanks for taking the time to talk with us today. This has been great.

Jim: Thank you.

Bookmark this: