Interviewers: Scott Swigart

Interviewee: Rod Johnson

In this interview with Rod, CEO of Interface 21 and founder of the Spring framework, we asked him about:

• Development methodology of Spring framework
• Contributors to Spring
• Selection of code and features for the Spring framework
• Ways in which developers support open source
• Open source development cycle
• Importance of technical support
• Code aggregation in open source development
• Choosing open or closed source within small businesses

Scott Swigart: Rod, thanks for taking the time to chat. If you don’t mind, please take a moment to introduce yourself.

Rod Johnson: I am the founder of the Spring Framework, which has become the effective standard for enterprise Java development. We also have a spring.net edition. I am CEO of Interface 21, which is the company behind Spring. I’m also the author of several books on J2EE development.

Scott: Great! Talk to me a little bit about the development methodology of the Spring framework. How does that work? Every open-source project seems to have its flavor. Tell me how Spring is built. How do ideas go from conception to features to in production or in development to release?

Rod: OK. I think when people try to understand what open source is, they instantly think about code and that the software is created by a disparate, far-flung group of people who work in their spare time. In fact, to understand the open-source model you need to step back from code and look at the totality of things, which includes documentation, includes support for users, and includes the discussion of ideas.

What we see in the core of our software is that the vast majority of the code is written by a small group of exceptionally good people. I don’t believe there is any other way to produce exceptionally good software. We have known for many years about what Fred Brooks calls the The Mythical Man-Month.

Scott: Right.

Rod: Open source doesn’t change that. We know that if you throw more people at a problem you don’t get a better solution. And it doesn’t matter what approach to distributing the source of the software you use. You’re not going to change that fundamental fact. If you want exceptionally good software, you want a cohesive team of exceptionally good people and you really want that team to be as small as possible.

Scott: How is that team motivated to work on the project?

Rod: The core people who connect to Spring daily are employees. They are being paid to do this. That model works very, very well if you look at open source in terms of the enterprise Java space and also obviously of the Linux space. You see that the model is evolving towards paying the contributors who write the code. For example, I think 21 out of the top 25 committers on Linux are employees of large corporations. The majority of the committers on Spring are Interface 21 employees. Another good example of this is JBoss, the Java application server. The majority of the code there comes from employees of JBoss, who are now employees of Red Hat.

Scott: I want to get your insight on the Linux kernel. My understanding of the Linux kernel is that unless Linus likes a certain patch, it’s not going to make it into the kernel. And yet, the bulk of the development is done by people who work for IBM, people who work for Red Hat, Intel, AMD, whoever. Right? Corporations are doing the bulk of the work. And yet what makes it in or out of the kernel isn’t really under the control of a company. Do you have any insight into how that works?

Rod: I am not an expert on Linux kernel governance. From what I see, one thing that is very, very important to developing high quality software and maintaining that software in high quality for the long-term is consequences: consequences of success, consequences of screwing up.

In the case of Linux, what we have is not a single entity, but we have a number of companies (and in Linus’ case partly also an individual, obviously) and a number of parties that are really baked into the system because there are large consequences if they screw up.

I don’t think one entity needs to be responsible for a piece of software. I think you can end up with a kind of partnership, but there have to be consequences if something is screwed up. That is one thing that scares me about the rather naive volunteerism view of open source because what are the consequences to a volunteer? Sure, in many cases they may be professionals. They may feel as a sense of personal ownership that they cannot be satisfied unless something is as good as it possibly can be. But there ultimately are problems because they don’t necessarily suffer the same kind of consequences if something isn’t taken forward.

Scott: What people have told me is that there is sort of a developer Darwinism that happens. In other words, to get your code in you have to submit it to a mailing list. You have to submit it to what one person described as a “full contact code review” and in some mailing lists you’ll just simply flat out get ridiculed and won’t be taken seriously if your code isn’t to a certain level. So, it seems reputation based, I guess. You can’t really get code in until it’s passed code review and up the chain of maintainers. So, on one hand there isn’t a lot of consequence to writing bad code, except maybe it was a complete waste of time. Your inability to do it well was pointed out for the whole world to see. So, it seems there is a certain amount of pressure to sit up straight and code well, especially if you’re going to try to get stuff into very high profile, very core open-source products like the Apache, or Linux, or some things like that.

Rod: Yeah.

Scott: One of the things that you pointed out is that people who are going to use open source shouldn’t consider it a free ride. It isn’t just something that you should just leech off if you want the stuff that you’re dependent on to continue to thrive, grow, and be there for you, and continue to be enhanced down the road.

But, at the same time, most of the core contributors to open-source products are highly skilled people who are employed by a company, and this is what they do all day long. So, if I’m going to be a user of open source, how could I best contribute?

Rod: The first thing is that I would hate to give the impression that openness means nothing. There is one interesting characteristic, for example, about a piece of software like Spring, which is overwhelmingly written by one company, compared to closed source.

You have the ability to work up to being a committer of the software. Not many people will take that up. And we may choose to approach you to see if we can hire you if you do it. And that is a way of expanding and renewing the developer base.

So, it is open in the sense that there is opportunity for people to come to be committers on the software through the community based on merit. That is interesting because if I want to be a contributor on the Java-based IBM’s WebSphere, I’m sure I could add some value based on my background to WebSphere, but I can’t put my code in there. There is no way I can do it without becoming an IBM employee.

Scott: Right.

Rod: So that is a genuine difference. There is a true meritocracy. So, coming back to your question, which was “How do you pay? How do you contribute back?”

Scott: Yeah, how do you contribute?

Rod: Well, I think the first thing is to remember that although it’s very important that there is that meritocracy based on code there are several reasons that the majority of users are never going to contribute to that code. The majority of users are not going to modify and compile it. The majority of users are going to use it essentially as if it’s a shrink-wrap product.

I make all of our consultants just use the binaries. If they’re working with our product with customers, we don’t like them to be working with the source code and making changes to the source code of our product.

The interesting thing is that it simulates more accurately how end users use it. So, often we’ll go into customers where we’ll see that they just realized that you can mount the source code and if you want to step through the same work source code…

Scott: Right.

Rod: …so, it just doesn’t make sense for most end users to even consider modifying the software. Consider that Spring is 1.2 million lines of code. Some of that code does things like transaction management, the fundamental configuration of your application. Some of it executes in virtually every runtime call set. It doesn’t matter whether its open source or closed source, you really don’t mess with code that does that kind of thing unless you know what you’re doing.

Scott: Right.

Rod: The process of knowing what you’re doing…For example, in our transaction infrastructure I, myself, as the original co-designer, made a change about a year ago. I spent almost a day conceptualizing the current state of the code (which one of our other guys had modified) and figuring out exactly how I could do these [changes] in a safe way.

I am a strong enterprise Java developer with 10 years experience. I originally co-designed this module so I know how it is meant to work, yet it still took me almost a day to feel that I can safely make a change without risking breakage of a core part of the software.

Imagine if I were someone coming from outside who didn’t have 10 years of experience, hadn’t co-designed the module. What is the probability that I’m going to make that change in a robust way unless I spend maybe five days fully getting inside it?

Now, if I am an end user and I am writing software, for example for a bank, like many of our users, they don’t want to pay me for five days to figure out how the Spring framework works internally. That is irrational. There is little economic benefit for end users. There is a high probability of being economically irrational in modifying the open source software.

Another good example is Linux. I did a lot of C-programming early in my career. I can code in C. Am I ever going to look inside the kernel and change any of the kernel? This makes no sense, whatsoever. So, code contributions and changes are not likely. However, there are many other things I can do if I wish to give something back.

One thing I can do is to help other users of the software. Most open source projects have forums and mailing lists. Every time a developer answers a question for a user that developer is distracted and can’t write code to improve the product. Both things are important but no one can do both simultaneously.

One thing that has worked brilliantly with Spring is, especially early on, we made a big effort in the core development team to answer users’ questions and be really, really helpful. That set up a virtual circle, where users saw the Spring development team was really helpful when they were first starting up. Gradually they used the software more and became experienced. Then, they see other people asking questions and they think, “Hey I know the answer to that question. Maybe Ron doesn’t have to spend time answering that question. I’ll get in there and I’ll help the other user.” That’s a valuable way of giving back.

Another thing is promoting the software because open source software, like any other form of software, can only be successful if it is popular. For example, you like a particular piece of software. You wonder whether there is a user group for that software in your city. Maybe you have time to organize the user group. Maybe you can persuade your manager to provide coffee and a room if someone else is willing to organize it.

Another way that you can contribute is through issue tracking. If you see something that you think can be better, go and report it. An issue tracker is the enhancement request. That works very, very well with open source because that is one area where access to the code really matters. We see people who make enhancement suggestions with pretty deep knowledge of the code. We might get an enhancement suggestion saying, “By the way, if this class extended some other class it would allow the possibility to do X, Y and Z. Have you considered that?”

What happens is that one of our developers looks at it and says, “Oh God, I would never extend that class. Oh, but I see what he’s getting at. Oh, yeah! He’s right. It would allow X Y and Z. So, I wouldn’t do it that way but I can see that there is a lot of value in what he suggested.” The ability to look at the software is quite interesting from that point of view. It’s not necessarily so interesting in terms of modifying the software but it’s quite interesting in terms of the ability to provide suggestions with as little or as much context as you want.

Finally, and this is the way in which I think most financial organizations will end up giving back, go find someone who is contributing to or sustaining that software and buy a support contract. Not only is that going to sustain the software very effectively but also open source software needs the ability to have maintenance, especially as open source moves into the enterprise. What happens if a system that does trading goes down at 2 AM?

There’s not much point if one of your guys gets paged. He didn’t ultimately write the software that is involved in it and the fact that it is open source doesn’t really make any difference. The need for a similar level of SLA and mission-critical support is exactly the same.

Scott: So, that’s one of the things that I want to drill into a little—two things actually. One is support and the other is the software development lifecycle.

It looks like if you take a look at something like Spring, there is nothing magic about the way it is built perhaps. You probably have internal meetings. You come up with plans. You come up with specs. Engineers are assigned. Software is written. It goes to quality assurance.

Open source gives you much tighter coupling to the user base and the community at large. They provide better issues because they look at the source code and even recommend a certain implementation and talk about the merits of a particular implementation. They file bugs directly into the same bug database that your internal people use. So, open source gives you that tighter coupling with your users. But—and I don’t think this is unique to Spring—I am finding this a lot; it is still built the way software has always been built.

Good people. Good process. Good QA. You have people assigned to do documentation. You have people assigned to do QA. Even though those are not the things that people would volunteer for, you’re not dealing with a bunch of volunteers. You’re dealing with paid employees.

Am I accurately summing it up?

Rod: I think you’ve got it. You raised one really interesting question there. You said that the community uses the same issue-tracker as we use. That is obviously true. We don’t have anything around the software in terms of management of bugs that isn’t publicly visible.

A big part of open source, to me, is free use of information. I know who wrote a piece of code, I know when it was last changed; I know what issue that person was trying to fix when they wrote that code. I can see all that information.
It also means that there is a great deal of security in adopting software that was produced by small companies. For example, at Interface 21, there are slightly under 50 people. We’re growing pretty rapidly, but that’s still a pretty small software company.

So, let’s suppose that our product was closed sourced. We would, in each deal, be dealing with escrow agreements, that kind of thing. Frankly, other people would not think of using our software. They wouldn’t necessarily know whether the internals of our software were any good. They could see it from the outside, but they wouldn’t know, for example, what’s our philosophy of bug fixes, what is the quality of the documentation internal to the code, or what is the quality of design.

I do believe that there is a very strong benefit in open source. I believe that all that information is freely available. You could look at the software, and I would strongly encourage [that for] customers making significant adoption decisions.
They’re probably never going to modify the code, but they should send some of their best guys to spend a day looking at the code. See what they think of it. See whether or not you think this code is maintainable, whether or not the language idioms reflect best practice. That’s a very interesting benefit.

Scott: One of the other things you talked about was support. That comes up a lot with open source, because the way to get support isn’t always as obvious as with closed source. If you buy something from Adobe or you buy from Microsoft or you buy from Symantec, it’s obvious where you go to get support.

In the open source world, it seems there are two models. One is, “OK, you pay MySQL, you buy their enterprise software and you get support.” It’s the same thing with RedHat and the same thing with Spring, I’m guessing – can you purchase a support agreement from Interface21?

The other option seems to be that you go to someone like Open Logic, who supports everything, and you just pay them for support across a lot of products. Now, you’re probably a little biased in your answer, but what do you think about the two different ways of getting support on a product?

Rod: There is nothing wrong with an aggregation model if you’re looking at, effectively, as a broker who is providing the ability to reach the bloke who wrote the software.

Where the aggregation model, I believe, is flawed, is in cases where essentially it’s not providing the ability to reach the people who wrote the software, and also where it is not providing the ability to pay for improvements to the software. So, what am I buying? So, let’s suppose we’ve agreed that we need to give something back. If we’re adopting free software for potentially the next 10 or 15 years, we need to give something back.

We care not only about how we get bugs fixed or how we get our production systems working, we also care that this software is still going to be mainstream in five or ten years or at least hasance of being mainstream because if it isn’t, we’re saddled with an expensive legacy. We also care, clearly, about the software evolving to accommodate new features that are coming forward in the industry over time. So that means: First, we need the ability to make critical fixes if necessary, to ensure that there’s no risk that our production systems are affected.

Second, we need to be concerned about where that investment is going to come from in terms of sustaining the product and moving it forward. So, we’re not just paying for an insurance policy today, we’re paying for the insurance company to be there tomorrow, and for the premium we’re paying to the insurance company tomorrow to be meaningful.

This is where some of the aggregation models, particularly what Open Logic calls it’s “Expert Community,” totally fall down. Because with the Open Logic model, not a cent of what you’re paying is going into writing software. You are not paying a cent towards writing software. That can never work in the long term. What you’re doing is paying, presumably- I haven’t looked at their price point, but I’m assuming their price point must be pretty low – a low price point for something that is not comparable with true enterprise support.

There are two problems with the model. One problem with the model is that it does not do anything for the health of the goose that lays the golden eggs. It says, “Well, I’m not going to feed the goose. I don’t care about feeding the goose; I’m just going to scrape up the golden eggs.” Obviously, the goose isn’t going to be there tomorrow, if that becomes the dominant approach.

The second problem is you just cannot get good quality support out there. Let’s suppose that I’ve got a piece of software that may have a lethal, critical problem in production. Now even though I think, for example, that the Spring Framework is a fantastic piece of software, the best pieces of software in the world have bugs in them. It’s impossible to write perfect software. It doesn’t matter whether it’s open source or closed source.

So, you’ve got a critical problem in production, and it’s really hard to resolve. You will see this happening with both commercial companies and closed source companies and open source companies. They all need to assign resources to work on a resolution of that problem until it’s fixed. When the developer in San Francisco clocks off work, he hands it over to the guy in Australia, who continues working on it until, probably, the guy in Romania starts work at the end of his day. And you need to be able to pay for that, and you cannot support that quality of support with essentially paying pocket money to volunteers.

Obviously, I have a vested interest. My company represents a different model. I hate it when people sell open source short. I hate the view that, “I only use this open source thing because it is going to be cheaper”. I feel so strongly about that because I used to be on the other side. I used to be the person making purchasing recommendations. In the financial industry companies where I worked, I would look at TCO. It made no sense to me to say, “Hey, I can save $100,000 here on a support contract.”

What I needed to look at was where, for example, one of the systems that I was involved in designing handles $5 trillion annually. If anything goes wrong with that system and we can’t fix it for a day, the cost to that company could be in the tens of millions of dollars. In fact, if it was down for two days in that example, the consequences to the UK economy could be quite catastrophic. So, I’m not going to be making the decision on, “Well, you know we can choose open source here and we can save $100,000 on a support contract.” I am more likely to be thinking, “OK, well if open source software seems to fit our requirements really, really well, what do I do if I have a mission critical outage? Who do I pick up the phone to call? Who can organize a team to work on this until it’s fixed?”

Scott: One follow-up: It seems that we are talking about the extremely large enterprise accounts that are handling enormous amounts of financial transactions or things like that. They have enormous vested interest in the quality and the future of the software that they are running. It seems like smaller organizations, whether they are choosing closed source or open source, kind of get to draft off of that.

Talk about the decision for the smaller organizations. For a 50-person company, how is the decision different than it is for an enormous financial exchange?

Rod: That’s an interesting point. For the smaller companies it depends on what you’re doing. For smaller companies who are not doing something that is as mission critical, it may be that the software effectively is free to them, and they don’t have the same requirements around mission-critical support.

Yet frankly, from my point of view as a chief executive of an open source company, if people who don’t need that kind of mission critical support don’t buy it because, for example, their company simply can’t afford it, good luck to them. I would love to see the 50-person company grow into a 500-person company with a website that has a huge amount of traffic for which they need to buy a support contract.

So, I think that open source expands the market and it brings the software to a new audience of people who don’t necessarily have the need or capacity to pay. Essentially, good luck to them! No one should give open source money out of the goodness of their hearts. They need to look at what they need as an organization and who can satisfy that.
So, for example, for every job where I have personally been involved, I would have been getting support contracts for the open source I was using because of the nature of those businesses.

However, let’s suppose I was working for a start-up, a dotcom that is trying to build up some kind of community site and is trying to get traffic. It doesn’t really matter if it goes down some of the time. The software that my company is developing is rapidly evolving. At that point I wouldn’t have a support contract. Let’s think about Twitter. Twitter is a great example here. Twitter clearly went through that. Then it became successful. Now they are encountering technology limitations, which are causing frustrating outage. Now you are getting into a different set of business needs. Now they are reaching the point where it is going to harm the growth of their business unless they improve the uptime of their infrastructure.

Scott: So, in other words, your goal as a start-up is to grow to the point where you need a support contract.

Rod: I have never put it in those words before but I think it’s an interesting way of thinking about it. In a way, you have a greater ability to pay the price that is appropriate to your business needs.

Scott: Right. At some point if you are a start-up and you’re not successful, then the open source that you are using is probably not as valuable to your business as if you are a start-up and you become very successful and you grow. You become an ongoing concern. Well, now your infrastructure has value because your business has value and you need to treat it as such.

Rod: Yeah. I mean what I would suggest is that in the early days people should look at purchasing things like consulting and training around open source.

Scott: Right.

Rod: I think that makes a whole lot of sense because it is simply going to reduce their total costs by making their developers more efficient.

Scott: Right.

Rod: So, the needs profile of the end-user is significantly different.

Scott: Well, I really appreciate you taking the time to chat.

Rod: No, I think that was an interesting discussion. Thank you for following up.

Scott: Thanks very much.

Rod: Thank you, Scott.

Bookmark this: