Interview with Krishna Ganugapati – VP of Engineering

Interview with Krishna Ganugapati – VP of Engineering – Likewise Software

Interviewers: Scott Swigart and Sean Campbell

Interviewee: Krishna

In this interview we talk with Krishna. In specific, we talk about:

Sean Campbell: Can you give us a little bit of background on you, your relationship to Likewise, and the company itself?

Krishna Ganugapati: I started my career in the Windows NT Development Group at Microsoft in 1993, straight out of college. My specialty was in distributed systems and networking. My personal dream was to see a mainstream, industrial strength desktop operating system in the consumer space. At the time, there was probably no better place to work on operating systems. I got to work in Dave Cutler’s group, who joined Microsoft in 1989 to build Windows NT. I think it would be safe to say that the experience in the NT development team would be hard to top.

In 1993, the NT group had 90 people. There were 30 of us in the Windows NT Graphics and Printing Team led by Leif Pedersen. There were also 30 people in Windows networking led by Dave Thompson, and there were 30 people in the Windows NT Base team led by Lou Perazzoli. There were probably another 90+ people putting together the basic foundation of what would become NT and the next generation of Windows operating systems. It was an incredibly gifted group of people to work with. I suspect that someday, people will compare Dave Cutler to Kelly Johnson or Robert Oppenheimer

My goal was to work with all three of Dave Cutler’s development managers. I started as the NT print spooler developer in Leif Pederson’s graphics and printing group. In early 1995, I switched groups to join the NT networking. I ended up working for two out of three development managers, so that was pretty good. When I went to work in the Window Networking Group, I joined the beginnings of what would end up becoming the NT Distributed systems group. Our sub group was called the Mars Group. The Mars Group, basically, was a group that focused on NetWare, around 1995 or 1996. Mars being the red planet (Novell had this big red logo).

Sean: NetWare was probably the dominant network operating system at the time, right? How did you approach that competitive situation?

Krishna: That’s correct; they had a significantly greater market share in the NOS business than Microsoft. There were two products that we started there, one was called Client Services for NetWare (CSNW), and the other was called File and Print Servers for NetWare (FPNW).

Our main goal was to build technologies to make Windows NT work with the NetWare file server, and the core problem was how to make Windows NT file clients talk to Netware File servers and how to make NT file servers look like NetWare file servers.

That was important, because Novell in those days would rather not see that happen. Novell being the dominant network operating system, the easiest way to shut out Windows NT and Windows 95 was to not provide file system client software to talk to their servers. CSNW and FPNW were interoperability products that allowed Windows NT clients to be first class citizens in Novell networks. The FPNW product was particularly compelling in that once your Windows NT clients could talk to a Netware File and Print Server, the FPNW add-on would now make a Windows NT Server system serve as a replacement for a NetWare server. I believe that the CSNW and FPNW products were critical in the battle for the NOS supremacy. There was a team of five of us in the Mars group. I should add that my job was not directly related to CSNW and FPNW. I was designing a directory agnostic client side directory API. Novell had moved beyond file and print services and had just released its Novell Directory Services 5.0 (NDS for short), now called eDirectory. We, at Microsoft didn’t yet have a directory at all! How do you then compete with a directory product when you don’t have a directory?

My project ultimately became Active Directory Services Interfaces (ADSI). ADSI’s strategy was about building a client side multi-provider architecture that would provide a common directory service programming API at the upper edge, and individual providers would communicate with a variety of directory services. We wrote providers for Windows NT (we made the NT SAM and even local machines look like directory services), Netware 3.12, NDS, and LDAP. Later on, ISVs wrote providers for all sorts of data stores and backend. The upper edge of ADSI was IDispatchable, which meant that Visual Basic programmers, COM programmers, and .NET programmers today could program any directory service. The neat thing was that applications written to one directory–say NDS–could easily be made to work against another directory, even a directory that hadn’t yet shipped! Active Directory still hadn’t happened.

ADSI was one of the first client side component object model (COM) libraries that would actually make it to the mainstream in the NT platform. ADSI debuted in NT 4.0 in 1996. Eventually, it became the API to Active Directory.

Around late 1996, the Cairo project was pretty much shutting down. Consequently, the next generation–which would then become NT5 (Windows 2000)–would basically absorb and assume the technologies that were in Cairo, and those technologies would end up becoming part of it. Cairo was supposed to the next generation operating system with distributed systems out of the box. At this point, the Windows NT Networking Group subsumed Cairo’s distributed system technologies and the combined Networking and Distributed Systems Group came under Dave Thompson. This is the group that delivered Active Directory. That would make Dave pretty much the father of Active Directory.

By about 1998, ADSI was pretty much done and I was ready to move on to something else. The Networking and Distributed System Group had grown so large that it had become two groups. Jawad Khaki, whom I’d known for quite some time, was leading the Windows Networking Group. I joined his team and took ownership of the Windows IPSec project. Microsoft was in a joint collaboration project with Cisco to build the first widespread implementation of IPSec. The project had hit some roadblocks, and they brought me on as someone with a reputation for moving projects along.

I ended up shipping Windows 2000 with two projects under my belt: IPSec and ADSI.

Sean: In that timeframe, wireless security would have been a hot topic. Did you have much involvement with that?

Krishna: Like all good things, my involvement into wireless technologies was accidental. After Windows 2000, the networking group was working with a variety of partners implementing the 802.1x protocol. The 802.1x protocol was designed for securing Ethernet switches (layer 2 security). Coincidentally, the same protocol was used for securing wireless 802.11 networks in the enterprise. For some reason, the IPSec group and the team that had begun this effort were merged and I became the development manager for both efforts. As time passed, the challenge of securing the wireless link took on the problem of automatically detecting and configuring your wireless enabled laptop to an available access point. Access points could be non-secured, secured with a key, secured with 802.1x which required certificates on the client machine and the access point, and finally secured with the end user’s Active Directory credentials. The wireless security problem had now morphed into a buffet of cryptic configuration choices for the end user.

To solve this problem, we put together a new piece of software called wireless zero configuration. The goal of wireless zero configuration was to ensure that your laptop could seamlessly detect and connect to any available access point. The zero configuration system would pop up a UI if the laptop detected a new access point and prompt you to connect. This was about six months before Windows XP shipped.

The Windows Client UI guys were really unhappy with us, because they didn’t know that we’d built this UI. But the cool thing was that wireless zero configuration made it into Windows XP, and it was touted as probably the one of the most game changing features in Windows XP. We were even featured on the Windows XP container: a singular honor! So now you know how the Windows XP wireless zero configuration came to be.

Sean: That’s a great story. What other common Windows technologies did you have a hand in during your time at Microsoft, and how did you eventually make your way over to the open source side of things?

Krishna: I spent my last two years at Microsoft as an architect in the Windows Real-time communications group. RTC had become the next major focus at Microsoft. The RTC group would integrate the erstwhile Hailstorm group and the NetMeeting folks (the Exchange Real-time communications group). All three groups had very different perspectives on where real-time communications should go. My job was to help build consensus and reconcile three different perspectives into one cohesive vision for real-time communications. The Windows RTC group believed that the future of real-time communications would be around standards-based SIP for controlling and signaling for disparate real-time communication streams. Ultimately, SIP did win the day at Microsoft and it is very gratifying to see that the Office Live Communications Group has become one of the leading providers of SIP based technologies.

When Windows XP shipped in 2003, we had that mainstream industrial-strength, consumer desktop operating system. I decided it was time for me to try my hand at building a company. So I left Microsoft in 2003 and started incubating a startup of my own and spent two years doing that. We had a really neat value proposition of building a high-end storage augmented residential/small business gateway. But we were unable to secure funding and I was beginning the process of winding down my company.

During that time, I met with Cameron Myhrvold and Richard Fade of Ignition Partners. They were invested in a company that was trying to accelerate the adoption of Linux into Windows-centric networks by building easy-to use administration tools for Windows system administrators to manage Linux file print and web servers. Part of Likewise’s initial value proposition was about extending Microsoft’s MMC management infrastructure to do this. MMC was technology designed in the Windows distributed systems group; the underlying distributed system technologies were things that I was familiar with and so I joined Likewise in December of 2005.

Around the time I joined Likewise, we changed the value proposition to be more ambitious: to making Linux systems first class citizens in Windows networks. For me, the parallels were hard to miss: in my first act, I was involved in making Windows clients and servers first class citizens in a NetWare NOS dominated intranet. This time, I would be involved in making Linux clients and servers first class citizens in a Windows dominated intranet.

Sean: If I understand Likewise’s business model, it seems a bit as if Samba, for example, had a company sitting on top of it, kind of the way Aquia sits on top of Drupal. Red Hat is obviously a much longer standing example of that, on top of the Linux kernel, building out their own distributions.

Likewise seems to draw from this idea of taking something that’s out there in the open and finding a way to build a model on top of it, and providing some type of open source solution that customers can use to get familiar with the product and deploy it in their environments. Then you also offer an enterprise product that sits on top of that.

To take the example of Samba again, that’s one of the things they don’t have. They’re sometimes criticized for not having the same level of GUI support as Windows, although that’s obviously not the Samba project’s specific goal.

Do you consider Likewise to be at all similar to Samba? It’s a different type of networking service you provide to an organization, but you also provide a level of enterprise tool support that they don’t, whereas they might just say, “Well, that’s what the config file is for.”

Krishna: Your observations are very insightful. And yes, the original value proposition for Likewise was just that. Linux had Samba for Windows-interoperable file and print and Apache for web serving. While Likewise, the company is very distinct and separate from Samba or the Apache foundation, the thesis was to be a company that could provide business solutions around technologies provided by Samba. Samba provided a file server and a print server. However configuring Samba could be very complicated. In this sense, Samba is more of a set of technologies that an OEM or solution provider needed to adapt for customers. As you’ve noted, Samba does not provide any GUI style tools. Likewise’s original goal was let Windows system administrators be able to point to a Linux server and quickly set up a file, print, or web server. The best possible way was to make them be able to use their existing familiar tools but instead point those tools at a Linux server.

However this was easier said than done. The smallest things can radically affect your value proposition. There was a social aspect we’d probably not considered. In the Linux world, systems administrators really get into the nitty-gritty of things. They’re different than Windows administrators. I’m not saying one is better than the other, but Linux administrators have always been very happy to roll up their sleeves and fire up a command line to get the job done, whatever it is. So building graphical things for UNIX and Linux didn’t really catch on in a big way. I think increasingly it’s going to happen, and in fact we’re starting to see some tools of that type, but at the time, there was really no interest.

However one thing did really catch on. People really liked the ability to join Linux server machines into Active Directory. Most of the Fortune 1000 and Global 2000 had large Active Directory installations and the Linux machines could not participate in the benefits that Active Directory was providing to the Windows clients and servers. In a sense, they were locked out of participating more fully within the corporate intranet, because of their inability to communicate with Active Directory.

Sean: What aspects of Active Directory seemed to be key to your user base?

Krishna: Three things immediately caught on. Interestingly, all of them were around securing Linux and UNIX servers

The first was single-user, single-password. Across corporate environments, Active Directory is the dominant NOS directory. If a customer wishes to deploy Linux servers in this environment, they could log in to the Linux server with their Active Directory credentials. This removed the need to have to deploy and manage a second directory.

Second was single sign-on. Once you entered your credentials once, you would not be prompted to enter your credentials again. Active directory group membership would be honored on Linux servers so access control to resources on Linux servers could be controlled through AD.

Third was centralized policy management. Because a machine was joined to Active Directory, we could push centrally configured policies out to groups of machines. Linux administrators were particularly interested in pushing out sudoers files . These are files that controlled what resources a user had access to on a Linux server.

All three of these feature requirements could be solved by integrating a Linux system into Active Directory. It appeared, from our first product line, that we had teased out a successful value proposition. Let me explain this a little further.

First, we were now providing a “must-have” product offering. By consolidating Linux, UNIX, and Mac systems into Active Directory, we were bringing in huge operational efficiencies. An existing, widely available NOS directory’s benefits were being extended to other operating systems. We were ensuring that Linux systems were now secure with single-user, single password, and single sign-on now available to all systems. Relative to our authentication offering, our management product suite was a “nice-to-have”. There are lots of examples of must-have versus nice-to-have offerings. For example, in the Windows ecosystem, one would consider Active Directory as a must-have relative to Windows Group Policy which is a nice-to-have.

Second, we were now solving an acute customer pain point, not something chronic. An acute pain offering is one where a customer is willing to pay you for alleviating his pain. On the other hand, customers have gotten used to their chronic pain, so they feel like it’s OK to live with it. For example, UNIX administrators might love to put together a bunch of convoluted Perl scripts or Python scripts or shell scripts to manage something. Giving them graphical tools, where they were comfortable with their scripts was us trying to solve chronic pain. There’s no appreciable benefit to them, offering them an alternative.

Third, you have to figure out a way to at least grow toward making the potential return on investment exponential as opposed to linear. If you’re writing MMC plug-ins, the way you add value is to write more and more plug-ins. Your return on investment is linear to the number of plugins that you build. Thus you want to be in the depth play business as opposed to the breadth play business. VMWare is an example of a great depth play. The core hypervisor technology is extraordinarily complex and hard to replicate, and so it is no wonder that they’ve captured the lion’s share of the market. Core authentication is a depth play – it is complex software; hard to replicate.

With the Likewise authentication product, we were providing a must-have, acute pain relieving, depth play product. We were securing Linux clients and servers in Windows dominated corporate intranets and reducing TCO. And the one thing that tied all these benefits was joining non-Windows systems into Active Directory.

Sean: So that brings us back to your relationship with Samba. Is that why joining Linux machines into Active Directory became so important to the overall Likewise value proposition?

Krishna: Samba consists of three key pieces of Windows interoperability technology: the file server, the print server, and the little known winbind authentication engine. The first two pieces–file and print–have always received a greater amount of attention than the authentication engine. In its first incarnation, Likewise would use winbind as the foundational technology to join Linux, UNIX, and Mac users to Active Directory. Once you joined machines to Active Directory, a Linux or Mac could benefit from all of the advantages that a Windows client would have in an Active Directory centric corporate intranet.

However, we did run into several challenges. For one, as you noted, Samba is not necessarily the easiest to use technology, and we made a significant number of changes to the code base. At one point, we were probably the largest contributor of software patches back into winbind. We also needed the authentication engine to be a programmable platform, so that we could create a larger developer ecosystem. Trying to push the kinds of changes we needed upstream could be quite time consuming and a challenge in itself.

We came to realize that most successful open source companies must be in a position where they control their own technology destiny. This allows them to be more nimble and adapt to their own market requirements in a timely manner. For example, Xen has XenSource (now Citrix) behind it and Mono is pretty much run by Novell. Other open source entities like Apache and Mozilla and Ubuntu are run almost like corporate businesses, the way they organize themselves.

So about two years ago, we decided we’d go ahead and we’d write our own authentication software system from the ground up. In January, 2008, a year later, we had written our own authentication software for Linux, which we called LWIS – the LikeWise Identity Service.

LWIS was written as an industrial strength, programmable, and diagnosable daemon. The goal was to ensure that customers could diagnose authentication failures rapidly. Suddenly, a lot of things started coming together for us. Now we had to make the decision: were we going to be an open source company or not?

Sean: Clearly, you decided to go the open source route. What fed into that decision?

Krishna: I am very proud of our company’s decision to go the open source route. Earlier with using Samba’s winbind technology, we were ethically and legally obliged to provide all of our source code changes upstream. With LWIS, we could have chosen differently. But we recognized that being an open source company made sound business sense.

First, it gave us the ability to reach a huge number of prospective customers almost instantaneously. Giving an open source dimension to your company is really valuable for purposes of seeding the market, so that’s what we did.

Another reason was our desire to create a platform developer ecosystem, in addition to an end user ecosystem. We wanted developers to program to our platform. We’d experimented with this before we released LWIS with our Likewise DCE/RPC system. A year before that, Novell had re-open sourced its version of the OSF DCE/RPC stack. DCE/RPC had pretty much languished over the past decade. This was probably one of the great ironies of platform interoperability. Microsoft’s MSRPC system is basically a fully compatible implementation of the OSF’s DCE/RPC stack. So when Novell open sourced its DCE/RPC stack, we thought “This is not exactly the best for us here, but let’s take it and clean it up, let’s rehabilitate the DCE/RPC.” So we fixed it and happily gave it back to the community. What this meant is that for the first time in something like 15 years, you could write Microsoft RPC-compatible DCE/RPC applications as first class entities on just about every UNIX platform.

We now owned our own intellectual property, we held the copyright to all our source code, and we found this amazing effect on all of the storage solution vendors and OEMs out there, who came to us and said, “Can we use this stuff? It looks pretty cool.” Our developer ecosystem had surely, but slowly begun to happen.

We’ve since continued down that path of building out this open source programmable Windows compatible distributed systems platform.

Scott Swigart: Just to back up a little bit, you said you had the choice of going with an open source approach or being a closed source proprietary company. You mentioned that one advantage of going that route was in getting exposure for your technology. Can you elaborate on that?

Krishna: One important advantage is the idea of seeding the market. There are a lot of wonderful things you can do with open source, and we all know that from a development perspective, it’s a great thing to do. But, here is why it makes sound business sense.

When you’re in the platform infrastructure business, your target markets are the IT administrators, system administrators, and application developers. At the end of the day, these individuals need to be comfortable with your product. With LWIS, we provided downloadable binaries for more than 130 platforms of Linux/UNIX and Mac operating systems. A sysadmin can now download the authentication agent for his platform install within minutes and have his Linux server authenticating to his corporate Active Directory installation. There is no pressure from us. He now gets to experiment, research, and ensure that our offering is appropriate and meets the requirements of his environment. This is really the power of open source. Customers are now in the driver’s seat, just the way it should be.

What happens next is that you actually find out that a lot of people have become champions for your product line in their organizations. If you do your job right, these people start adopting it, and the viral installation effect you can expect is rapid and enormous. So when we go into companies, we already have a little edge. I don’t have to go to a customer and talk about my value proposition. When I meet with a new customer, they often times say, “We’ve already downloaded and installed your system, and it’s working great.” An open source distribution model allows us to directly go into the demand fulfillment phase of a sale, rather than spend expensive cycles in demand creation and market education.

In addition, seeding the market with our authentication product puts us in a great upsell position with our customers. We’re in the business of building the best, the easiest to use, the easiest to deploy, and the best performing authentication engine to Active Directory you can find. What happens is that enterprise class customers need several additional features that help them roll out our authentication stack throughout their corporate networks. These include features like remote management, remote diagnosis, and centralized security policies. Indeed, there are a lot of upsell features we can actually sell to customers.

It’s really remarkable, because you quickly end up with contented and willing customers.

Scott: There is a balance to be maintained there, and I think that many aspects of it aren’t necessarily unique to open source. With a lot of software, there is a free version and a paid version.

For instance, consider what Microsoft does with Visual Studio. They have the free Express editions as well as the full paid versions. And just as Likewise does, there are a lot of companies that have a free community edition as well as enterprise upsells.

Obviously, you guys are very successful at it, but it seems that it must be very difficult to figure out where exactly to draw the line. Pragmatically, if you give too much away, people won’t take advantage of the upsell. They might pay for support or something like that, but they won’t buy the upsell version.

On the other hand, if you don’t give enough away, you don’t get that viral effect you mentioned, and you don’t end up with these really strong evangelists for your product inside of companies. What kind of insights can you share about how to decide what goes in the free, open version, as opposed to what gets reserved for the paid enterprise version?

Krishna: You described the problem really well, and it’s a hard one to gauge effectively. I don’t know if there is any one good answer. The key for us was building upsell features that enterprises need when they are rolling out the authentication software to several hundreds or thousands of machines. Things like user provisioning, reporting, auditing, and administrator troubleshooting become must-have features for the large enterprise.

We spend countless hours reviewing our feature set and most importantly focusing our attention on alleviating customer pain. Over the past few years, we’ve formalized our thinking around what our enterprise customers are willing to pay for.

User provisioning is a good example to describe our thinking around upsell features.

In the Windows world, every user has something called a security identifier that designates his identity in the corporate environment. A security identifier is an elaborately long string of bits that makes the user globally unique.

In the UNIX world, to this day, you are basically restricted to a 32 bit user ID. What ends up happening is that you actually need to have some mechanism that converts that security identifier to a UID every time a UNIX user logs in.

With the open source version of the product, typically we are dealing with departmental administrators who do not have access to modify their Active Directory information. You want to give them the easiest way to join Active Directory without having to make changes to adapt AD to Linux and UNIX. In this case, we give them the ability to synthesize a UID from the Window’s security identifier. However for the enterprise version, we provide an add-on pack that gives the customer control over what type of UID it’s going to be and store that UID in AD. They can make sure that a particular user gets a specific UID, and we charge customers for that capability.

One thing we definitely cannot do is roll back a feature that we made available in the open source version. People in the community don’t take very kindly to the fact that you’re giving them a less than full featured version of your product after promising a whole lot more.

Scott: It also seems pretty clear that you can’t take a feature back once you have offered it for free.

Krishna: Exactly, and it’s also true that once you decide to be an open source company, it’s really important that you be true to the principles of open source that you signed up to.

In the open source world, your audience and subsequent customer base is made up of people who believe passionately in the merits of an open source model. Your products appeal to this audience because of your open source stance. To no small degree, they choose to do business with you because your products are open source. Thus, you really want to make sure you come across as an authentic open source company.

It is important to note that open source doesn’t necessarily mean free. I’ve never once been in a situation where a serious enterprise customer wanted to deploy our solutions for free just because the source code was available. Remember we’re building authentication software for a corporate network. The open source binaries are the same binaries that ship with the enterprise product. The same single binary goes through our rigorous QA test matrices. Yet, we’re building something that is fundamentally mission-critical to an enterprise. Customers therefore want to know that there is somebody contractually obligated to support them if needed and therefore, there are always willing to purchase product licenses.

Scott: You mentioned that Group Policy is sort of a nice-to-have element. What have you done around that?

Krishna: I should have said relative to core authentication, group policy is a nice-to-have element. However within an enterprise, centralized deployment of policies to groups of machines is increasingly a must have feature. We support a full implementation of group policy natively on all of the 130+ platforms we support.

If you have 3000+ servers in your corporate extranet, then you want to make sure the systems are configured with a certain set of policies and that you can discover and verify that those particular policies have been applied. We support several hundreds of UNIX group policies. Here we decided that we would make UNIX/Linux artifacts as first class entities in their own right. So you can deploy cron jobs, sudoer files, and selinux policy settings. This appeals to Linux administrators because as experts on Linux operating systems, they are the best people to decide what the policies should look like, yet they get to leverage the Active Directory group policy delivery system.

Scott: I want to be sensitive to the time, so is there anything that we haven’t touched on that you wish we had? Also, is there anything you would like to add about future directions that Likewise is taking?

Krishna: I believe that both the open source community and Microsoft need to continue to make real strides towards true interoperability. I also believe that both Microsoft and the open source community have a great deal to learn from each other and a greater degree of product interoperability results in happier customers.

I spent 10 very intense years working on Windows at Microsoft, and I spent the last five on the other side of the fence working with open source. Some people ask me, “You worked on Windows a lot, how come you’re working on open source now?” The truth is that the answer has less to do with the open source versus proprietary offerings than it has to do with a fundamental focus on customers. Most customers are definitely less than thrilled with both camps, because of the challenges they face making these disparate systems work together in their networks.

For example, you brought up Samba, which is some really brilliant technology developed by some really gifted individuals. The Samba engineers are no different from the great Microsoft engineers I’ve worked with. However, Samba’s lack of usability and ease of configuration limits mainstream Samba options. And on the other side, it is no longer viable for Microsoft to think that they don’t live in a heterogeneous world. In their own self interest, the Microsoft “Better Together” campaign needs to extend beyond its own product offerings.

Imagine if you are a customer running several hundreds or thousands of Red Hat/JBoss application servers in your extranet. Imagine that you also deploy Active Directory as their primary corporate NOS domain controller. It is terribly frustrating for you if you have to deploy a second directory just to manage your Linux servers. At that point, I think you’d hope that both vendors could have your interests in mind rather than engage in a zero-sum game.

As far as Likewise’s future is concerned, we will continue to espouse the customers’ interest by building the very best of class Windows compatible distributed systems fabric for non-Windows operating systems.

Scott: That’s a great place to close. Thanks very much.

Krishna: Thank you.

Bookmark this: