I appeared recently on a podcast by Flagsmith, an open source feature flag and remote config solution, and hosted by Ben Rometsch. We discussed the licensing and business models emerging today for commercial open source developers.
In June 2019, the US House of Representatives Committee on the Judiciary initiated a bipartisan investigation into competition in digital markets, spearheaded by the Subcommittee on Antitrust, Commercial and Administrative Law. In its investigation, the Subcommittee examined the business practices of Amazon, Apple, Facebook, and Google. After over a year of investigation, subcommittee issued a report last week criticizing various business practices, and noting that “each platform now serves as a gatekeeper over a key channel of distribution.”
Among many other details, the report mentions issues surrounding the dominant market position of Amazon Web Services (starting on page 315). It observed, “The COVID-19 pandemic has underscored the centrality of cloud computing to the functioning of an increasing swath of businesses—highlighting how cloud services have come to resemble critical infrastructure.” (page 321).
Notably, on page 327, the report mentions the recent tension between commercial open source businesses and AWS. “Amazon’s practice of offering managed service versions of open-source software has prompted open-source software companies to make defensive changes, such as closing off advanced features and changing their open-source license to be less permissive…. Amazon’s conduct has also reduced the availability of features in open-source software. Confluent, Redis Labs, and CochroachDB, along with several other open-source software vendors, have made similar license and business model changes, reducing the level of access to their software.”
The NGINX dispute took another turn recently, as Lynwood Investments petitioned the U.S. Trademark Trial and Appeal Board (“TTAB”) to cancel six US federal trademark registrations of F5 Networks for the NGINX web server software. F5 acquired NGINX in an M&A transaction in 2019. Lynwood had previously filed notices of opposition to three pending NGINX federal trademark applications being pursued by F5.
This move appears to be the next step in Lynwood’s lawsuit against F5 and others alleging that former employees of Rambler engaged in a skunkworks project to create and later monetize NGINX, before selling it to F5.
Echoing statements in its prior lawsuit, Lynwood alleges that F5 “conducted extensive due diligence” before the acquisition, and therefore knew that the trademark registrations were fraudulent.
F5 will likely file its responses to Lynwood’s TTAB proceedings in the coming weeks.
The petitions for cancellation are here:
- 5978630 NGINX Unit
- 5973515 NGINX Controller
- 4837175 NGINX Plus
- 5973527 NGINX
- 4200791 NGINX (logo)
The oppositions are here:
Open Research Institute (ORI), a non-profit organization dedicated to scientific research for the public good, received a favorable final determination letter from the DoD regarding ORI’s worldwide microwave satellite network, which is designed to help ham radio operators support disaster communications.
In the United States, export of technology is regulated by two agencies: the Department of Defense and the Department of Commerce. The regulations for each of these agencies limit certain kinds of exports of technology, including software.
ORI has received a determination letter from the Department of Defense, stating that ORI’s international collaboration on the satellite network is not within the scope of DOD’s regulations, the International Traffic in Arms Regulations (ITAR). ORI now faces the much easier hurdle of clearance under the Export Administration Regulations (EAR), the parallel regulations of the Department of Commerce. The EAR contains certain exceptions that allow export of most open source software.
ORI’s description of the the project says that ORI’s phase 4 ground satellite network project will provide “an open source implementation of DVB-S2 and DVB-S2X for both satellite and terrestrial amateur radio use.”
In January 2020, Mycroft AI, a Missouri-based developer of an open source voice assistant, was sued for patent infringement by Voice Tech Corp., a non-practicing entity, in the Eastern District of Texas. (The suit was then dismissed and refiled in the Western District of Missouri.)
The Prior Art Landscape
The claims of the patents are broad. For example, the first claim of the ‘679 patent is:
A method of accessing and controlling a computer from a mobile device, comprising: receiving audio data from the mobile device, at the computer, at an audio command interface; the audio command interface decodes the audio data into a command; the audio command interface selects, from at least one operating system and at least one application, one operating system or one application, wherein the audio command interface decides is the appropriate operating system or application to execute at least one process in response to the command; executing with the selected operating system or application the at least one process in response to the command; generating output data in response to the selected operating system or application executing the at least one process; and transmitting the output data to the mobile advice.http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&p=1&u=/netahtml/PTO/srchnum.html&r=1&f=G&l=50&d=PALL&s1=10491679.PN.
Mycroft commented about possible prior art:
Voice technologies like the one that Mycroft is building can be traced back more than 50 years. In fact, Mycroft is named after a voice assistant that appeared in Robert Heinlein’s 1963 Hugo Award winning novel “The Moon is a Harsh Mistress”. All of the underlying technologies are described in the novel and they have been broken out time and again over the past half-century in popular science fiction. “Hal” from 2001 a Space Odyssey, Star Trek’s “Computer”, Knight Rider’s “Kitt” – all of these are examples of how voice technology might work in the real world. They’ve also been disclosed in real-world tech like Honda’s Asimo and more than 3 decades of automotive technologies from Nuance.Troll Hunter – Mycroft’s Position on Patent Trolls, dated February 5, 2020
Using science fiction as prior art is not unheard of, but the most famous example is probably in the Apple v. Samsung case over tablet design, which involved a design patent rather than a utility patent. Inventions in science fiction suffer from the impediment of being “non-enabling” (which is legal-speak for fictional). But while we’re at it, we could throw in Halo’s Cortana.
The lawsuit was initially a set piece in NPE patent enforcement, though one wonders why patent assertion entities ever make the decision to go after open source projects. It is almost as if they are not doing their homework. Open source projects make tragically bad targets for patent lawsuits. A case in point was the patent suit against GNOME, which ended badly for the plaintiff. Open source projects — even those created within commercial enterprises — tend to have shallow pockets, but also lots of friends who hate software patents and love to help invalidate them. That is not a good combination in a defendant profile, particularly when asserting patents that seem so susceptible to prior art challenges.
The Discussion Turns Ugly
However, in this case, the war of words has tipped into a different kind of trolling on on the defendant’s side.
The tone of Mycroft’s public blog post cited above is not surprising for an open source project. It states publicly that it will not settle the lawsuit and intends to fight it out. It calls Voice Tech a bully and a troll, ending with: “Patent trolls get paid because…it is usually cheaper in the short run to pay a troll than it is to litigate. It is also cheaper to give a schoolyard bully your lunch money than it is to visit a doctor. The thing is, once you pay the bully, he’ll just come back again and again and again.” The blog pictures Montgomery “armored up” as the “troll hunter” for the metaphorical battle.
But if you look at the archived version of the blog post, it is more aggressive. The blog engaged in what were probably intended as colorful metaphors, saying, “In my experience, it’s better to be aggressive and ‘stab, shoot and hang’ them, then dissolve them in acid.” This was actually a quotation of an article about an unrelated municipal broadband ban, linked in the blog. The archived version also says, “in the long run the best way to deal with a bully is to punch him square in the face.”
Unfortunately, the archived version also reproduced email correspondence from the plaintiff’s attorney — correspondence marked as “highly confidential,” though the basis for that designation is unclear. This correspondence contained the attorney’s return email address. The resulting community “support” allegedly resulting from the blog post may have been more extreme that Mycroft expected.
In April, Voice Tech filed a motion for “Relief to Require Decorous and Civil Conduct.” That motion reproduces an email received by Voice Tech’s attorney. The email is publicly available as an attachment to the motion filed by Voice Tech. (Case 4:20-cv-00111-RK Document 1-4 Filed 02/18/20 Page 2 of 2). Though I have cited it here for transparency, I have not linked to it, because I don’t want to dignify it with any unnecessary attention. I don’t recommend reading it, but in sum, it is vulgar, threatening, and deranged, uses a wide variety of hate speech. In fact, it almost seems like a general purpose spewing of hatred, without any particular relevance to the case. And while clearly such a horrific rant would not be “decorous” conduct by a litigant, the relationship of the email to Mycroft or the blog post is not entirely clear. Voice Tech implies that Mycroft exhorted this behavior in its blog post, a practice sometimes referred to as doxing. Voice Tech asked the court to order Mycroft to “remov[e] comments that have been published on Mycroft’s website threatening, suggesting, and/or inciting violence and death to Voice Tech’s counsel.”
According to Tech Dirt, the court was sympathetic to Voice Tech’s concerns. The court ordered Mycroft to delete a portion of its post that allegedly resulted in the threats: “I’d like for everyone in our community who believes that patent trolls are bad for open source to repost, link, tweet, and share this post. Please help us to get the word out by sharing this post on Facebook, LinkedIn, Twitter, or email.” But according to Tech Dirt, “The story had gone viral on Reddit, and…some immature Reddit users did some immature things, sending … some angry emails….There was no reason to believe they were coming from Montgomery himself.” The Tech Dirt article commented, “A company should certainly have the right to notify its community that it is in the middle of a costly legal battle (one that it believes is frivolous)…” The characterization of the email attached to Voice Tech’s motion as “immature” is accurate, but woefully insufficient.
More conventionally, Mycroft has publicly stated its intention to file an inter-partes review (IPR) to seek invalidation of the patents. Such actions are expensive, but Mycroft apparently has the support of some pro bono legal help from EFF and OIN.
Special thanks to Alero Egbe of O’Melveny for help with this post.
Update written as of December 17, 2020: Unified Patents, a member-based anti-trolling initiative, has instituted a PTAB action to seek invalidation of the patent asserted in this case. The PTAB has instituted trial on all challenged claims of the patent.
Scott Peterson’s recent article, Making Compliance Scalable in a Container World, is a good explanation of some challenges and opportunities in open source compliance for software is delivered via containers. The article advocates the use of container registries to store and deliver portable compliance information. Also, it reminds us that container continue to be a challenge for open source compliance.
Today, much software is delivered via containers. A container is a standalone, executable package of software that includes everything needed to run an application: the application software, libraries for the operating system or language platforms, system tools, and configuration settings. Containerization is the “plug-and-play” way of running software, without complicated or time-consuming installation.
Most computer users are familiar with installing applications on a desktop or laptop computer. It takes a while to install them, and for an application to work, the computer must already have a variety of standard software that the application needs in order to run on the computer platform. If installing an application is like making a recipe, creating a container is opening a takeout box.
A container registry is a service that facilitates delivery of container images. It helps developers build and manage containers, and decide who can access them. So, a registry can be used to deliver containers and track information about them.
Show Me the Code
A lot of open source compliance is about managing information. Complying with open source licenses requires you to know what open source software is in your product (its bill of materials), and what licenses cover all of it.
The average software application installation on your desktop or laptop copies software into local directories, and once installed they are relatively easy to examine. In these old-style installations, it is not too difficult to figure out which binaries correspond to the source code that generated them. So, if you needed to know what open source software was included, you might be able to figure that out from an installed software directory.
But containers are opaque; once they are created, it is nearly impossible to “unpack” them to determine their bill of materials. And it turns out that associating binaries with source code is one of the keys to open source compliance.
Delivering source code upfront, where feasible, is always the best way to comply with open source license notice requirements. That is because the license notices are “baked in” to the source code. In contrast, delivering object code only, without the source code, is more work – requiring distributors to pick out the license notice files and pass them along.
Many companies struggle with providing notices for containerized software, in part because the notice requirements of the major licenses are outdated. For example:
- GPL2 requires that redistributors “give any other recipients of the Program a copy of this License along with the Program.”
- BSD requires notices to be “in the documentation and/or other materials provided with the distribution”
- MIT requires a copy of the license “in all copies or substantial portions of the Software”
These notice requirements were all written before the web came into common use, much less containers, so they presume a means of delivering software that is about 25 years out of date.
The question of how to deliver the notices “along with” or “provided with” or “in all copies” of a container can be complicated. Every year, notice delivery gets harder to accomplish, given the increasing number of open source components, and the way software is deployed in the real world – often with no user interface and with containers being spun up and down on demand. There seems little appetite for revising the licenses to include more workable notice requirements, so many users are left in a quandary about how to do the right thing.
Using Containers as a Force for Good
“We should build a container ecosystem with compliance that is portable across registries.”Scott K Peterson (Red Hat)
Mr. Peterson’s article goes into some detailed suggestions about how to use container registries and automated tooling to help with compliance by preserving the information that associates containerized code with its source code. However, any solution using container registries needs to ensure that the means of delivery will actually fulfill the notice requirements for the licenses, and that may be a difficult problem to solve with tooling.
Fortunately, most community open source enforcers do not pursue foot foul violations when the source code has been made freely available in an effective manner, such as posting it on a publicly available web page – or, presumably, delivering information via a container registry. Such notices may or may not be in the copy or along with the software, exactly, but they help users get access to the source code nonetheless. That fulfills the spirit of the open source license, if not necessarily its exact conditions.
Is there a way to bring best practices in lines with literal license requirements by leveraging container registries? Not exactly, or at least not yet. But more information is always better than less, and more automation is the only realistic way to pursue open source compliance properly. Tooling developments in this area are worth tracking.
On June 8, 2020, Lynwood Investments CY Limited brought a lawsuit in the Northern District of California against various NGINX entities, various individuals, Runa Capital, E.venture Capital Partners II, LLC and F5 Networks, Inc., alleging the improper release and subsequent use of the popular open source software NGINX (pronounced “EngineX”), as part of a conspiracy to misappropriate corporate assets. All of the following is according to the complaint:
The NGINX software was developed primarily by Igor Sysoev, who is named as a defendant in the complaint, while he was employed at Rambler Internet Holdings LLC (“Rambler”), a Russian entity. (The plaintiff in the lawsuit, Lynwood, is an assignee of Rambler.) Sysoev also developed a commercial extension called NGINX Plus. This development was done in the course of employment and using Rambler resources. Rambler alleges that it owned the software by virtue of the work-made-for-hire doctrine (albeit its Russian equivalent).
Sysoev was employed by Rambler from 2000 until 2011. NGINX software was first developed in 2001, and released in 2004 under the BSD license. For the next seven years, Sysoev continued to be the primary author of NGINX code, making commits during what the complaint describes as “business hours.”
Sysoev’s supervisor was Rambler’s CTO, Maxim Konavolov. The complaint states, “Konovalov’s key senior management position at Rambler enabled him to provide Sysoev beginning in 2008 with an ecosystem within Rambler that was free from oversight or accountability.”
The complaint goes on to say, “Even though Konovalov and the rest of the Disloyal Employees were fixated on misappropriating the NGINX Enterprise, which they viewed as a highly valuable business, Konovalov uniformly gave the NGINX Software a rating of “1” on a scale of “1-5” with “1” being deemed “worthless” or “no value.” Konovalov’s designations were designed to … lull Rambler into complacency with respect to the value of the NGINX Software” and to avoid “any serious oversight by Rambler’s senior management or board of directors.”
While still employed at Rambler, Sysoev, with Konavolov and several other colleagues, formed a new company called NGINX, Inc., and obtained financing from defendants Runa Capital and E.Venture Capital (then, BV Capital). The complaint alleges that Runa Capital and E.Venture “knew … that Rambler maintained the ownership rights to the NGINX Software” but nevertheless “assisted and encouraged Sysoev and Konovalov while they were still employees at Rambler, to breach their duties to Rambler … for the benefit of the fledgling business that Sysoev and Konovalov were forming.” The complaint makes much of a trademark application filed by NGINX, Inc. on its first day of incorporation claiming first use of NGINX in commerce in commerce on March 1, 2011, a date on which the team were still employed at Rambler.
The funding of the new entity was not without its challenges. “Greycroft pulled out of the closing because its concerns over Rambler’s ownership of the NGINX Software…. In contrast, Runa Capital and BV Capital went forward and closed on the Series A financing on or about October 23, 2011 after conducting their own due diligence, with full knowledge that Rambler was the legal owner of the entire NGINX Enterprise ….”
NGINX, Inc. was sold in 2019 for $670 million to F5 Networks, Inc., also named as a defendant. The complaint alleges that as a result of due diligence, “F5 was aware prior to the consummation of the merger that the conspirators had stolen the NGINX Enterprise from Rambler…”
Rambler and Lynwood learned of the defendants’ alleged conspiracy when a whistleblower provided evidence to them.
This complaint, which alleges many claims including civil conspiracy, is long, complex, and makes for dramatic reading. It promises to be just the beginning of activity in this case. It was previously reported that Rambler was pursuing a criminal case in Russia based on the facts of this lawsuit, until Russian state lender Sberbank, which owns 46.5% of Rambler, exhorted Rambler’s board of directors to stop the pursuit. Rambler apparently dropped the criminal case in late 2019, instead pursuing “negotiations” with F5.
This complaint describes a situation that is, unfortunately, becoming more common – developers have been known to write code on company time, release it under an open source license without proper authorization, and proceed to form a competing business leveraging the code, claiming the right to use the code under the open source license. In such cases, the question of authorization of the release is often key, and points up the need for companies to have a formal open source release policy. But the level of misdoing alleged in this complaint is unusual, reaching up to the CTO level. The complaint is also interesting in that it names the venture investors and buyers of NGINX, Inc. – an unusual move that seeks to circumvent the corporate veil. That is a troubling development for companies contemplating due diligence on potential investments and acquisitions.
Last week, Jelruida Ltd., the maker of Nxt blockchain, announced that it had filed suit in the Netherlands against rival Apollo Fintech (“Apollo”) for breach of the Jelruida Public License, a self-described “coinleft” license.
The complaint, which was not published with the announcement but was quoted liberally in Jelruida’s press release, alleged that Apollo’s software was “cloned from Nxt.” It further states, “Apollo copied source code files from the Nxt Software one-on-one into the Apollo Software. In addition, Apollo modified source code files of the Nxt Software and included these in the source code for the Apollo Software.”
The press release goes on to say, “In 2018, Apollo launched as an Nxt clone but failed to adhere to the JPL license terms. After ignoring four cease and desist letters, Apollo belatedly complied in August 2018. The compliance proved short-lived, however, and in October 2019 Apollo replaced the JPL in its software with a proprietary license. This constitutes a violation of the JPL, which … requires that any derived work continue to be distributed only under exactly the same license. This viral “copyleft” requirement is a cornerstone of many open source licenses, most notably the GPL.”
The press release further stated that “A writ of summons has been filed in the Netherlands and the case will be heard on August 25 in Amsterdam. The court case has implications for the widespread illicit practice of cloning blockchain code, and open source software in general….”
The Jelruida Public License is self-described in its FAQ as similar to GPL, but there are significant differences.
The JPL has a “10% airdrop requirement.” The JPL FAQ says, “as a consequence of the 10% Airdrop requirement, internal use (a private or permissioned clone) is not allowed except for an evaluation purposes no longer than three months. This is because a private/permissioned blockchain clone cannot fulfill the Airdrop requirement.” The relevant language in Section 3.4 of the JPL is fairly impenetrable to the lay reader and specific to the blockchain context:
3.4 If the Covered Work is a DLT Software [i.e. a distributed ledger], after your modifications it must continue to work with the original DLT Instance without violating the consensus algorithm or resulting in a permanent fork…. If your modifications result in a different DLT Instance you must satisfy the following airdrop requirement:
3.4.1 The token holders from the original distributed ledger instance shall be allocated a portion (an “airdrop”) of the tokens in that new DLT Instance proportional to their token balances.https://www.jelurida.com/sites/default/files/JPLv1.2-NRS.pdf
These terms violate the open source definition on face, given they require a fee — albeit in crypto-coins (violating plank 1 of the OSD) — and discriminate against DLTs (violating planks 6 and 10) . Moreover, most open source licensors do not describe their own licenses as “viral.” So, Jelruida’s efforts to characterize the prosecution of this dispute as advancing the spirit of open source may be difficult.
In response, Apollo brought and action for declaratory judgment (Case 7:20-cv-00186 Document 1 Filed on 07/08/20 in TXSD) asking a US federal court to find:
- that Apollo is the owner of the copyrights in its computer software and that no ownership and/or co-ownership rights therein extend Jelruida;
- Jelruida must cease and desist from asserting that they own and/or co-own any rights therein
The complaint alleges that the Nxt software claimed as original by Jelruida was “in actuality, a public domain work which was anonymously deposited into public domain for the benefit of the entire open source community and not as something that Defendants or any of them could assert exclusive ownership or exclusive control rights over.”
It is unclear whether the reference to “public domain” refers to the absence of a copyright interest or other intellectual property interests. Outside the US, it is notoriously difficult to confidently conclude that any copyright has fallen into the public domain before its expiration.
The DJ complaint contains a number of statements that contradict the Jelruida press release. According to the complaint, the Nxt blockchain was “licensed and license fees valued at over $5 million (U.S.) dollar were fully paid” to Jelruida, who “now claim that no license or other rights exist in any aspect of the previously fully paid open source use.”
Of course, a truly “open source” use would not require a license fee, but the JPL is not exactly open source, which might explain the seeming inconsistency of this statement.
The complaint also alleges, “Since 2017 and on a continuing basis and with over 200 person-years of software engineering, Plaintiff continually wrote, re-wrote, improved and extended …its APL software. Apollo’s source code
and its overall functionality have significantly changed….The Apollo team wrote and re-wrote significant replacement and significant additional computer software. As a result, Apollo is radically different from anything owned or claimed to be owned by [Jelruida]. All copyrights of [Apollo] are owned solely by [Apollo].”
An allegation of violation of an open source license is not exactly germane to ownership of code, and does not imply “co-ownership.” Contrary to the persistent and pernicious “virality” meme still alive in open source discussions, violating an open source license does not alter ownership of derivative works of the licensed code. However, Apollo seems to be alleging that the JPL was irrelevant, because it rewrote the code to the extent necessary to avoid any copying, and thus avoid the need for a copyright license — a difficult thing to do even in a “clean room” process.
Oddly, Apollo is asking the court to declare that it owns all of the code in its software. DJ actions normally only ask a court to declare that an allegation of infringement is not true. There is a wide gulf between owning every line of one’s software and merely having the right to use it — the former being a fairly rare subset of the latter.
The complaint goes on to state, “Put simply, it is against the custom and usage of the open source community … for [Jelruida] to assert that Plaintiffs may not continue to use and continue deploy the software of Plaintiffs for the benefit of its users in the United States. No trade secrets and no patents exist in any of the
software of Defendants and all ideas, concepts, processes, algorithms, systems, and methods disclosed in the published software of Defendants have been and now are in the public domain.”
Of course, ideas, concepts, processes, and so forth are elements expressly not governed by copyright law under 17 USC 102b, so the statement appears to be saying that Apollo is not infringing any copyrightable element in the Jelruida code.
What is this Case About?
We won’t know until the facts become clearer, but the crux of the dispute seems to be whether Apollo re-wrote the Nxt code or copied it. This set of lawsuits may develop into a dispute over what elements are protectable under copyright, which would be interesting given the pendency before the US Supreme Court of Oracle America v. Google. But given the JPL is not a true open source license, this case promises to further muddy the waters as to the meaning of “open source.” It may also provide a view into the highly competitive world of crypto-currency development. Most blockchain systems are in fact heavily based on true open source software, so allegations of non-compliance are likely to arise in the field, even after this case is over and done.
For transactions lawyers, negotiating intellectual property infringement indemnities is an unfortunate and often painful fact of life. Allocation of risk terms are notoriously difficult to resolve, and often are the last issues in a deal to be agreed. Business persons consider them abstruse “lawyer work,” and lawyers consider them business issues that get short shrift in deal memos. But for transactional lawyers, negotiating indemnities is part of the life we have chosen.
As open source software has become integral to technology development, negotiating allocation of risk terms for IP infringement has become even more challenging. Open source software does not fit well into the traditional paradigm for allocating IP infringement risk, so a difficult negotiation topic has become even more difficult.
Clients, often frustrated with this process, always ask two questions: What is “market”? And what is reasonable? The answers are anecdotal at best, and usually not the same for both questions.
Clients repeatedly ask two questions: What is “market”? And what is reasonable?
This analysis is intended to help lawyers and business people understand how to analyze allocation of risk terms for third party open source software in commercial transactions. I refer to this issue as the “IP Question.” This analysis posits a negotiation between a vendor of a technology product that contains third party open source software, and a potential customer.
But the principles of the IP Question can also be applied to indemnities in other kinds of transactions, such as investments, acquisitions or joint development deals. My thesis is that vendors today are regularly asked to bear an unreasonable amount of liability due to a misunderstanding of the IP Question. While that may seem on face like a windfall to customers, it leads to unsustainable business agreements for vendors and customers alike.
Beyond Good and Evil
One of the hurdles in negotiating the IP Question is that negotiating parties tend to view it as a moral issue. In the moral conflict, the vendor views an indemnity against third party open source infringement as an unfair cost. The customer wants the vendor to be morally responsible for any harm that may befall the customer as a result of using the product. Regardless of which view you take, it is clear that reducing the IP Question to a moral question makes it a zero-sum game, and those are always difficult to resolve. While the moral view may be tempting, it is not very useful to get deals done.
The IP Question manifests when, at some point in the negotiation, the customer utters the phrase, “You need to stand up for your product!” And once this gauntlet has been thrown, and the IP Question has thus been reduced to a moral dilemma, the vendor and customer have no choice but to ham-handedly exercise their relative bargaining power tor resolve it, without either side engaging in the burden of critical thought. If the vendor is small and the customer is big, the customer wins. If the vendor is big and the customer is small, the vendor wins. But if you are an intrepid soul willing to engage in a more thoughtful approach, this analysis will help you “think outside the box” about the IP Question.
Indemnities are Costly
In fact, indemnities are not moral choices, but economic mechanisms to share risk. An infringement indemnity reduced to its purest form is an insurance contract. If I get hurt, you pay me. No one makes the mistake of thinking that insurance is free of charge, or a moral dilemma. But customers often expect vendors to bear broad indemnities for their products at no additional price.
Suppose that a third party insurer were willing to write a policy to indemnify the customer against third party open source intellectual property infringement. What would happen?
- There would be a money premium for the insurance
- The policy would contain limitations on coverage, and the premium would be priced accordingly
Unfortunately, in negotiations between vendors and customers on the IP Question, infringement indemnities are not negotiated in this way. If they were, the result would be simple:
- The customer would choose whether or not to pay extra to get the indemnity.
- The price of the indemnity would be calculated based on the type and amount of coverage the customer chose.
- The vendor and the customer could, if they chose, share the cost of the premium by negotiating a discount.
Indemnities are difficult to negotiate because they are never reduced to a priced deal point in this way. But why not, when doing this is so obviously sensible? Mainly because third party insurance for such risks is generally not available, making most vendors self-insured. Also, when parties negotiate the terms of contracts, they treat indemnities as undisclosed “legal” terms rather than essential deal terms — meaning they have merely kicked the can down the road for the lawyers to argue over.
But it does no good to lament this phenomenon. If the vendor and customer will not view the IP Question as a pure economic decision, then how do they actually come to agreement?
What is Not Relevant
First, let’s understand what the IP Question is not. There are two similar issues that arise when negotiating IT procurement deals, that should not be conflated with the IP Question.
Vendor Compliance. The first of these is the vendor’s open source license compliance. Because open source compliance claims are usually cast as copyright infringement claims, non-compliance is potentially an IP risk, but not a risk arising from third party actions. A vendor who supplies a customer with third party open source software must follow the license terms that facially apply to that software. That duty does not arise from the customer contract; it arises from the open source licenses. The customer contract may require the vendor to pay legal fees to defend a compliance claim against a customer, if that claim arises from a failure of the vendor to comply with the open source license terms when it delivers the product to the customer. But few vendors attempt to avoid responsibility for their own compliance. The IP Question, instead, is about third party open source software that is infringing of third party IP rights, even when the vendor has complied with the facially applied license. In other words, it is a risk the vendor cannot control.
For example, suppose a project is on GITHUB and bears an Apache 2.0 license, but the project contains code that was improperly contributed by a person without the right to make the contribution, or a cut-and-paste of third party code under GPL. That could result in copyright or trade secret claims against re-distributors or users of the project. It is one cause of the IP Question. Alternatively, a project may, unbeknownst to the project’s maintainers, infringe third party patent rights. That can result in claims of patent infringement against users of the code. But neither of these arise from malfeasance by the vendor, or by the customer.
Performance Warranties. The second is warranties or indemnities arising from the performance, or non-performance, of software. These are commercial warranties, and vendors often undertake them for third party open source elements, because the vendors are engaging in quality control, maintenance and support for their products that happen to include the third party open source elements. These are not IP claims at all, and they are not part of the IP Question.
Two Theories: Control and Internal Pricing
Now that we know the boundaries of the IP Question, and accepting the dismaying premise that we cannot simply price it out as insurance, we consider other ways to rationally allocate the risk it represents. In contracts, there are a couple of common theories as to why one party or another should bear risk: control and economic efficiency.
Control. Most lawyers focus on the control theory, under which the party who is best able to control the risk bears the liability. The cost of bearing that risk will tend to change the bearer’s behavior toward reducing the risk. This approach works well for risks like products liability. If a vendor manufactures a light switch, the vendor can make sure the light switch is properly built, will not short circuit and injure the user, and is properly tested for compatibility with local electrical standards. Moreover, the vendor can easily insure against products liability risk. So, it makes sense for the vendor to undertake that risk, because of its relatively high level of control.
The difference with the IP Question, of course, is that the vendor has almost no control over whether third party open source software infringes IP rights. So, if a vendor sells a smart light switch, it may include open source software that interfaces with a mobile app to set automatic on and off cycles via Bluetooth. The vendor probably gets that software from a third party open source project.
Customers will argue that the vendor can make a build-or-buy decision to address this risk. For example, instead of getting the Bluetooth control software from an open source project, the vendor could write its own software. Obviously, that would raise the development cost for the light switch, perhaps substantially, and the vendor would pass that cost on to the customer. And a sophisticated vendor will also point out that home-grown software is unlikely to be as reliable or secure as existing open source software. Moreover, if the light switch is intended to conform to a larger specification for IoT, like a Google Home or Apple Home system, writing home-grown routines will tend to make the device incompatible, or require the vendor to reinvent the wheel to figure out how to make it compatible. In such cases, if the vendor exercises control over developing the software, it will actually make the product worse. For these reasons, control is not a very useful theory to resolve the IP Question.
The flip side of the control argument is that risks better addressed by the customer should be borne by the customer. For example, if a customer elects to use the vendor’s product in high risk activities or in ways that violate the agreement between them, the customer would usually bear liability for those uses, because the customer can best control those decisions. While indemnities from customers to vendors are not as common as vice-versa, these allocations of risk sometimes take the form of express terms limiting the vendor’s liability in the contract, rather than allocating them to the customer. Risks that are not expressly allocated in contracts will fall to the parties in accordance with background law.
The control theory is also significantly out of alignment with the way open source infringement problems are solved in practice. If there is an IP problem with an open source project, particularly a project that is widely used, that problem is not solved by one vendor. If there were an IP problem with Linux, or Hadoop, or Firefox, that problem would be solved by the maintainers of the project — probably with plenty of help from the community. For a single vendor using that project to try to resolve it would be inefficient and counterproductive. At a minimum, it would cause the vendor to have to fork the code to engineer around the problem, defeating many of the benefits of including open source software in the product. In fact, licenses, like GPL3 actually limit the possibility of doing so, by requiring licensees to clear patent rights for everyone if they clear it for themselves via a license. So in sum, the vendor has no reasonable way to either prevent IP problems, or resolve them.
Allocation of Internal Risk Premium. The other theory or risk allocation is based on economic efficiency, and under this theory the vendor essentially loses the argument. In any deal, one party is usually paying and the other is getting paid. The party making the profit from the transaction can therefore more easily bear the cost of the indemnity, and take a reasonable reserve against its profits as self-insurance. The problem with this approach, of course, is that the vendor is likely to build this reserve into the product cost, thereby raising the product price. So, while a customer may always win this argument, it may be a Pyrrhic victory, and it places a high burden on the vendor to amortize an unknown risk.
These two approaches once worked fairly well for products developed by a single vendor — the cathedral rather than the bazaar. But in today’s landscape of heavy use of open source, this analysis is broken.
What is the Product?
Now, finally, we can find a path through the IP Question, but only if we leave the old ways behind. Contemporary IT systems are increasingly vertically dis-integrated. Once upon a time, you may have bought a computer and all of its software in one transaction from one vendor, but those days are ancient history. Yet vendors and customers are still negotiating the IP Question as if IT systems are monolithic technology of the 1980s. When the customer utters the battle cry, “Stand up for your product!”, the next question is: what exactly is the product?
Vendors in today’s computing world are, more than ever, systems integrators of layered technology solutions that largely consist of third party open source software and IP. Taking an extreme example, a company like Red Hat, which sells subscriptions to Linux distributions, is mostly selling quality control. The software is free, but the QC has a price. Most IT products today, of course, are not so starkly reliant on third party open source software, but even the most “proprietary” products today are not developed in a vertically integrated manner. That’s a good thing; it means that because of the wealth of open source software in existence today, vendors no longer need to reinvent the wheel. The job of vendors today is to select and integrate open source components with their own technology,add their own value in the form of unique product functionality, and provide quality control.
So, it makes sense to develop a more nuanced notion of what constitutes a vendor product. In the old, monolithic model, it is everything that the vendor delivers to the customer. But in the more nuanced model, the vendor delivers a substantial amount of third party software as a courtesy to the customer — much of which is not reasonably considered part of the vendor’s product.
Consider, for example, a software vendor contract in which the vendor, FOOBAR, Inc., delivers its application, FOOBAR for Linux. Once upon a time, that product would have been delivered on a diskette for the customer to install on its own Linux system. In those circumstances, the vendor would be asked to indemnify for IP infringement arising from the FOOBAR application, but not the Linux operating system. This business expectation is by no means unique to open source; if the product were FOOBAR for the Windows operating system, no customer would expect the vendor to indemnify for IP infringement arising from Windows.
The Product of Today
Today, FOOBAR will just as likely be delivered as part of a virtualized image or container that includes FOOBAR and Linux. It’s the same product, but packaged differently. It would, of course, be possible for the customer to get Linux on its own, free of charge. But that would only cause technical problems, because the vendor can better ensure that it is delivering compatible versions of Linux and FOOBAR. The delivery of the operating system is a convenience for both parties, but it doesn’t mean the operating system is part of the vendor’s product. This approach to delivery is possible only because the operating system is open source, so the vendor has the right to distribute it free of charge.
Why, then, does this result in a customer demanding infringement indemnities for the entire package? It is because the parties have failed to correctly define the vendor’s product in light of contemporary delivery practices. To correct this, those negotiating IT deals need to understand the difference between software and the environment in which it runs. Here is a very simple version of the difference:
The above is the “birds and bees” version of the modern software landscape. It is a very simple abstraction using only two computing layers — the application and the operating system, but it gets the point across. It’s no wonder the parties have trouble understanding the scope of the product. It is quite possible that that vendor will make a warranty about the performance of the entire container, including both the FOOBAR application and the operating system, when it is the vendor’s job to deliver a quality, integration solution. But this should not drive the definition of the product for the purpose of the IP Question.
The problem of defining the product is more stark when one considers a more realistic version of what vendors today actually deliver.
Any application sold today now rests on a formidable stack of open source software that makes it faster, and more fault-resistant, flexible, and powerful. This is the breakthrough in software that allow companies to run thousands or millions of applications in parallel, and maintain consistent data bases across them. That stack of software is sometimes referred to as the “LAMP stack” (Linux, Apache, MySQL and Python) but today, even LAMP is an oversimplified view, so let’s just call it the computing landscape.
If we want to think rationally about the IP Question, we need this more nuanced view of the technology landscape. And it isn’t hard to understand. Suppose you want to buy a boat. A boat dealer sells you the boat. You cannot sail the boat without an ocean. Is the boat dealer responsible for the ocean? Of course not. Today, the open source stack is like the ocean. It is the basis on which software products are developed. But it is not the vendor’s product.
That open source landscape now represents the backbone of the world’s information technology. So, when a customer demands that the vendor indemnify against IP infringement for this “product,” the customer is essentially making one vendor take responsibility for the entire technology landscape of the world.
Moreover, the customer probably already has all this software within its organization. It gets the open source stack from each of its vendors, as well as its own internal IT activity. The use of a product of one vendor rarely contributes more than a fraction of the marginal risk arising from the use of that software.
The UCC Gets this Right
There is a long-standing precedent for this view in the Uniform Commercial Code.
Unless otherwise agreed a seller who is a merchant regularly dealing in goods of the kind warrants that the goods shall be delivered free of the rightful claim of any third person by way of infringement or the like but a buyer who furnishes specifications to the seller must hold the seller harmless against any such claim which arises out of compliance with the specifications.UCC Section 2-312(3)
Moreover, although the UCC does not expressly provide for this, it has long been market practice in technology contracts to absolve the vendor of liability for products that infringe only because they meet enunciated industry standards that both parties elect to use. This makes sense in light of the UCC. A customer would usually specify that it wanted to buy products that meet industry standards, and vendors will conform to industry standards because their customers demand it.
If we view the open source landscape as part of the customer’s specifications, and not the vendor’s build-or-buy decision, then it makes more sense for the vendor to avoid liability for the landscape. Alternatively, we can view the software landscape as an industry standard, for which neither the vendor nor the customer should undertake liability on its own.
But even if we can agree that the vendor should not be responsible for the software landscape, that doesn’t release us from the IP Question entirely, because we still need to understand where the vendor product ends and the landscape begins.
The Build or Buy Decision
For negotiating parties to solve the IP Question, they need to separate the definition of the vendor product from the software landscape stack. Of course, if the parties have settled on the exact stack to be delivered, the product and the stack can be listed ad hoc in their agreement. But for those seeking a more general approach, one useful point of reference might be the definition of “Linux” promulgated by the Open Invention Network (OIN). OIN is a patent pool covering Linux, but its definition of Linux is broader than merely the Linux kernel, and includes many of the major components in the landscape stack. https://www.openinventionnetwork.com/joining-oin/linux-system/
Even after the parties make that distinction, there will be some third party open source embedded into in the vendor’s product that is not part of the landscape. Examples might include small routines to do generic calculations, or libraries that are included in the vendor product executable. The vendor should be far more likely to accept liability for this software, given it has made more granular decisions to use these elements in its products.
Some Provisions for Your Toolkit
Below are some suggested contract provisions to help differentiate the vendor product from its open source landscape.
“Open Source Computing Stack” means any open source software created by third parties that is so referenced in the specifications for the computing environment of the Product in the applicable purchase order, which software may include operating systems such as Linux, web server software such as the Apache web server, language engines such as Java, PHP, Python or PERL, and database software such as MySQL. The Open Source Computing Stack includes without limitation all software included in the definition of a Linux System promulgated by the Open Invention Network.
Vendor will have no liability under [reference indemnity provision] for infringement of third party intellectual property rights by the use of the Open Source Computing Stack; provided, however, that the foregoing sentence will not limit Vendor’s liability for compliance by Vendor with the terms and conditions of the open source licenses applicable to the Open Source Computing Stack.
Alternatively, focusing on open source software already in use by the customer — which is likely to include much of the open source computing stack:
Vendor will have no liability under [reference indemnity provision] for infringement of third party intellectual property rights by the use of any open source software made generally available by third parties that is in use by Customer prior to the delivery of the Vendor Product hereunder ; provided, however, that the foregoing sentence will not limit Vendor’s liability for compliance by Vendor with the terms and conditions of the open source licenses applicable to such open source software.