Red Hat and the Clone Wars
It’s been an exciting week for people who care about Linux distributions, FOSS licensing, FOSS distribution, FOSS business models, and the future of open source in general. Red Hat’s announcement that CentOS Stream will be the sole repo for public RHEL-related source code releases has generated a lot of chatter and exposed a lot of misconceptions about what the GPL requires and doesn’t.
Sadly for you, dear reader, I have a lot of thoughts on this topic and plan to tackle them in stages here on the blog. Going to start with some really common misconceptions / questions and answers as I understand them. Note that I am not a lawyer, and I no longer work at Red Hat. If you see something inaccurate, please feel free to comment and I’ll make any warranted corrections.
Doesn’t Red Hat have to release RHEL source code publicly?
No. The GPL requires distributors to provide source code to those entities it has provided the GPL’ed software to, but no one else is entitled to those changes, not even the original authors.
It also says that the recipient receives all the rights under the GPL that the distributor has, and the distributor can’t impose additional restrictions. So I can’t receive GPL’ed software, make changes, and then distribute those under a new license with additional restrictions. If I get and distribute GPLv2 software, I have to pass it on as GPLv2 software. (Or a later version of the GPL like GPLv3 if the original had the provision to choose a later version. But that’s a tangent here that’s not relevant…)
But isn’t Red Hat imposing a restriction on distribution with its subscription agreement?
I don’t believe so, no.
If I have a RHEL subscription and receive RHEL 8.5, for instance, I’m entitled to those sources. But as I understand the subscription agreement(s) from Red Hat, it has the option of terminating my subscription if I choose to distribute those sources in a way that Red Hat decides is outside its subscription agreement.
That might seem like a restriction of my rights, but it isn’t. I entered into an agreement with Red Hat to do business with them, which includes receiving GPL’ed software. If I sever that agreement, let’s say via non-payment, then I still have rights to distribute that GPL’ed software. Red Hat can’t take those rights away once they have conveyed them to me.
But Red Hat doesn’t have to keep distributing new software to me if I stop paying. And they don’t have to keep doing business with me if I start compiling that source code and shipping it as JZB Linux. Red Hat can’t claw back the (GPL’ed) sources from RHEL 8.5, but it can absolutely refuse to distribute RHEL 8.6 or later versions to me.
This is all part of IBM’s evil plan
Yes and no, but largely no in my opinion.
First of all, I don’t think that this is evil. It might be bad strategy and/or communication, but I don’t think this is evil for a number of reasons that I will cover in a future post.
Yes, this is ultimately on IBM. That is, IBM owns Red Hat now and it’s responsible for whatever Red Hat does for good or ill, because they choose the people who run Red Hat. Presumably if they are taking notice of the way CentOS has been changed since 2019, they’ve had ample time to weigh in with Red Hat leadership.
But I don’t believe there’s a master battle plan at IBM headquarters that dictates what Red Hat does or doesn’t do with CentOS and RHEL sources or RHEL development overall. I find it interesting that so many people simultaneously credit IBM with having a multi-decade plan to undermine open source, but suggest that IBM is hopeless at strategy.
My tenure at Red Hat overlapped with the IBM purchase and subsequent takeover of Red Hat. My sense is that IBM communicates with Red Hat’s leadership and has goals for Red Hat’s sales, and has ways it wants to collaborate between IBM and Red Hat offerings. But I don’t believe that IBM leadership is directly telling Red Hat what to do with CentOS.
Rather, I think that Red Hat’s leadership is looking at their sales goals and the market at large and trying to keep RHEL sales healthy and keep RHEL from being commoditized or sidelined.
Red Hat is betraying the community
Again, no. I know this will be an unpopular opinion, but I don’t believe this is so – assuming this is the end state of changes to CentOS and distribution of RHEL sources.
A lot of folks want a perfect RHEL clone that’s binary / bug-for-bug compatible and downloadable without any subscription friction. To that I say… it’s nice to want things, but if you’re not a customer and you’re not even willing to sign up for a no-cost subscription, you’re not entitled to it. You’re just not.
Red Hat is producing CentOS Stream and releasing sources, for free, to anyone, that have all the features and enhancements going into RHEL. Remember that they’re still releasing far more than they’re actually obligated to release. Some of RHEL’s software is not GPL’ed and requires no source from Red Hat at all. Some of RHEL is owned by Red Hat and they could hold that back, but don’t.
If you want to run something like RHEL, you can run Stream and have a fine Linux distribution. Competitors to Red Hat could do the work of testing Stream releases and re-brand them as their own, and get a huge amount of development and testing work for free.
Here’s what you don’t get, the one thing that Red Hat is effectively holding back now: The ability to easily recompile sources and claim bug-for-bug compatibility with RHEL, and the ability to say they are RHEL sources.
When a project compiles the sources from Stream there’s some uncertainty about whether it exactly matches the RHEL release. For most scenarios you don’t need certainty. You just need a distro that feels like RHEL and that can run a workload.
If you need certainty, it’s probably for business reasons. You need to be able to tell a vendor “I’m running this on RHEL 8.5.1” or whatever. The vendor needs to be able to test against specific releases. And, if you need those things, you should pay for them or at least sign up for the no-cost subscription that Red Hat provides for development purposes. If you’re unwilling to do those things (or unable) then you’re going to have to settle for almost-as-good.
True, it also hits community projects and developers who’ve been doing The Lord’s Work in writing free and open source goodies for RHEL. I don’t blame those folks for being annoyed (or worse) and for dropping RHEL support. This is a risk that Red Hat took. (That falls under a strategy discussion and that’ll come later…)
The unnamed players in this drama
But I wouldn’t assume Red Hat was ignorant of that risk, nor do I think “the community” was in Red Hat’s crosshairs here. Red Hat is in a really hard position here that doesn’t get enough consideration. They have the most popular paid Linux distribution with an enormous impact on the operating system market. Red Hat is also, even as part of IBM, still the underdog. Remember that Red Hat is in “coopitition” with companies like Microsoft, Amazon, Google, Oracle, VMware, and many more.
This post is already getting longish, so I’m going to save the strategy and rationale thoughts for another post. Suffice to say that I don’t think Red Hat has any interest in preventing small organizations and individuals or community developers from accessing RHEL or a RHEL clone.
Red Hat’s leadership is smart enough to know that it benefits from a certain amount of unpaid use, and it isn’t going to extract money from all users and use cases. Red Hat isn’t the RIAA here, they aren’t trying to shake down individuals or businesses/organizations with a few dozen employees. They may be caught in the crossfire, however, and the repercussions from that are going to be difficult to predict or control.
More to come, stay tuned.
Thank you for your thoughtful comments!
Moving to debian
Okay.
Lol go ahead
You are no Linux King
Well reasoned post, as usual. I agree with your assertions here. FOSS is having a moment now where they try to figure out how to stop the moneyed freeloaders (eg Oracle) while keeping what is great about FOSS.
Back in 2020 I thought there was a lot of whining from business who wanted RHEL without paying for RHEL and that’s just not sustainable. Let us hobbyists have Fedora or CentOS stream (or even over of 16 free RHEL licenses), but business should pay.
Well put.
Seems to me that Red Hat is in a panic because subscriptions aren’t as high as they want them to be.
Conversely, though, Red Hat staunchly continues to market exclusively to the “whales”, the Fortune 500 companies who can pay for huge contracts. Red Hat needs to open its market and start promoting themselves to the hobbyists, to schools, and so on. If people learn Red Hat instead of Ubuntu, then they’ll bring Red Hat to their workplaces and tell their bosses to buy support. That’s why Canonical is even in this market as a serious contender, because that’s been their model from the start.
Well said, JZB
As you write, a recipient of GPL’d sources have all of the rights which the distributor (Red Hat) received. However the rubric is that Red Hat is trying to prevent the recipients from redistributing those sources (or the object code after compilation), and GPL explicitly allows redistribution.
GPL does not prevent charging for a warranty. Perhaps Red Hat could forbid recipients to sell the software in turn by adding any warranty, by subscription and providing support. Fair enough – how hard is it to roll your own distro?
As you say, a no-charge (and non-commercial) subscription is available.
Wouldn’t that violate the GPL too?
Even if they give warranty, the code is still GPL and the customer is allowed to redistribute it.
GPL allows redistribution, which Red Hat doesn’t / won’t prevent. It may, however, choose not to do business with you in the future depending on what and how you redistribute that software.
We should be honest here that Red Hat isn’t trying to prevent access to the source code itself or people redistributing it. What Red Hat seems to be trying to prevent is wholesale copying of RHEL and pointing to it as “bug-for-bug” compatible.
We should also not pretend that Red Hat doesn’t make the sources widely available – even more than required. I would be hard-pressed to point to any actual harm done to the user or author of the GPL’ed code.
The only thing that Red Hat is really restricting in truth is the ability to claim wholesale RHEL compatibility. Since RHEL comprises far more than GPL’ed packages, I don’t see where Red Hat is in reality trodding on the rights conveyed by the GPL.
(I’d also note that if the GPL does prevent what Red Hat is doing, it has other remedies that will also achieve this – but are far worse for the larger ecosystem than pointing people to CentOS Stream sources.)
What things are you thinking of that’s in RHEL but not GPLed? Just stuff with other open licenses?
For the most part, yes. I think if you look at everything included in a RHEL install / the set of packages supported with RHEL, somewhere between 1/3 and 1/2 are under permissive licenses and not some kind of GNU license.
So another option for Red Hat would be to say “you know what, go nuts with the GPL’ed source and distribute it to your heart’s content. We just won’t release source for Apache HTTPD, Python, runc, xorg, etc. etc. etc. But have a lot of fun with the GPL’ed stuff.”
If source code become paid, then redhat is no more open source product.
Even the Free Software Foundation allows charging for products including GPL’ed software.
They can charge for the product, but they can’t charge for the source code – according to the GPL, it must be made available to recipients of the product for at most however much it costs to make it available.
I feel like you haven’t read the full posts.
Why does RH already have as many subscriptions as it does, if this was such a huge problem that needed addressing?
I think we are going to see a lot of orgs draw the conclusion that IBM/RH cannot be trusted not to screw things up a la Oracle, and move on. I run a large scientific computing org which is currently running Bright on RH on its primary cluster. When it comes time to design a new system there is no way in hell I am going with RH; I don’t care if I have to change package management systems entirely, I will find something else. Much of our HPC software stack is already containerized so I’m not super worried about changing the underlying infrastructure. What RH has shown me here is that they have 0 commitment to the community. Okay! Message received.
Red Hat is aggressivly monetizing the EOL of CentOS 7, which is used a lot. Especially in HPC environments with high node count, that’s why there was Scientific Linux 7 amongst CentOS 7.
It seems to be a sales KPI to convert as many former CentOS 7 installations into original RHEL. And RH is using everything in their arsenal to max that number. There are currently aggressiv RHEL discounts for large scientific HPC, VFX HPC compute clusters, etc. And at the same time RH is trying to cause Fear, Uncertainty and Doubt regarding the future of RHEL clones. First, RH took over CentOS to cripple and discontinue it. When that didn’t help and with one year to go with CentOS 7 EOL (the criticial time where orgs make a migration deciscion) there’s this move now that makes everyone question if RHEL clones like Rocky and Alma will survive until the planned EOL. It’s not certain. And that alone might make some orgs consider migrating from CentOS 7 to RHEL.
What will happen if RH doesn’t meet its planned sales numbers for new RHEL subscriptions from former large CentOS 7 users? Will leadership be forced to move the sources for CentOS Stream behind the “terms of services” wall?
And what time will new RHEL customers with their steep welcome discounts see their first price increase? (There’s always a price increase and audit at some point – we have all seen how Enterprise IT suppliers work).
Red Hat’s job is aggressively monetizing things. That’s just part and parcel of running a software vendor business in IT.
It’s also worth noting, Red Hat is constantly in competition for that money. We shouldn’t pretend that Red Hat is this rapacious company while everybody else is offering hugs and puppies and freebies.
I wouldn’t say Red Hat is trying to cause FUD, but I think it’s fair to say Red Hat wants to deny others the ability to call something “bug-for-bug compatible” with RHEL. And that seems fair to me – they earned all of the things that go with calling something RHEL 8.5 or whatever, and nobody else did.
A lot of questions about the future of clones and such, I agree. I have a lot more to write about on that topic…
If Red Hat follows the “land and expand” model *by force* the way that other vendors do, I think they’ll lose a lot of business and deserve to do so. But we’ll see what happens.
(“Land and expand” by *earning* more business is fine. “Hey, you like our Product A would you like to try Product B?” is fair game.)
I am a Red Hat guy since 1997 or so, downloading many 1.44 MB disks from the internet.
Worked with IBM in early 2000s when they were the cool company bringing Linux to the Mainframe and defending Linux against SCO’s claims.
I’ve seen Red Hat make fun of SuSE for not all packages being open source.
When I started replacing my RHEL 7 systems with AlmaLinux 9: Whats the first thing you do after installation? Activate EPEL and other repositories which make it usable. I am stuck to Red Hat like distributions because I can use them blind, even with systemd (good) and journald (bad).
If you really need support from Red Hat, nothing was fixed: Thunderbird with GPG change was ignored because of an additional crypto-library. Packages stay on old broken levels (dovecot) even if the fixes are available upstream. And still no working inplace upgrade for enterprise distributions.
I really tried with Red Hat and IBM, but finally it’s over.
The rest of my RHEL7 system will go to Debian. I’m annoyed that I didn’t already do this with the killing of CentOS.
I’ve seen similar situations with several (smaller) shops that were former Red Had customers; packages they wanted paid support for were no longer part of the official distribution.
I wonder what percentage of subscriptions have historically been for software like dovecot or desktop applications, and what kind of software Red Hat projects their main customer base will want support for in the future.
When will JZB Linux be available? In all seriousness, this is very well written and a great explanation of what is happening. If we’ve learned anything after the Red Hat announcement, it’s that a lot of people don’t understand the GPL (myself included).
I’m waiting on VC funding. So far I’m up to $2.52 and a couple of tokens for Skeeball.
Thanks for this series Joe.
Do you know if anyone would have the history and insight for a supplemental post on EPEL and other prominent community repositories, and how they might impact the future of RHEL, Stream and rebuilds alike?
Somebody might, that’s an interesting angle.
From my perspective as someone working for a software company where exactly one customer insists on using RHEL and everything else we run is Debian-based RHEL already was a pain in my ass for many years that I would love to charge extra to support because anything I learn about RHEL is worthless knowledge for any future projects due to the very outdated nature of the versions in use there. Being at least able to use Centos or Alma or Rocky in a few places like our build or internal test servers took some of that pain away and third party repositories at least somewhat alleviated the pain that is the tiny number of packages RHEL includes. I expect using RHEL to become even more painful in the future and the quality will further decline compared to our Debian-based systems and take up even more of our time.
I hope RHEL dies as quickly as possible after this move and that I will never have to work with anything made by RedHat again. I will certainly tailor any future advice to customers to achieve that and it is largely RedHat’s own fault.