Recent searches
Search options
@maxamillion@fosstodon.orgI get the feeling too many people are sleeping on CentOS Stream because of how the CentOS #Linux EOL thing went down.
I mean sure, the messaging wasn't great but it happened and it is what it is. Don't let that detract from the greatness of the #CentOS Stream project in its own right and as a collaboration point for all the #RHEL based distros.
The gravity seems lost on most that for the first time ever the RHEL development process happens in the open. That's amazing. I love it. #community
@maxamillion I think they're sleeping on CentOS stream because of the legitimate concern that the CentOS EOL debacle represents IBM's true colors when it comes to interacting with the Linux community. When people show you who they are, believe them.
@hal_pomeranz the irony is that IBM had absolutely nothing to do with this but everyone loves to blame them. 
@maxamillion I think it’s not (only) that. It’s simply now a wrong positioning which does not fulfill the needs it used to
@herrorange that's entirely possible, I'm curious what needs it doesn't deliver on anymore other than "I want RHEL but don't want to pay to support the developers who make it" 
@maxamillion there's still the high risk that CentOS stream versions just completely disappear if a new RHEL release comes out. Nobody wants to risk their business on an OS like that, otherwise they might as well run on Fedora Server releases with in place upgrade each six months.
@ragectl or, here's a wild idea, support the operating system your business relies on or pay someone who does.
@maxamillion that's a pivot from 'why did people stop using this free thing' to 'people should pay for thing'? I mean sure companies that can should support RHEL. But that won't be possible for a lot of CentOS users.
@ragectl I'm not saying there isn't a real world situation in which not paying for software is advantageous from certain perspectives, but my point still stands. Was the "I need one of these that costs no money" the only thing? If so, I feel like Stream still delivers but technically now with fixes even faster. Maybe I'm missing something though.
@maxamillion the perceived risk has changed greatly caused by the uncertainty of all the changes that created Stream. And users will rightly be cautious of coming back, now AlmaLinux and Rocky filled the mind share CentOS used to have
@ragectl that's fair. The artifact itself is different.
@maxamillion I think Red Hat did a solid job with allowing a certain number of free subscriptions for developers etc. But I see it mainly as being the risk factor overtook any technical or financial factors.
@ragectl honestly, I would have liked to see more. Something like 50 subscriptions for free or something and better articulation of the plan to offer effectively unlimited free RHEL to open source projects, but I think we can all agree the roll out of the changes was not ideal. 
@maxamillion FWIW we moved to RHEL because we got our business case approved for it, otherwise we'd probably have waited for AlmaLinux or Rocky too
@ragectl I'm a Red Hatter so RHEL is always first on my list for all the reasons (it was first on my list long before I worked here, I used to pay for personal subscriptions of RHEL for many years prior to employment because I believe in voting with my wallet). I think Alma is a solid choice, I like their governance model. I'm pretty sure Rocky is a slow burning grift by a dude who likes to claim he was a CentOS Founder and who's startup has complete control over the project. YMMV.
@maxamillion yeah I have been pointing a couple of people towards AlmaLinux
@ragectl @maxamillion People using those distros are slowly rediscovering the flaws that classic CentOS had. Namely, they can't fix bugs or accept contributions. CentOS finally fixed these flaws by moving slightly upstream of RHEL. Not everyone has to like it, but they need to at least be honest with themselves about the limitations of the rebuilds.
@carlwgeorge @maxamillion oh no doubt some customers will run into the same old types of complaints. I was just stating my view on why users didn't jump straight back to CentOS.
Personally I think interesting work is happening with FCOS, and importantly in Silverblue. I am wanting to see that gets well supported by Red Hat and not left to wither
@ragectl @maxamillion There is no risk of that. Classic CentOS was a literal afterthought for RHEL. CentOS Stream is critical for creating and developing RHEL. It's the major version that RHEL minor versions branch off from every six months.
@maxamillion At the time of the EOL event, I moved home workloads to RHEL with the developer plan, or more on the edge with Fedora.
At the time it felt like CentOS would have a faster version cadence without a way to do in place upgrades. With Fedora, I've had great success with the dnf system upgrade plugin. RHEL you don't have to worry about it for much longer and then do the lift or LEAPP.
@maxamillion All in all, I guess if I'm making something for work, I can build on the real deal with RHEL, and if I'm looking for the fun stuff on the cutting edge, Fedora has it covered.
The Fedora community also seems to have an accessible presence online with ways to onboard and start scratching an itch. I haven't seen much in the way of community outreach from the CentOS project in the same way, but that could just be the tech sphere I find myself in.
@garret Fedora definitely has a much larger community presence and a more open governance structure. I've been a Fedora contributor for over 15 years and have a lot of love for Fedora. 
@garret I am actually very curious what will happen with in-place upgrades for Stream 9 -> 10 in the future. That's a good point.
@maxamillion While I agree, it's definitely not helping things that CentOS Stream still goes EOL well before RHEL, AlmaLinux, etc.
@maxamillion That doesn't mean it isn't good or useful, but that it takes more intentional consideration in deploying it than CentOS Linux did.
@maxamillion This is unfortunately not trivial. I've had some trouble with filing Github issues against CentOS Stream because the upstream only now "supports Rocky", etc. Having to cut through the FUD around it gets tiring, honestly, but CS is still really nice for having fast track container patches and as a CI platform. I mean, I'm still using it very much, but I understand why there's preference for the rebuilds.
@maxamillion I would also be quick to point out that CS is part of the success story for AlmaLinux as they build against CS repos ahead of their own releases, which is part of why they're so quick to have stuff out when RHEL releases (especially compared to Rocky and Oracle who tend to have significant more lag).
@maxamillion I don't have the bandwidth for this right now, but sometime I'd be curious to test the differences between CS and AlmaLinux with the testing repo enabled.
@vwbusguy biggest difference is you can actually contribute to CS. Alma is a rebuild and if you need to fix something or get a feature into Alma then you submit a merge request to CS and wait until it lands in Alma downstream.
@maxamillion Oh sure, I understand that as someone who is into contributing to open source and building stuff. It's a harder sell in a place that's just looking for a platform to run their production stuff. No one wants to risk having to explain that prod is down because of a patch in CS that broke something and the vendor no longer "supports CentOS" (even if it's eventually going to be a problem for the vendor in the next RHEL release).
@vwbusguy @maxamillion Translates to “we don’t want to pay, but we want assurances and garantuees for free and will blame others happily” #SarcasmButOnlyHalf
@jwildeboer @vwbusguy seems that way
@maxamillion @jwildeboer That's one way to look at it. I say this as some someone who maintains CentOS Stream, RHEL, and the rebuilds in my dayjob. Frankly, the rebuilds mean you're likely to get vendor support and don't have to deal with rhsm (or acquiring contacts if they don't already exist). Often those decisions get made more out of convenience than social responsibility or even brand loyalty.
@maxamillion @jwildeboer Personally, the biggest reasons *I* might choose AlmaLinux over CentOS Stream or RHEL for something:
* Longer lifespan
* Desire less frequent updates
* Don't feel like messing with subscription-manager
At least with AlmaLinux, we can also contribute with testing and such for ELevate, which also helps upstream (and other downstreams).
@maxamillion @jwildeboer The CentOS SIG projects like Hyperscale are a major reason I might pick CentOS Stream. None of the others have btrfs options at install, for example.
I've also found CentOS Stream to be really nice for standalone container hosts since the container runtime (podman, crun, etc) patches land sooner.
@vwbusguy @maxamillion @jwildeboer Yep, subscription manager is really not great. Number 1 reason to avoid RHEL.
While these days with OCP4 things are all fine, in OCP3 times with subscribed RHEL hosts, subscription manager was the main reason for outages. Don't get me wrong, not frequent, but if hosts didn't come up, then 8/10 times it was a problem with rhsm.
@sheogorath @maxamillion @jwildeboer So much this. Having a mixed environment between traditional RHEL and ose3 hosts somehow caused a terrible mess of things as entitlements kept getting misplaced. I had an Ansible playbook run constantly to fsck rhsm. And if a box was offline long enough to not trust rhsm keys anymore, it was difficult to recover.
@vwbusguy @sheogorath @jwildeboer OSE3 was a bit of a mess. OCP4 is much better
@maxamillion @sheogorath @jwildeboer OCP4 has been its own awkward mess. It traded crazy complex Ansible stuff for complex terraform and operator abstracted stuff, which is great until it needs lower level debugging and then it's generally more complicated, especially with the low level stuff needing to go through ignition/ostree/manifests, etc. Still, I very much like that it plays better with the rest of the k8s world outside of RH than ose3 did.
@vwbusguy @sheogorath @jwildeboer oh yeah, debugging operators is insanity
@maxamillion @jwildeboer (I'm aware that ELevate is leapp based, but I've actually had success with ELevate where I've had straight leapp royally fail on RHEL.)
@maxamillion @jwildeboer Oh yeah, and the rebuilds post security errata, which you don't get with CentOS Stream.
@vwbusguy @jwildeboer I'm curious where they get that errata from if they're rebuilding from CS.
@maxamillion @vwbusguy @jwildeboer They aren't rebuilding from CS. They're rebuilding from exported RHEL sources. Their errata is copied from RHEL with text search and replace, no validation.
@carlwgeorge @vwbusguy @jwildeboer isn't redistribution of errata a violation of the data usage terms? Or did I imagine that?
@maxamillion @vwbusguy @jwildeboer IIRC you can view the errata without a subscription, thus no usage terms. But I don't know for sure.
@maxamillion @carlwgeorge @jwildeboer Yeah, I believe it's public. If it's security errata, it's going to be tied to a CVE anyway, so it should be public in some manner anyway unless it's embargoed.
@carlwgeorge @maxamillion @jwildeboer Correct. To clarify, I didn't mean to suggest that official AlmaLinux releases are from CS and not RHEL sources, but rather that they pull from CS sources for their own internal development and testing ahead of RHEL based releases. AFAIK, the errata is copied from RHEL, but it's still less clear about what CVEs are patched when in CS repos unless it's in the rpm changelog and there's something that can scrape and aggregate that info.
@vwbusguy @maxamillion @jwildeboer The "desire less frequent updates" point doesn't make sense to me. You get updates when you want to apply them. If you want to update weekly, monthly, quarterly, etc., then that's how often you get updates. Nothing is forced on you.
The difference is that the updates are made available regularly as they pass QA, rather than the staircase model (many updates at minor release time, few updates in between).
@vwbusguy @maxamillion It sounds like you're describing a shop that applies updates in prod without first testing them in non-prod. How do they explain when a regression or change in RHEL brings down prod? It's not really different. Testing updates before deploying them in prod is a best practice no matter what OS you're using.
@carlwgeorge @vwbusguy yes, this.
@maxamillion @carlwgeorge Everyone has a test environment. Some are fortunate enough to have one that isn't their production.
@vwbusguy @maxamillion @carlwgeorge for everyone else, there's rawhide.
@vathpela @maxamillion @carlwgeorge You mean I shouldn't be running production on ELN?! 
@vwbusguy @maxamillion It's going to be very close, because CentOS Stream is very close to RHEL itself. When I've measured it, it's typically 90-95% the same software versions depending on where in the cycle you are.
@carlwgeorge @maxamillion Sometimes things that are in CS that aren't in RHEL end up in the dev/testing repo for AlmaLinux. For example, when OpenLDAP got abruptly pulled and reinstated in RHEL, it became available in CS first and was in the Alma testing repos ahead of the next RHEL release that added it back in.
@vwbusguy @maxamillion I don't think that's accurate. Have they said that anywhere?
@carlwgeorge @maxamillion It's something I've observed directly in their git repos. For example, leading up to the RHEL9 release, many of their repositories had CS9 branches. Same is true for their cloud images that had CentOS Stream 9 as the base image before AlmaLinux 9 was released and the repo rebased to it.