Recent searches
Search options
I got my delivery of my new #kubernetes nodes (#minisforum #ms01). Now I'm waiting for ram and cables. Rather than decide between using the 10Gbit SFP+ ports and the USB-4 ports for the ring network (for #Ceph), I'm just going to set up both. That should give me 30Gbit between each node without using a switch.
Now I need to decide which OS I'll use. Stick with #RHEL 9, or change to #RockyLinux or #Fedora?
This is what the new cluster looks like cabled up. Sexy, eh?
For the OS, I decided to go with RHEL 9.5. There are reasons. Red Hat really made some super dumb moves around CentOS Stream, but I don't think they are evil.
Setting up the routing for the ring topology network using #FRR seems to work. I'm getting about 26.6Gbit/sec across the Thunderbolt 4 interfaces. That is using ipv6. ipv4 is not quite working on those for me. And that is my blocker for #rookceph.
For #rookceph, I first tried to use #multus for the cluster networking. While I got it to work on my test pods, I could not get it to work through the cluster.yaml config for Rook. The issue being that it needs a single interface to tie into and my FRR ring topology has two NICs per node. I've given up on that and switched to provider.host while specifying the internal cluster network. My hope is that will work after I get my ring net working with ipv4. Uncharted territory.
I also ended up replacing my Ubiquiti routers with #mikrotik. I want working load balancing in this cluster, so I have tried out #metallb. However, I could not get it to work in L2 operating mode with either #calico or #cilium. So, I've overhauled my home network to support BGP in order to have more options. This project keeps getting bigger.
@makuharigaijin nice! What do you use as network switch for the 10Gbps link?
@madduci
I'm not going to use a switch at all for the 10Gbit links. Port A on k1 will connect directly to port B on k2. Port A on k2 to port B on k3. Port A on k3 to port B on k1.
Then I'll do the same with the USB-4 ports for a second ring network.
I'll use the 2.5 Gbit port(s) for uplink to the house network.
@makuharigaijin interesting, thanks for sharing
@makuharigaijin i would suggest Talos OS. Personally I love it
arc) @oliverwiegers @makuharigaijin have heard similar
@makuharigaijin Years ago i left Fedora for their short lifecycle, and went to CentOS with a 10 year lifecycle. Now CentOS became beta stream for RHEL, i went to Rocky. The only question is how long Rocky can uphold the "bug for bug compatibility" with RHEL. I need that because a software i use advertises only RHEL compatibility, apart from some completely different distros. But if you don't have such dependency, you could also choose AlmaLinux. They chose to be only binary compatible with RHEL.
@makuharigaijin jfc, what the hell boxes are those even with SFF and SFP 10GB?
@arichtman
Crazy, right? They are mini work stations. The IO and small footprint is wild.
https://www.servethehome.com/minisforum-ms-01-review-the-10gbe-with-pcie-slot-mini-pc-intel/
@makuharigaijin@fosstodon.org I'd use either rocky/alma or ubuntu (this is what I switched back to after the fiasco)
@makuharigaijin 26Gbps. Nice. Measured with iperf?
@benoit Yeah, iperf3