Though I have loved my time on #Fosstodon, it's time for me to find a new home on the #Fediverse where I can engage in political discourse. See *waves hands* this.
Any suggestions? I'm thinking about #MastodonSocial or #Hachyderm.
Recent searches
Search options
#Hachyderm
2 posts2 participants2 posts today
Hello #FediHire and #HachyJobs / #HachyHire ! I'd like to take a moment to talk about how awesome Emelia is!
Both #Hachyderm and #Nivenly benefit from Emelia's expertise, and so do many on the Fediverse due to her Trust & Safety work. This means if you're on the Fedi, doubly so if you're on a Mastodon instance, you're benefitting from her work.
Emelia takes donations as her work centers the community: support.thisismissem.social
Emelia also takes paid contracting work, which I can confirm she does a brilliant job with. You can reach out to her directly about this.
Her post about this:
Hachyderm.ioEmelia 👸🏻 (@thisismissem@hachyderm.io)I've currently been doing a tonne of work on Doorkeeper, a Mastodon dependency for OAuth, and currently that work is mostly uncompensated.
I'm also doing a reasonable amount of work on the standards side (W3C & IETF), which is also uncompensated.
I'm not making ends meet at the moment, so if you can spare €5-10 a month, please do consider supporting my work on Mastodon, Doorkeeper, standards, and the wider Fediverse.
I'm currently running a deficit of about €1000-1600 per month, which is completely unsustainable.
(there's also one-time, but ongoing support is what enables me to do this important work)
https://support.thisismissem.social
#mastodon #fediverse
Hey #hachyderm, are we just going to sit around while cybersecurity companies are cowed into submission? Is this how it all really goes down? President takes away clearance and everyone is just quiet about it? I hope you realize that if we don’t hang together we will hang apart. #Infosec as an institution is being attacked the same way as Education, Science, Health, and the Law are under assault. Guys everything that is good and true is under attack by literal evil doers this is your time get off your asses get to work we have a big job ahead
Hachyderm is an amazing server. The mods are the best, and the infrastructure team manages the infrastructure better than most companies! If your account is on #hachyderm I highly encourage to donate to help the team keep doing what they’re doing.
https://mastodon.social/@flexghost/114292526374604847
Hello, hachyderm! we've been working hard on building up our ansible runbooks and improving hachyderm's overall resilience. Recently, we've been focusing on is database resilience.
We're getting close to retiring our original database server (finally!) and preparing to move to a fully ansible-managed set of databases servers, primary and replica on new hardware. We'll send another announcement when we do the cut over. The team has done excellent work to make this highly automated, quick, and painless! 
Done:
author ansible roles for managing postgresql, pgbackrest (backups), pgbouncer, and primary/replica failover decide to continue with pgbouncer and *not* use pgcat rotate database passwords order new replica database hardware order new future primary database hardware
To do soon:
rebuild replica database with ansible scripts prepare primary database with ansible scripts start replicating to new database replica cut over to new database server 
We're also planning on open-sourcing our ansible roles in the coming weeks - just a little housekeeping & tidying up before we do!
Hello again Hachyderm! 
As promised, here is the announcement blog post regarding the Pixelfed vulnerability, outlining the impacts to our community.
Our blog post outlines the data behind the decision and steps for admins to reach out to us to refederate after they patched the vulnerability.
Although the number of following relationships was low, the number of instances was high relative to that number. Basically: we're still in the process of reaching out to impacted instances as of this publication. Thank you for your patience!
https://community.hachyderm.io/blog/2025/04/03/pixelfed-vulnerability-and-impacts-to-federation/
As a reminder: we're happy to answer questions and engage with the Fediverse community. We're here to help 
Hachyderm Community · Pixelfed Vulnerability and Impacts to FederationPixelfed CVE-2025-30741
I think the #hachyderm admins take their infra more seriously than some companies do. Well done!
https://hachyderm.io/@esk/114242919601414036
Hachyderm.ioEsk 🐌⚡💜 (@esk@hachyderm.io)Attached: 1 image
howdy, #hachyderm!
over the last week or so, we've been preparing to move hachy's #DNS zones from #AWS route 53 to bunny DNS.
since this could be a pretty scary thing -- going from one geo-DNS provider to another -- we want to make sure *before* we move that records are resolving in a reasonable way across the globe.
to help us to do this, we've started a small, lightweight tool that we can deploy to a provider like bunny's magic containers to quickly get DNS resolution info from multiple geographic regions quickly. we then write this data to a backend S3 bucket, at which point we can use a tool like #duckdb to analyze the results and find records we need to tweak to improve performance. all *before* we make the change.
then, after we've flipped the switch and while DNS is propagating -- :blobfoxscared: -- we can watch in real-time as different servers begin flipping over to the new provider.
we named the tool hachyboop and it's available publicly --> https://github.com/hachyderm/hachyboop
please keep in mind that it's early in the booper's life, and there's a lot we can do, including cleaning up my hacky code. :blobfoxlaughsweat:
attached is an example of a quick run across 17 regions for a few minutes. the data is spread across multiple files but duckdb makes it quite easy for us to query everything like it's one table.
#sre #devops #mastodon #infrastructure
howdy, #hachyderm!
over the last week or so, we've been preparing to move hachy's #DNS zones from #AWS route 53 to bunny DNS.
since this could be a pretty scary thing -- going from one geo-DNS provider to another -- we want to make sure *before* we move that records are resolving in a reasonable way across the globe.
to help us to do this, we've started a small, lightweight tool that we can deploy to a provider like bunny's magic containers to quickly get DNS resolution info from multiple geographic regions quickly. we then write this data to a backend S3 bucket, at which point we can use a tool like #duckdb to analyze the results and find records we need to tweak to improve performance. all *before* we make the change.
then, after we've flipped the switch and while DNS is propagating -- 
-- we can watch in real-time as different servers begin flipping over to the new provider.
we named the tool hachyboop and it's available publicly --> https://github.com/hachyderm/hachyboop
please keep in mind that it's early in the booper's life, and there's a lot we can do, including cleaning up my hacky code. 
attached is an example of a quick run across 17 regions for a few minutes. the data is spread across multiple files but duckdb makes it quite easy for us to query everything like it's one table.
What a great work of moderation #hachyderm admins do.
Well, I'm back. #introduction
#genderqueer 
dà-anam 
storyteller/beat poet making their way in the greasy underbelly of the 'net. #filk, #scifi, and thinking about #aviation are pastimes. #activism , #GroenLinks , #ClimateChange , #TransrightsAreHumanrights , stomping #antisemitism; #TikkunOlam means *everyone*.
I am Universalist with strong Old Celtic leanings, Jewish-adjacent, Baha'i-adjacent, my bestie is Christian, and my circle is generally eclectic, including at least one athiest. None of them try to *impose* their views.
I am also a former professional and current amateur journalist. Sources get cited, mistakes corrected; I invite you to do likewise.
Love wins, hate gets used for a DANCE FLOOR. Can't stop the 
AND MY INTRO FITS IN ONE! Thanks, #hachyderm !! It's good to be back.
howdy, folks - it's been a bit since our last #hachyderm infra check in.
stuff in motion:
- ditching #terraform cloud & tf for #opentofu and #atlantis. we are just about to import our dev environment and put it through its paces.
- bringing #postgresql under ansible management. the team has been doing awesome work, and we've started to spin up dev nodes using the new playbooks. soon: production!
- moving #DNS zones away from AWS route 53. we chose bunny DNS as our provider and have been doing basic tests in dev. we'll likely prep our records for production this week with a plan for a cutover in one of the coming weekends.
and if you filled out our volunteer form and haven't heard from me in a bit - you're still on the list. we'll onboard a new batch of folks in the next couple of weeks.
JFC, what a #Hachyderm shitshow...
Replied to Anders Eknert
@hachyderm, have you reviewed this case involving @anderseknert? If not, why‽ Do you wish for #hachyderm to become known for unreasonable judgements?
I didn't expect my post about Swedish Television mistakenly labeling Vladimir Putin as "president USA" yesterday to blow up the way it did. What I *really* didn't expect was #Hachyderm's admins deleting my post just as it was approaching a thousand boosts. This because someone allegedly reported it as "misinformation", and that what I saw on TV "does not appear to have actually happened"... which ironically is about as Putinesque a statement can be.
So how would you fact check this? A few people commenting on the post had already done that, the way people normally do — by asking for the source. And they got it. I even boosted the link to SVT Play, where the broadcast can be streamed by anyone. It's still there https://www.svtplay.se/video/jAM5B9o/aktuellt/tor-13-mar-21-00?position=135
The Hachyderm "fact checking" process doesn't ask for a source though. In fact, they didn't ask me for anything. Since there were no news headlines mentioning this blunder, it must not have happened.
I appreciate the lecture for me to "fact check my posts before posting", but the assumption that I'd want to post anything on this instance again could certainly need some fact checking.
To the rest of you, enjoy this last screenshot of Putin definitely not getting labeled as "president USA" on SVT2 News on Swedish Television. And for those of you who can't get enough of things that never happened, I'll post a video in a comment below too.
I was looking for an Ansible role to manage tailscale certificates for the Hachyderm infrastructure when I found this "collection" https://robertdebock.nl/ansible.html.
Unfortunately, Robert doesn't seem to be on Mastodon so I can't mention him. Anyway, I would like to thank him for his INSANE contributions to Ansible. Look at all the playbooks and roles. The question is what software is not covered here?
R. de BockAnsibleSimple and compatible on many platforms.
Continued thread
The other root cause of our outage last night was a misconfiguration in how we handled requests through to mastodon-web (rails), which resulted in rack_attack believe all requests were coming from 127.0.0.1, instead of their actual IP address.
Essentially our infrastructure looks like this
| CDN | web server
-> nginx —> nginx —> mastodon-web (rails/puma)
We were correctly setting X-Real-IP, X-Forwarded-For and X-Forwarded-Proto headers on nginx at CDN edge, however when the request went through nginx on the web servers, these headers weren't passed through correctly all the time.
We've since deployed changes to both our nginx configuration and our mastodon configuration (TRUSTED_PROXY_IP’s) to ensure that requests are correctly identified with the requestor's IP address, and rate limited accordingly.
This should reduce the rate of “Rate Limit Exceeded" errors some people may have been seeing.
Most folks running mastodon are only using a single nginx server in the request pipeline, so this is a very "us" problem, as we run our own CDN powered by nginx.
Heads up Mastodon operators: libvips 8.16.0 doesn't work with ruby-vips 2.2.2, which is currently used by Mastodon.
*** watch out if your system automatically updates libvips to 8.16.0 ***
This was one of the root causes of our media processing outage last night. Essentially when we were trying to process media, the conversion between different file sizes and metadata stripping was erroring out do to an incompatibility between ruby-vips 2.2.2 and libvips 8.16.0, resulting in the error:
```
Error while optimizing <name>: VipsForeignLoad: "/tmp/<name>.<ext>“ is not a known file format
```
This occurred for jpg, webp, and several other file formats, despite `vips -l foreign` showing everything setup correctly.
ruby-vips 2.2.3 is known to fix the issue, and will be used in mastodon 4.3.4 when it is released. Until then either don't update libvips or switch MASTODON_USE_LIBVIPS to false until mastodon 4.3.4 (unless you're comfortable modifying the Gemfile to use the newer ruby-vips version)
What we still have absolutely no explanation for though is that libvips 8.16.0 was installed on hachyderm systems on January 20th 2025, but we only started seeing issues yesterday after running regular system updates wherein libvips was not touched.
Ruby and gems were reinstalled yesterday due to an update of libicu, but libicu was also updated on Jan 20th 2025 so we did the same reinstall steps on that date too.
We didn't receive any error reports from people using hachyderm of not being able to upload media or media failing to load for them.
Volunteering for Hachyderm is a breath of fresh air to me.
On the technical side, creating and adding Ansible roles for PostgreSQL (including PgBouncer and pgBackRest) is fun. I'm learning a lot, even with more than 10 years of experience.
On the human side, I'm honoured to be part of such an amazing worldwide team sharing the same values.
Apologies hachydermians for the outages tonight and the issue with rate limiting and images not loading.
We discovered several issues that needed resolutions, and everything appears to be working fine now. There'll be a write up sometime in the future.
Only negative? We've had to disable image processing with libvips, as it was failing to process media, so we're back to the imagemagik or whatever pre-libvips.
Continued thread
so, #hachyderm: funny story, turns out we *did* break image processing for a bit.
we aren't entirely sure what happened, but it looks like the latest version of arch linux broke $something with libvips. this was causing mastodon to blow up on any image processing - attaching images, pulling remote images, creating thumbnails.
as a temporary fix, we failed back to use image magick.
additional hypothesis, but less sure about this one: we think because the image processing was failing, it was then triggering clients to re-request more frequently, which then triggered rate limiting in our nginx/mastodon-web.
full write up coming soon (tm)!