Table of Contents
I want to write this for now a little while, even though I don’t like sharing much of me online. I’m the admin of a fediverse instance, udongein.xyz, a bit against my will. Let’s explain this.
I was used to selfhost a single user instance until the day I want to FOSDEM 2020, the morning I took train for Bruxelles my server crashed and had no way to reset/reboot it (and because of Covid, it got reset only 3 months later). A few weeks earlier I registered on a friend’s instance because we are both appreciator of Udongein character (from Touhou). So, with my server crashed and a bit panicked, a day before meeting several (wonderful!) people from Fediverse, I decided to make my udongein.xyz account my main. Re-followed bunch of accounts during event, and got bunch of mutuals too, great. Since my instance wasn’t planned to be up in a forseable future, I decided to make this account my main and it’s still at the moment I write these lines.
Times Change #
Fast forward on 2021, I become the most active user here (got 42 posts per day on average at a point).
Fast forward again on 2022, geo-political issue: admin can’t pay for server because international sanctions decided citizens from a certain country are forbidden to doing payments; I lend a VM so instance don’t disappears with paymet termination.
yet another time on late 2022, and admin is forced to leave their home country to, at brightest not be forced enroll in army, at worst, passing away.
I have admin access to server (because it’s running on my hardware), but the “real” admin vanished (but we know they are alive). After couple attempts I manage having a conversation and understand the situation: a burn-out.
A New Leader #
So, I guess I’m the new admin, right? What does this mean? I’ve never done that by past. What should I do? Why VM is on this distro I dislike?
I won’t lie, I was really clueless, and I’m still much clueless. We aren’t much active users, I’d say dozen at most.
Dark Days #
The Database #
Pleroma is sadly known for it’s database issues, the “Pleroma rot”, and we were no exception. I don’t know how, but we managed surviving it, that’s great.
I tried moving database on other servers through various methods, they all miserably failed.
- PostgreSQL logical replication: leading to insane index errors
- Dump and restore: insanely slow to rebuild indexes (forecasted a month of building these on an ARM board (Honeycomb), brought instance down for two weeks)
I have still a few tricks up in my sleeve, but failed attempts depressed me and isn’t giving bright future days. My current main issue is having a giant DB size, ~120GB1, for a tiny userbase.
Broken API #
For an unexplainable reason the API became suddenly broken. Small queries were working fine, but doing HTTP Post a big too big was halting and broken.
After painfully troubleshooting with several friends (thank you so much to everybody involved!), we discovered that was an MTU issue… Once lowered, everything was working as good as it was.
Two weeks of suffering, and migrated off users.
Why MTU issue? I have no idea, probably a Wireguard legacy that’s doing weird with networking stack, I haven’t installed this server nor really managed it.
The Uploads #
I olse had failure because of my own stupidity, when trying to move uploads on S3 bucket. It ended in having all uploads deleted from server. 60GB gone in a couple seconds.
My sincere apologies to users, I should have done backups.
Now, What #
That’s an excellent question! Since I haven’t installed the VM and Pleroma service, I’m too scared updating, it could break the fragile stability we currently have.
I try being the admin I’d like having as user, and the thought of having all users gone terrifies me: I’m not buying Freenode. Udongein.xyz is, in my believe, not a single user instance; everybody is welcome2. In fact, there are recent new users who I absolutely like the vibes. (and of course I like older users!)
All dark episodes really affected me and made me feel terrible. Since I got instance admin I’ve considered closing and deleting instance at least a dozen times, for various reasons. But I never did it, for Udongein users. They deserve the best and I can’t lose my shit and closing shop at first issue, dropping everything.
I guess Udongein users are wonderful people, I never receive report for them. Either they act as a secret moon society, or they are just so great. I lean for the second answer.
I’m mentally painted in a corner:
- database is a gargantuous monster I can’t move nor reduce size
- I witness weird bugs with Pleroma and debugging is whole story
- I can’t update system without fearing to break something with no doable rollback
if it’s not broken don’t fix it
I’m stuck in this situation, but that’s only postponing the sword of Damocles for later.
Isekai: This Time I Got Reincarnated As A
Maybe switching to another instance software would help, but if people come registering for Pleroma (udongein.xyz is featured on pleroma.social homepage) and if we are not Pleroma any more, they might be disappointed. And I would understand them. I’m currently trying GoToSocial3, which is the current most sane microblogging software alternative on fediverse, in my opinion, but it’s still pretty Alpha, lacking some features (but I appreciate it nonetheless!). There are also other options, e.g. Misskey and its several forks.
One option is creating a new clean Pleroma and migrating on it, doing everything state-of-the-art since day 0. Everything should be fine, right? There’s still a migration process though. I could manage to dump and import the whole userbase, so it’s a smaller step.
This option makes me a bit sad, instance will be on it’s 4th birthday on December. There’s history on this instance.
So, Now What (bis) #
I don’t know, I need time to consider everything, it’s a process that can’t be rushed, I don’t want to rush it. Ditching Pleroma to only have another software isn’t the wise solution in my opinion. The idea of having a new clean Pleroma instance and maybe a few others to vary tastes seems appealing. This is what do several instances, like the Crashing Pigeon.
Let’s see what future will be, the brightest, I hope and wish.
To a certain extend, otherwise I’ll show you the door. ↩︎