OG post in case my instance dies, which is not unexpected:

The last couple of weeks have been truly eventful for lemmy.fan.

tl;dr (and first paragraph):

My plan was to wholeheartedly go all in on PieFed for lemmy.fan. I planned a cut-over date, started moving some things, and I was on track to get everything up and running solely on PieFed[.lemmy].fan¹. Then lemmy.fan died. It’s back now.

/slash tl;dr

The reasons behind the migration were varied, but essentially boiled down to a lack of development from two full-time, donation-sponsored-but-still-underpaid developers, nutomic and dessalines from lemmy.ml.

The lifecycle of open-source software development is well-established in lore if not in fact: under- or unpaid developers work on a project that started as a labor of love. The love disappears, and the labor quickly turns to animosity and dread, as Git repos devolve into loud, angry people demanding this or that, reporting bugs but not contributing to fixing existing ones, and always the politics, politics, politics.

Then, PostgreSQL made the decision to utterly shit itself. Lemmy.fan suffered a catastrophic database failure; from what, who knows.

Lemmy.fan, as I once knew and loved, now lies in a pile of corrupted Postgres garbage files, gnashed angrily together by some destructive, demonic, database daemon.

I know little to nothing about PostgreSQL, and that is why I absolutely despise it. My life has been spent using, manipulating, troubleshooting, and migrating MariaDB and MySQL, two very sane and easy-to-use database systems that Just WorkTM. I do not want to learn something new. I fear I now have no choice but to learn this garbage database system and adopt the same relationship with it as I have with so many other things in my life: Don’t fuck with me, and I will not fuck with you. Cross me once, though, and you best be prepared for total annihilation.

I think Postgres has that sorted now, as we’re circling the saloon old-west style, revolvers pointed at one another, shaking slightly in unsteady hands, and eyeing one another for the moment one of us so much as blinks.

Also, it’s good to have backups. I did, and still do, but I decided not to restore them, and here’s why:

A long time ago, when lemmy.fan was but a tiny baby Docker container nestled snugly in a NASsinette in a suburban basement, I created the very first federated lemmy.fan using Yunohost to test things out. I was new to ActivityPub and had little idea as to how the federation worked in broad terms, so I set about this and that. Before long, lemmy.fan was puttering along, populated by the lemmycommunity bot that would dutifully scrape and subscribe to popular communities and instances across the fediverse. I had no local communities and was very new to Docker, so lemmy.fan and I expanded our knowledge: I by learning Docker, Portainer, and other tools, and lemmy.fan sucking in content from across the fediverse, growing and becoming better in its own self.

Then, I broke something, or lemmy.fan itself broke, or something happened that resulted in me having to destroy the instance.

I mistakenly then thought that the domain, lemmy.fan, was no longer available to federate with because I had used it already, exchanging messages using the ActivityPub protocol. Now that the domain was established in the greater fediverse, I thought that I had to go to a subdomain.

So I added real.lemmy.fan as the federation source, CNAMED it back to lemmy.fan, and believed that perhaps everything would just work.

And it did, for about a year or so.

As I trudged along keeping lemmy.fan mostly running, I grew and fostered weirdnews, a community that surpassed 1,000 subscribers. A few friends joined, I added a few other communities, and I kept the small instance chugging along splendidly.

Then something changed.

Lemmy.fan became slow, unreliable. Server errors were pretty commonplace and, while restarting the Docker containers fixed the problem, the underlying cause was a mystery to me. Lemmy.fan’s performance began dragging down other containers on the NAS, the PostgreSQL and lemmy frontend containers putting load averages in the 20s. Logs were useless and showed no particular fault.

I decided to migrate the instance from the basement NAS to a VPS, my thought process being that allocating more RAM and throwing a few more processor cores at lemmy.fan would fix things. For us, it was the vacation preceding the divorce; we being the hypothetical couple who tried to save a failing marriage by going to Hawaii and renewing our vows. Instead, lemmy.fan fell asleep on the couch while I gamed and watched reruns of old Star Trek episodes. Year of Hell Parts 1 and 2 back-to-back on Pluto?

Yes, please!

This is a death knell for any relationship, be it human and human or human and silicon/electron.

I’ll back up a moment. I am not naive. I will ever conflate lemmy, or really any open-source software written by a small handful of volunteer or underpaid developers, with stability. And that’s OK. I accepted the fact that I would be in for a few bumps and scrapes here and there: like the time a new lemmy UI version was released that cocked up any form fields, resulting in a shitty UI experience. Or the time that the lemmy backend would just fuck around and die, taking others down with it in a spectacular blaze of error messages, all cryptic to me. Or the time when never-ending scrolling was dismissed because one person who happens to be the main developer just does not want it.

Concurrently, as lemmy.fan slowly grew and went through its adolescent phase, development on lemmy became less predictable and eventually stalled to the point where significant bugs and other issues were, and still are, being neglected as lemy version 1 is developed. I will NOT be that loud, vocal, open-source criticizer who laments the lack of work and progress from underpaid developers not giving into my demands and wants, so I began to research other options.

Some medium-sized instances had come to similar conclusions; whispers of moving to PieFed and PyFedi were becoming louder. Intrigued, I began looking into the PieFed project. I started doing some research, dodging thinly-veiled threats from my VPN provider that I was responsible for slowing down their shared server and I Better Fix It Soon Or Else. After waiting more than two minutes for lemmy.fan to crank out a single page of content as the server load averages climbed into the high twenties, I made the heartbreaking decision that lemmy.fan, the real.lemmy.fan, needed to be put out to pasture, and right soon. A deadline was set, I put some more popular communities (double-digit user counts, baby!) to read-only or migrated them to the new PieFed.lemmy.fan instance, and started the countdown clock. Archive.org has a cache of it if you are interested.

Then, lemmy.fan had its stroke.

To be continued…

¹ The original URL for the new PieFed instance I wanted to bring up was piefed.lemmy.fan. In the time between the plan to migrate and spin up PieFed Fan I purchased the piefed.fan domain. Because I have not yet worked much on PieFed after resuscitating lemmy.fan, I have not changed the federation URL in PieFed so the instance is not running yet. My goal is to have PieFed.fan up and running by the time part 2 of this post is published.