Hello World!
We’ve recently added PieFed.World to the Fedihosting Foundation portfolio.
PieFed.World is still in its early stages, and we still need to port some of our automations we already have in place on Lemmy.World. This includes functionality to inform people about moderation actions taken against them, as well as some other moderation tooling. Administration is currently done by the same team responsible for Lemmy.World, and the same rules that apply to Lemmy.World also apply to PieFed.World.
What is PieFed?
PieFed is a Fediverse/Threadiverse platform similar to Lemmy or Mbin/kbin. You can find a description and feature comparison with Lemmy on their website.
While PieFed has a range of features currently not present in Lemmy, it also is a a lot younger and isn’t quite as robust as Lemmy currently is. There are still many bugs and missing features that you will likely run across compared to Lemmy, which will take time to be addressed. PieFed has fairly active development and is seeing a lot of issues addressed fairly quickly, which is especially important recently, as the number of active PieFed instances and PieFed users increased significantly with a range of Lemmy instances opening up PieFed instances as well. PieFed currently does not have proper “stable” releases and no test suite, so it’s not unlikely for things to break from time to time. Although 1.0.0 has already been released a while back, there are still too many issues addressed in more recent commits to stay on that version.
As PieFed is part of the same federated network as Lemmy and Mbin, all PieFed communities can be accessed from Lemmy and Mbin, as well as other Fediverse platforms. Likewise, PieFed can access communities from Lemmy, Mbin and other Fediverse platforms. Whether you use a PieFed instance, a Lemmy instance, or an Mbin instance, it does not matter what type of instance the community is on. The software affects your own user experience, but the content is available regardless.
Creation of communities
Creation of communities will be limited to admins for the first week of the public launch. We will reserve this time to allow community moderators of established communities to claim the name on PieFed.World before we open community creation to the public. We will limit this to communities with the same name and at least 2k monthly active users. In case of multiple qualifying communities with the same name on different instances expressing interest, Lemmy.World communities will be given preference, afterwards the number of monthly active users. Please reach out if you’d like to discuss an exception. Requests can be posted in [email protected]. After the first week, community creation will be available to anyone.
Migration of communities
PieFed has a feature to migrate communities to a local instance. We will not be offering PieFed’s community migration feature initially.
We still need to research the details of how this works and the impacts this has on federation before we will make a decision on whether will support this in the future. If requested, we may reserve some names for potential future community migrations until we have made a decision to allow community migrations.
This does not prevent you from moving communities in the classic way, by opening up a new community and posting in the old community that people should move over.
Private voting
We had previously disabled private voting for PieFed.World before opening the instance to the public, as the original implementation has a range of drawbacks when it comes to federation, and our team overwhelmingly believed that the individual benefits of private voting did not outweigh the impact this has on the Fediverse beyond the user’s instance. Additionally, due to the implementation of that feature, it was also trivial to identify the original voter, which significantly limited the promises of this bringing actual voting privacy.
Since then, the implementation of private voting has been changed to provide the option of federating or not federating votes. While this is more likely to result in vote differences across instances, it does not feed bad information to other instances, which could make it a lot harder for other instances to identify manipulation.
Non-federated voting is available for all PieFed.World users.
Topics
Topics are a kind of “starter packs” or collections grouping multiple communities that people can follow, curated by the admin team. We don’t have a clear vision for the structure of these yet.
You can see an example structure on piefed.social.
Feel free to let us know your thoughts on this.
Feeds
PieFed supports feeds, which are user-created groups of communities, similar to topics. These are currently in a global namespace and all users can create public feeds in the same shared namespace.
Reputation and vote weight
PieFed has options for admins to treat certain types of content differently for “reputation” calculation, as well as options for weighing votes of specific instances differently compared to others. We currently have all options for treating certain content, communities or instances differently disabled.
How does PieFed compare to Lemmy?
PieFed has various features not present in Lemmy, check out their website!
There is also various functionality that Lemmy has, which you may be missing currently with PieFed for now:
Limited API support
In Lemmy, the default web interface relies entirely on the Lemmy API. This has the major benefit of all functionality available in the default web interface also being available to all third party clients. PieFed currently uses separate code paths and implementations for the default web interface and its API. To make it possible to access functionality in third party apps, dedicated API endpoints have to be created, even if this functionality is already available in the default web interface. This also includes alternative web-based UIs.
Multiple developers of alternative UIs and mobile clients are already working on PieFed support, some already released experimental versions.
Limited availability of Markdown previews
Markdown previews are currently only available in posts. There are many other places that accept markdown, but you can’t preview the rendered comment before submitting it. This is tracked in #532.
Image uploads only on post creation
Images can’t be uploaded to comments currently. You’ll have to host them externally for now. This is tracked in #768.
Autocompletion of users/communities
Usernames and communities can’t be autocompleted when typing their names currently. This is tracked in #799.
Limited availability of modlog
Modlog is currently very limited. While there is an instance modlog, there are currently no filters available, so it’s not possible for users to see actions taken against a specific user or within a specific community. Community modlog exists, but it is currently only available to community moderators and admins. Filtering modlog is tracked in #846.
Moderator hierarchy
Lemmy has a moderator hierarchy based on the time a moderator was appointed, relative to other moderators in the community. This allows moderators to add other moderators, but they can only remove moderators that were added later than they were. There are a few other actions that check moderator hierarchy as well, including deletion only being possible by the top mod. In PieFed, communities have one or more owners, who can add and remove moderators, while all other moderators are currently on equal level. Community owners currently cannot be changed without editing this directly in the database, if you’d like to change owners in your community please reach out in [email protected].
Donations
Similar to Lemmy, PieFed development is supported by donations. You can donate to PieFed development through Patreon.
Additionally, we would appreciate donations towards the Fedihosting Foundation, the non-profit organization operating PieFed.World, Lemmy.World, and a range of other Fediverse platforms.
Problems and questions
Please report any issues and questions about PieFed.World in [email protected].
For topics about the software PieFed, please visit [email protected].
Bugs can be reported on Codeberg.
TLDR: New platform with similar functionality available, Lemmy.World will continue to exist.
edit: reordered sections and minor wording changes
edit 2: updated community owner information
Seems orthogonal?
I am 100% for the accessibility of Python, but won’t scaling up to large instances/userbases be an issue? Unless some underlying library does the heavy lifting I guess, or they eventually use some optimized subset. Or maybe the bulk of the costs is disk/bandwidth anyway, not CPU, IDK.
These features though:
All these are fire! I have no idea why Lemmy devs wouldn’t implement them (and am open to explanations/arguments).
And this is good too, though it makes me worry about power-trippin’ admins: https://join.piefed.social/2024/06/22/piefed-features-for-growing-healthy-communities/
Rust is probably the most high performance future-proof choice language of choice. While still not extremely popular, it is steadily growing. Of course the way code is written is more important than the language. But the fact that Lemmy is already on Rust and Piefed on Python, puts Piefed a foot behind unless their code uses far more efficient algorithms.
This is an awful idea because bots become invincible. Also how would this even be possible in the fediverse? Publishing votes is required for federation. Even if it is hidden publicly, someone could make an instance which federates with all other instances and simply see the votes in their back-end.
This promotes hiveminding.
Probably good
The best thing in their list seems to be community migration of accounts but they have not implemented that yet.
After having discussed with a few people, Rimu agreed to drop the private voting. The system will switch to two votes button
a local one that will stay local
a federated one that will be sent to other instances
https://piefed.zip/post/148961?sort=new#post_replies
https://chat.piefed.social/#narrow/channel/5-admins/topic/Private.20voting/near/332
This provides better transparency than Lemmy, as users are now complaining about tool like lemvotes:
People are complaining because… they have been detected for vote botting? I think this kind of proves my point. With public votes the users can see whether any manipulation is taking place. Because the admins do not seem to take it into account.
Two sets of vote buttons sounds obtuse. If users really do not want their votes published a simpler option would be to have a toggle in their profile or on the header bar to decide whether their votes will be local or global
The issue comes more from users not being aware that federated votes are public due to federation.
There is indeed a checkbox in the settings
https://chat.piefed.social/#narrow/stream/3-general/topic/Public.20voting.20UI/near/2125
Votes UI
https://chat.piefed.social/#narrow/stream/3-general/topic/Public.20voting.20UI/near/2132
I don’t think “people are complaining” is correct. Just one person started complaining because they were caught using that tool and they were caught because they were harassing users and their harassments were getting a weird amount of upvotes for being a piece of shit. A couple other people half-ass agreed with them, but there’s never been any major complaints or movements.
Saying “people are complaining” because of one unhinged person is highly inaccurate.
https://lemmy.dbzer0.com/post/43096615?scrollToComments=true
Lemmy.ml has defederated gregtech.eu , the instance used for lemvotes
https://lemmy.ml/instances
when we started setting up piefed.world we subscribed to all active communities known to LW, which allowed us to already run with production level load on the inbound federation side. we also already discovered various performance bottlenecks that have since been improved, but i expect there to be plenty more to get addressed.
in its current state, i don’t think piefed actually performs better than lemmy when exposed to the same traffic, but i don’t have any numbers to confirm this. a large part of performance optimization can be done even without switching languages though, simply by structuring your code correctly.
when really optimizing both languages for performance, rust would easily outperform python, but it depends a lot on the workload whether this is relevant. ensuring that the database structure and queries are optimized is just as important as the application itself.
It’s interesting that it’s even in the ballpark at this early stage! And it’s great that they get a ‘real world’ stress test to see what paths chug.
Yeah, this is what I was thinking!
If piefed ends up having a few CPU hot spots, they could theoretically ‘break out’ of Python just for them and/or get replaced with select precompiled libraries. Python is great orchestration language, hence that’s precisely what many projects do.
There’s also the occasional effort at a sped-up python runtime, like Pyston, Mojo, GraalVM and such. It’s possible that could be an option for piefed some day.
And like you said, database structure is more important, and runtime overhead may not be too relevant.
choice of programming language for web apps often has minor effects on scalability if any, it’s all about the design and database
and yea these instances need a lot of disk space, but programming language doesn’t affect that either
After having discussed with a few people, Rimu agreed to drop the private voting. The system will switch to two votes button
a local one that will stay local
a federated one that will be sent to other instances
https://piefed.zip/post/148961?sort=new#post_replies
https://chat.piefed.social/#narrow/channel/5-admins/topic/Private.20voting/near/332
This provides better transparency than Lemmy, as users are now complaining about tool like lemvotes: https://lemmy.dbzer0.com/post/48233901/19876557
From my experience contributing to Piefed, the bottleneck is almost always the database. So, doing optimization on that front has yielded improvements. I suspect that the db is likely the main bottleneck of lemmy as well (though I know network speed/latency has been an issue in the past).
NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O lemmy-postgres-1 1717.01% 48.6GiB / 78.12GiB 62.21% 1.84TB / 26.9TB 59.2TB / 67.7TBthe container has been up for 3 weeks
I was referring more to responsiveness and page loading time than overall resource usage. I don’t know enough on the sysadmin or database side of things to really speak authoritatively about things at scale. I just mean that it’s usually postgres instead of the language of python that limits what I have worked on.
i guess to put it in perspective i should have included other containers as well.
this is on a 32 physical core box with 96gb of memory:
NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O lemmy-lemmy_ui_1-1 13.59% 1.864GiB / 94.15GiB 1.98% 198GB / 1.34TB 5.5GB / 4.45GB lemmy-lemmy_ui_3-1 20.12% 2.764GiB / 94.15GiB 2.94% 312GB / 2.22TB 6.36GB / 4.24GB lemmy-lemmy_ui_2-1 54.07% 2.042GiB / 94.15GiB 2.17% 222GB / 1.52TB 5.61GB / 4.61GB lemmy-lemmy_api_3-1 10.16% 3.629GiB / 94.15GiB 3.85% 5.6TB / 1.71TB 290GB / 411GB lemmy-lemmy_sched_1-1 0.01% 42.79MiB / 94.15GiB 0.04% 9.72GB / 1.23GB 1.38GB / 2.56GB lemmy-lemmy_api_2-1 30.97% 4.049GiB / 94.15GiB 4.30% 5.07TB / 1.57TB 281GB / 401GB lemmy-lemmy_api_1-1 27.39% 3.913GiB / 94.15GiB 4.16% 4.95TB / 1.54TB 286GB / 404GB lemmy-postgres-1 1500.05% 49.99GiB / 78.12GiB 63.98% 1.84TB / 27TB 59.2TB / 67.7TB lemmy-lemmy_fed_1-1 405.39% 802.1MiB / 94.15GiB 0.83% 3.67TB / 5.67TB 12.9GB / 19.4GB lemmy-pictrs-1 514.29% 1.495GiB / 94.15GiB 1.59% 8.79TB / 5.36TB 511GB / 114TB lemmy-pictrs_db-1 2.76% 2.956GiB / 68.36GiB 4.32% 531GB / 325GB 13.1TB / 9.25TBwith lemmy it’s a lot easier to separate frontend performance from backend performance, with piefed that’s harder to tell when not just using the APIs and requires more analysis. proper indexes and well crafted queries have a lot of impact though. most logic that you can move to the db side is probably better done there, but it can make queries a lot more complex.
Oh that’s fascinating. I wonder what the hot code for the postgres instance would be like in a profiler, or if it would ‘like’ a SMT-heavy server like a IBM Power10 one since most of the load is db calls.
…Also, what is lemmy-pictrs doing? Is it just serving, or does it transcode/optimize too?
pict-rs also does transcoding and resizing of thumbnails, yeah. it caches the results in s3, but it there is lots of new content all the time obviously. i did pick one of the more active load snapshots, it’s not always at 500%.
Random thing, but if you are looking for lossless png/jpeg optimizers, I did a big shootout a while back and found this little nugget:
https://github.com/fhanau/Efficient-Compression-Tool
It often beats the filesizes of big (zopfli) PNG compressors with a fraction of the CPU time. It can also losslessly convert jpegs to a progressive format, so they load better on the user’s end (and also shave off like 1% of the filesize).
Or maybe you already handle this! IDK, I am not an admin :)
that’s all handled by pict-rs, we don’t have a custom setup for this separately. i think it’s mostly just wrapping imagemagick and ffmpeg for image manipulation.