I have received word that there are people combing through the PieFed code looking for anything that might be harmful. This is excellent and can only make PieFed better and less harmful.
We appreciate their interest in PieFed and look forward to answering any questions and showing people around the code. Please join us at https://chat.piefed.social/ or https://matrix.to/#/#piefed-developers:matrix.org.
There’s no need to listen to rumors and amateur speculation when we’re right here and happy to help. Come on in, the water’s fine!
Maybe not what you wanted but why did you pick Python and Flask? More interested in the flask part over say fastAPI. In companies that I worked for apps that use Flask always end up running into limitations with the framework and we end up having to build things on top of it. Obviously the biggest thing from what I remember Flask still uses wsgi ? so you kind of lack true async support.
I like FastAPI and PieFed uses it as well as Flask. Doing SSE (which involves holding HTTP connections open for a long time) was not going to work with Flask so that part is written using FastAPI.
Before starting development I did some experiments with Quart, an ASGI port of Flask. It was fine but didn’t offer significant performance benefits until there were many many concurrent users, a situation that is still far in the future. Also as a more obscure framework it seemed like a risky bet. And if ASGI becomes absolutely necessary, we can migrate from Flask to Quart relatively easily. There’s a good chance that PieFed 2.0 will use Quart.
I have also experimented with gevent as a way to make Flask do async. I got it working but every 5 or 6 hours everything falls over in a heap and needs to be restarted (DB connection leakage). Despite it being unreliable I was able to run it in production long enough to see how it performs and at our current scale it was not any better than what we have now. A bit worse, really.
It’s possible that the importance of async is a bit oversold - https://hackeryarn.com/post/async-python-benchmarks/
Also at the same time the latest versions of Python have done away with the GIL that used to ruin threaded performance so that could eventually mean that wsgi might start to use threads well enough to be pretty great. We’ll see.
GILless python is a good point it makes python scaling alot better. Though every python service ive maintained or created in the long term I always wish I never used python. I understand the calculus though for using the python, reason feels somewhat outdated in the modern world of AI coding imo.
Actually as an aside how much of the code is created by AI
Performance is something developers like to use as a way to assess quality but there are far more important things (which are harder to put a number on to be objective about) like how easy it is for new developers to contribute to. Besides, it’s what you do with it that counts - e.g. despite Lemmy being written in Rust people are finding it much heavier on the server than PieFed.
People can make slow software using any language or framework.
I don’t have a way to know whether code that others contribute is written by AI (except when the quality is really bad, then the PR is rejected) so I bet there’s some in there but I avoid it. I can’t afford the brain-rot of becoming dependent, at my age the rot is happening naturally pretty fast already. There is a whole spectrum of ways to do AI-assisted dev and it’s changing all the time so I’m not trying to police that and just focus on the quality of the code in the PR.
Does piefed image proxy by default? I noticed the homepage for lemmy.today is really heavy since the proxy’d images seem to be full resolution. I ask this since that thread is about people saying piefed runs smoother they said image proxy is by default on lemmy due to CSAM issues. The network utilization blogpost also seems kind of disingenuous since it puts equal weight on the network utilization on javascript, image compression, and a bad api pattern by the lemmy dev. Through those are issues it seems like its 99% you guys more heavily downscale the images vs lemmy.world (Though I like your guy’s solution better).
Do you also manage the piefed.social servers? What kind of cpu/bandwidth/memory utilization do you guys run for the instances serving the api gateway, am curious about the infra setup in general.
The image thing is complicated.
I run piefed.social’s infrastructure, it’s just a medium-sized VPS that has the database, web app, api, etc all in one. There’s also an old server at my home that’s used for some auxiliary stuff like translation services which are shared across all PieFed instances and which runs chat.piefed.social and translate.piefed.social.
rimu talked about this a bit in a fireside fedi chat a while back (~24:00 for tech stack discussion and ~33:00 for python specifically).
As for async, all of the federation work happens asynchronously through the use of celery tasks. Federation work is actually a pretty sizable portion of the computational demand, so offloading that to workers in the background helps a ton. So, the only thing that flask is really responsible for in terms of serving pages is the web UI and the API. It might not scale to the size of reddit very well without a ton of work, but it has been fine for piefed.social. Just like lemmy (or honestly most other applications like this), the main bottleneck is the database rather than any kind of computational overhead for the python framework or rendering speed.
A big advantage of the stack is that it is relatively simple. If you know some python and some pretty standard html/bootstrap, then you can make meaningful contributions (I wrote the dev docs and I am not a professional developer for example). This has led to a large number of contributors that have worked on the features that they feel are important to them since the barrier to contribution is fairly low.
I think the primary disadvantage of using a framework like flask is that it led to PieFed being initially created as just a web interface without an API. Trying to tack an API onto the application after the fact has been a pretty heavy lift, and there are still areas where the API is lacking compared to the web interface. It is something that we have gotten better at now, we add things to the API at the same time as the web interface, but there is a decent backlog of features yet to be added to the API.
Nice insight