Today I had to downgrade fastapi from 0.114.0 to 0.112.4 to make a software work. And it just hit me - what if pip didn’t support 0.112.4 anymore? We would lose a good piece of software just because of that.
Of course, we can “freeze” the packages into an executable that will run for as long as the OS supports it. Which is a lot longer. But the executable is closed source. We can’t see the code that is run from an executable.
Therefore, there is a need for an alternative to which we still have access to the packages even after the program is built. That would make it safely unnecessary for pip to store all versions of all packages forever more.
Any ideas?
If its an open source project, the answer is to rebuild from the tagged source.
Eg: https://github.com/fastapi/fastapi/tree/0.112.4
With the right repo setup, you can
pip install git+https://github.com/fastapi/fastapi.git@0.112.4(example only, not sure it works), so pypi doesn’t need to keep all previous wheels, its just easier for it to do so.There are hellish things such as guix for that.
Or ideally the software needs to be upgraded.
Nix as well
btw :)
There’s plenty of open source projects that distribute executables (i.e. all that use compiled languages). The projects just provide checksums, ensure their builds are reproducible, or provide some other method to verify.
In practice, you’re going to wind up in dependency hell before pypi stops hosting the package. E.g. you need to use package A and package B, but package A depends on v1 of package C, and package B depends on v2 of package C.
And you don’t need to use pypi or pip at all. You could just download the code and directly from tbe repo, import it into your project (possibly needing to build if it has binary components). However, if it was on pypi before, then the source repo likely had all the code pip needs to install it (i.e. contains setup.py and any related files).
Nix
what if pip didn’t support 0.112.4 anymore?
What do you mean by that? If new versions of Python didn’t run that version of fastapi? If PyPI removed it?
If prior versions were not support by pip anymore, so yes, if it were removed. There are cases of packages not being supported by the platforms, aren’t there? I’ve run into cases where the package was fully deprecated and not useable or downloadable anymore.
What do you mean “not supported by the platforms”? And do you mean that or “removed”?
I couldn’t download it even if I wanted to. That’s what I mean. It returns a message saying it isn’t supported.
“It” being the PyPI server not finding it? Pip not supporting the API? Or it downloads correctly but the setup.py prints that error?
Anyway, more access to the open source packages can’t be bad.
I think it said it’s deprecated or something? I’m not sure, I just know I had problems downloading packages before.
I don’t think it was setup.py . I think I tried to download it directly through pip install xx==0.4.0 or something (the version was required by the program) and it said the package doesn’t exist.
Nixpackages or Flatpaks/Appimages(/Snaps 🙈) are a general solution to this (not in this specific case probably), no?
But do Appimages make the dependencies code available? They pack everything into one working program, but what about the packages?
Isn’t this the whole idea behind flatpak but everyone seems to hate it
Who hate flatpak?
I hate them ! How can something be flat and a package at the same time ? Those flatpackers are the flatearther conspirators of IT ! Just burn them on hot pile of logs like we are in the 17th century !!! Burrrrn !!!
i knew i should’ve gone with roundpack instead :(
I hate them (seriously).
It’s basically a second distro inside your distro (try
du -chs /var/lib/flatpak/) and if something breaks (eg. last year mesa with my graphics card) it isn’t easy to identify were the problem is (because all libs update at the same time), plus you can’t just try a newer (or older) version of some lib as you would in your distro.Moreover, you can’t flatpak CLI tools (also servers and OS components, but I guess the ubuntu folks are the only ones who care about those).
So, I’m not gonna pretend flatpak doesn’t use more space then normal apps, but due to deduplication (and sometimes filesystem compression), flatpaks often use less space than people think.
[nix-shell:~/Playables/chronosphere]$ sudo /nix/store/xdrhfj0c64pzn7gf33axlyjnizyq727v-compsize-1.5/bin/compsize -x /var/lib/flatpak/ Processed 49225 files, 21778 regular extents (46533 refs), 22188 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 53% 898M 1.6G 3.6G none 100% 499M 499M 1.0G zstd 34% 399M 1.1G 2.6G [nix-shell:~/Playables/chronosphere]$ du -sh /var/lib/flatpak/ 1.7G /var/lib/flatpak/I only have one flatpak app installed, and
dusays that takes up 1.7 GB of space… but actually, when using a tool that takes up BTRFS transparent compression into account, only half of that space is used on my disk.I recommend using compsize for a BTRFS compression aware version of
duandflatpak-dedup-checkerfor a flatpak filesystem deduplication aware checker of space used.I think flatpak absolutely does use up more space, because yes, it is another linux distro in your distro. But I think that’s a tradeoff people accept in order to have a universal package manager for graphical apps.
Also, you can flatpak cli tools. They are just difficult to run at first because you have to do the
flatpak run org.orgname.appnamething, but you can alias that to a short command. Here is a flatpak of micro, a terminal based text editor.(I prefer nix for cli tools though, and docker/podman/containers for services).
This is why the marketing around flatpak bothers me. It’s touted as some kind of “universal Linux package manager” but Linux is just a kernel - all the stuff that apps depend on comes with the distro. So, in order for flatpaks to be “distro independent” they basically have to supply all the stuff that normally comes from the distro - effectively building a second distro on top of your existing one.
Nix and Guix are the same but at least I think they’re more up front that they are effectively distros that can run on top of your existing distro or as a standalone operating system directly on top of Linux.
@obbeel What? You didn’t explain how. Your “why” is a little scatter-brained, dude.
What do you mean?
I just find that if pip did not support that version anymore, the software would be lost. As that is covered by making executables, as I mentioned them. But what if I wanted to have access to the libraries that were used in the program? That wouldn’t be possible. Because all we get in the source code is the dependency fetching, not the dependencies themselves.
It would be good to have an alternative where you get all that you need to compile the code again, not depending on fetching them from websites that might not even have them anymore.
This mentality of ephemeral code just adheres to the way big tech would like to do things, with programmed obsolescence.
An alternative to that way of doing things would be nice and would make sure we get access to the same working open source program in 30 or 40 years.
I’m not sure there is a “mentality of ephemeral code” in open source projects. The source is literally available on github or similar, and anyone can mirror it as they like.
If it is popular enough, then the project is probably backed up in the github artic vault as well.
While I don’t know of a solution I totally get what you mean… A lot of cool projects don’t work anymore because of that…
Something similar to the container technology, where everything is packed into an image with all it’s dependencies to the correct version. The app would probably be a security nightmare, but could still work in it’s own contained system.
This sounds like a very neat technology that Linux is missing !
