The idea is that someone is checking the code. And by building it yourself, you can at least ensure that you’re getting what’s built from the code. It is possible that some malicious stuff was inserted while building the binary that doesn’t show up in the source code. Building from source solves that problem.

Reproducible builds try to solve that problem by generating some provenance from a third party. A middle ground can be building the binary using something like GitHub Actions, since that can be audited by others. That comes with its own can of worms since GH is owned by M$, but I digress.

So it is technically sane to do it, just not very practical in my view. But for lesser known apps, I do sometimes build from source.