Software: First and foremost: must be unix-like, must be able to communicate in both ways with an open-wrt router firmware distro and the devices on the local network (android, windows, linux, ipadOS systems). Must be very secure, like enterprise-grade or almost like that. Must be free and open-source. Must be somewhat fault-tolerant (so no Arch or gentoo or anything like that, i don’t feel like recompiling the server’s system daily). Must have these in base repos or easily installed in other methods: secure ssh client (like openSSH or such caliber), a software that enables me to securely control and see the gui of the server from android (Rustdesk? or such), (optionally i2p, dnscrypt, vpn clients, not needed if the router has them, just in case of emergency), ip camera management software, high-security intrusion-detection system, https server with css and js support (preferably command-line). Window manager: must support a very easy to use and lightweight tiling window manager (like i3wm) or if not, its installation and configuration needs to be possible and documented.
Hardware: affordable, x86_64 architecture, should be able to handle all of these at the same time, without freezing or overheating (i live in Hungary, so should be able to handle up to 40°C air temperature with stock fans or there should be space for more fans. liquid cooling is no-go).
I have considered these operating systems. Are any of these bad ideas? What you recommend that is not here?
AlmaLinux Alpine Linux Ubuntu Server Rhino Linux (unofficial ubuntu rolling) Debian Testing Void Linux FreeBSD
Proxmox has been pretty good to me. I have an ancient office PC that has proxmox installed as the hypervisor. It’s based on debian but everything is done via a web interface (you can ssh or whatever into it too if you needed to). Then I have debian with docker containers, TrueNAS, and home assistant all installed as VMs. Benefits to this means you can put mission critical stuff on the “boring” debian and then have fun with whatever you want to tinker with on an entirely different os/Virtual Machine. I also use wireguard easy which is stupid simple to setup a VPN with. I would strongly recommend keeping all management of the server on the local network and use a VPN to connect. This will get you the “enterprise grade” security. Anything public should go through a reverse proxy/DMZ VM if you host something on the Internet. Use cloudflare or similar as an extra layer if you need a domain name and want a buffer between users and your network. Keep that device and software up to date and you should have a great defense.
IDS wise, it’s a lot of work. You’re better off spending that time working on building security by design by doing the above and ensuring anything that touches the public Internet has as little permissions as possible (no running the web server as root/user account), firewall management, etc. If you do want the challenge, or are Interested in learning something like security onion, wazuah or whatnot, don’t let it stop you.
Hardware wise, affordable and uptime could mean it might be cheaper to have a backup machine. Proxmox has features to support high availability where if one of your physical servers go down, another can take over (2 physical servers that are copies of each other). You could have a decent workstation and then a used PC or whatnot as the backup. More important is probably a UPS and some workstation gear unless you want a screaming server jet in whatever room it goes in. Nothing you’ve mentioned seems too performance heavy so technical PC recommendations are going to vary based on expected traffic or use cases. My 2014 DDR3 office PC manages just fine but it’s for very few people and in air conditioned space. You could probably price out mid grade consumer equipment for the main server and a used office PC for redundancy.