There is a project I’ve discovered recently which is similar to GPT4All, except you can throw multiple GPUs at the workloads (and yes it can use Vulkan): https://github.com/LostRuins/koboldcpp
I haven’t messed much with it but it builds and works fine on Linux. The only thing I don’t like is that the source tree has a bunch of Windows binaries in it.
May I ask if you could point to a compiled list of them for the uninitiated and tech illiterate?
If you’re looking for online tools, here are a few Chinese based ones to check out
For open source models you can run locally
There is a project I’ve discovered recently which is similar to GPT4All, except you can throw multiple GPUs at the workloads (and yes it can use Vulkan): https://github.com/LostRuins/koboldcpp
I haven’t messed much with it but it builds and works fine on Linux. The only thing I don’t like is that the source tree has a bunch of Windows binaries in it.
Oh nifty, another handy one is Jan, it can even use MCP with the beta version https://jan.ai/