In April 2023, I had tried to get GPT4All, a tool to run LLMs locally, to work on one of my servers running Debian 12 at home without avail; I just couldn't get it to install. (I have to say, GPT4All does look much more developed now compared to back then.) So when I learned of Jan, an open source alternative to ChatGPT that runs 100% offline on your computer, I had my doubts. However, I could easily install it, download models, and use it on macOS, Debian 12, and Windows 11! I just had to find the link from the Download section for the relevant version and OS.
It runs the fastest on my Windows laptop using a Nvidia GPU and getting it to work with the GPU just consisted of clicking a checkbox.
There's a list of LLMs that can be downloaded directly from Jan such as Meta's Llama 3.1 8B Instruct Q4, which was what was used above. You can also use models from Hugging Face, which I haven't tried yet.
If you want to use LLMs and worry about privacy, give Jan a go! From their GitHub repo it does say that they are completely free (for now).
This work is licensed under a Creative Commons
Attribution 4.0 International License.