Introduction

This article is a practical guide to installing Large Language Models on your own hardware, along with local inference tools, and use them for assisted software development through locally executed agents.

It is primarily oriented toward Linux users; the same tools are available on macOS as well so it shouldn’t be too complex to adapt.

Why local ?

It is nearly certain that Generative AI is a financial bubble waiting to burst at any moment, and that the environmental impacts of AI data centers hosting frontier models are completely unsustainable for generating slop pictures and replacing what used to be simple web searches.

I believe one of the reasons for this sad situation is that we have been convinced by the major providers in the AI space that LLM inference must absolutely happen in the cloud ie. on the servers they own. Of course, they have an economical incentive to make their customers believe that: they really want to be cloud companies (not “labs” as they want their customers to believe) and extract this sweet recurring rent from their customers, while hoarding as much infrastructure as they can to price everybody else out of hardware ownership; at the same time they get precious new training data by collecting all their users’ interactions with models. And hardware providers are playing along as it enables them too to price non-AI consumers out of owning any hardware and turn that into rent too.

Still as a developer, it’s a technology that can have some usefulness when used responsibly: speeding up boring tasks such as configuration files, getting up to speed faster with new languages and frameworks, investigating cryptic error messages, quickly grasping new codebases …

So, going self-hosted is the most ethical way to do it:

  • Privacy: Everything stays local, no spying
  • Resources usage: Local inference with small models is certainly not worse for the planet than gaming, for instance, as it only peaks at a few hundred watts for short bursts VS 300W continuously for long gaming sessions on your PS5

It is still not perfect. To my knowledge, no truly Open Source quality models are currently available, only so called “open-weight” models where only the training results are made available for free; the training data and process typically remains a secret.

On a more personal note, I enjoy the idea of owning my computing, my data, and taking back some control of my digital life. I’m not interested in a future where I have to depend on SaaS providers for basic computing.

The Hardware

Hardware requirements for local LLM inference are not that demanding for small models. Still for satisfying results, a discrete GPU is required, or a beefy APU such as a Apple Mx chip. I personally own a RTX 3060 from 2021 with 12GB of VRAM and it is up to the task; more VRAM would be useful to load larger models, but even 5 years later only absurdly expensive cards have more VRAM that that (thanks again NVIDIA !). An alternative is to use a machine with a unified memory model such as a Mac Mini or a Ryzen AI based PC: it will be slower for small models, but capable of running larger ones, so that’s a good trade off.

The LLM

LLM Inference: LM Studio

LM Studio is essentially a desktop frontend for llama.cpp, with built-in model discovery and downloading, and implementing various standard API (such as an OpenAI-compatible API) to make local models available to other apps. It offers a lot more configuration options than ollama, for instance a finer control of what gets loaded on the GPU and what is offloaded on the CPU, context window management, etc … resulting in better performances.

It is available as an AppImage for easy installation, and on Arch it is wrapped by this AUR package:

yay -S lmstudio-bin

Choosing the right model

Here are some well-known models that work reasonably well for coding tasks on this setup:

  • Gemma 4 12b QAT
  • GPT-OSS 20b

Don’t forget to increase the context window to at least 64k tokens in LM Studio, anything less will make tool usage impossible.

A good alternative for long reasoning tasks is to use a larger MoE model such as Qwen 3.6 35B A3B. Thanks to its architecture, it can still run reasonably well on constrained VRAM with the right settings: in LM Studio, force GPU Offload to the maximum number of layers (40 in the specific case of this model), and tune the “Number of layers to force the experts on the CPU” setting until it successfully loads and fits into VRAM.

Coding agent: Zed

Zed can be configured to target LM Studio, both for its integrated coding agent, as well as completions.

Merge this into your ~/.config/zed/settings.json:

{
  "language_models": {
    "lmstudio": {
      "api_url": "http://localhost:1234/api/v0",
    }
  },
  "edit_predictions": {
    "open_ai_compatible_api": {
      "prompt_format": "infer",
      "model": "google/gemma-4-12b-qat",
      "api_url": "http://localhost:1234/v1/completions",
    },
    "provider": "open_ai_compatible_api",
  }
}

Coding agent: OpenCode

OpenCode is my favorite coding agent as it can be run standalone in a terminal, or integrated in Zed through ACP (Agent Client Protocol) integration.

Put this in your ~/.config/opencode/opencode.jsonc:

{
  "$schema": "https://opencode.ai/config.json",

   "model": "lmstudio/google/gemma-4-12b-qat",
   "small_model": "lmstudio/google/gemma-4-12b-qat",

   "provider": {
    "lmstudio": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "LM Studio",
      "options": {
        "baseURL": "http://localhost:1234/v1"
      },
      "models": {
        "qwen/qwen3.6-35b-a3b": {
          "name": "Qwen 3.6 35B A3B"
        },
        "openai/gpt-oss-20b": {
          "name": "GPT-OSS 20B"
        },
        "google/gemma-4-12b-qat": {
          "name": "Gemma 4 12B QAT"
        }
      }
    }
  }
}

What is such a setup good for ?

It can still be surprisingly useful. For instance, I wrote a skill to automatically search on the web if upgrades are available for the various apps installed on my server, automatically discovering them from an existing Ansible playbook.

No LLM was used to write this post.