mi·do·naミ・ド・ナ

Home / Library / Help / Set up a runner / llama.cpp

Guide

Set up llama.cpp

llama.cpp is the inference engine most other runners are built on. Its llama-server program runs a model on your own machine and serves it over a local API that mi·do·na can talk to. Nothing leaves your computer. It hands you the controls directly - you manage the model file and the flags - which is the appeal. This is the short version; llama.cpp's own docs go deep.

Get a model running

  1. Get llama-server

    On macOS, the quickest route is Homebrew:

    Terminal
    brew install llama.cpp

    Otherwise, download a prebuilt binary for your OS from the llama.cpp releases, or build from source. Either way you'll get the llama-server program.

  2. Get a GGUF model

    Download a model in .gguf format from Hugging Face, or let llama.cpp fetch one for you with the -hf flag. Start in the 7-8B range for a model that fits most machines; larger is smarter but heavier.

  3. Start the server

    Point llama-server at your model file and pick a port:

    Terminal
    llama-server -m model.gguf --host 127.0.0.1 --port 8080

    It now serves an OpenAI-compatible API at http://localhost:8080/v1, with a built-in web UI at http://localhost:8080.

  4. CORS is already on

    Because mi·do·na runs in your browser, the server has to permit browser requests. llama-server sends permissive CORS headers by default, so mi·do·na can reach it with no extra flags.

Next: connect mi·do·na

With the server running, llama.cpp is ready. Open the connection panel in mi·do·na, point it at http://localhost:8080/v1, and load models. See Connect a model.

Served over HTTPS and can't reach localhost? The browser can block an HTTPS page from talking to http://localhost - most often in Safari. See

Reaching a local model over HTTPS

.

This is a quick-start. For anything deeper, see the official llama.cpp documentation at github.com/ggml-org/llama.cpp.