llamactrl

A terminal UI for managing local llama.cpp instances across multiple backends (cuda / vulkan / rocmfpx / turboquant) via transient systemd user units.

It shows what's running, lets you stop / restart / swap models, browse the model catalog, and tune launch parameters (context, parallel, MTP, KV cache, cram, env…) with a live VRAM estimate. Crashes (OOM / segfault) are auto-restarted by systemd and surfaced in the UI + a crash log.

Built around a two-GPU setup (ROCm + CUDA), with per-backend llama.cpp builds and a separate ROCmFPX build for the specialised ROCmFP4-format models.

Features

• Configurable slots — each slot is {name, backend, port} and independent of the backend, so you can run several servers on the same backend on different ports (e.g. two cuda instances). Add / edit / delete slots right from the instances pane (n / E / D); they persist to config.toml. A vulkan slot also hosts rocmfpx models (the build is chosen per model). Defaults: cuda on :1235, vulkan on :1234, turboquant on :1237.
• Multi-GPU slots — a slot can pin specific GPUs: the slot form lists the backend's devices (from the binary's --list-devices) as a checklist and a --split-mode cycle (layer / none / row / tensor), plus a main-GPU pick (--main-gpu: with split none the whole model runs on that card; other split modes currently ignore the flag), e.g. one slot spanning CUDA0,CUDA1, or two single-GPU slots on one backend. Persists as [[slots]].devices / split_mode / main_gpu.
• Backend rules — ROCmFP4-format models (ggml type 101) are auto-locked to the rocmfpx build so they can't be wrongly launched on cuda/vulkan.
• Auto-discovered catalog — recursively scans your model directories, classifies main models vs mmproj / draft aux, collapses multi-file shards, hides standalone imatrix calibration files, reads GGUF metadata (arch, quant, trained context, KV-cache dims, chat-template presence) and pairs sibling projectors / chat templates. Duplicates across storages (local + NFS) are deduplicated.
• Smart launch form with a live footprint breakdown:
  • -c is computed as context/slot × parallel (llama.cpp's context is shared across slots);
  • VRAM estimate = weights + KV cache, architecture-aware (hybrid linear/recurrent models like Qwen3.6 Gated DeltaNet count only the full-attention layers; MLA / pure-SSM models are flagged approximate);
  • KV cache type (-ctk/-ctv: f16 / q8_0 / q4_0 …; on the turboquant backend the ← → cycle also offers its runtime KV quants turbo2 / turbo3 / turbo4 = 2.125 / 3.125 / 4.125 bpw; context settings are remembered per quant — cycling restores the ctx that fit it) and cram (-cram/--cache-ram, with an auto-suggestion from the context size);
  • flash attention (-fa: auto / on / off — auto lets llama-server decide; the backend's flash_attn config, e.g. rocmfpx, forces on);
  • unified KV cache (-kvu): one shared pool — each request sees the full context instead of ctx/parallel, and the cache occupies O(ctx) not O(ctx×parallel); contexts above the model's native one are clamped to native per slot in both modes — with -kvu the pool is native shared, without it -c = native × parallel (explicit --rope-* flags opt out);
  • KV cache reuse (--cache-reuse): min chunk size for KV shifting;
  • MTP --spec-draft-n-max and acceptance --spec-draft-p-min, with an on/off toggle to disable MTP entirely for capable models. MTP is detected from the GGUF itself (nextn_predict_layers — e.g. Qwen3.8, GLM-4.6, DeepSeek V3 embed the MTP head, no "mtp" in the filename needed) or from a paired standalone mtp-*.gguf draft (--spec-draft-model);
  • dflash speculative decoding: models with a sibling dflash-*.gguf draft are auto-paired and launched with --spec-type draft-dflash --spec-draft-model; toggle on/off per-launch. DFlash2 drafts (llama.cpp PR #27342, block-diffusion drafter) ship as <model>-DFlash2-<quant>.gguf in their own directory (e.g. z-lab/Muse-Glimmer-30B-DFlash2-GGUF) and pair cross-directory via GGUF-name identity + tokenizer vocab gate + arch gate — all quant candidates cycle in the "DflashDraft" field (Q8_0 > Q4_K_M > BF16), n-max defaults to 15 (the z-lab recipe; classic dflash keeps 16);
  • DSpark speculative decoding (experimental, llama.cpp PR #25173): DSpark drafts ship as <model>-DSpark-<quant>.gguf in their own directory (e.g. erlidev/Qwen3.8-27B-DSpark-GGUF), so pairing is cross-directory via token-boundary name matching — one draft serves every quant of the same base model (Qwen3.8-27B-DSpark-Q8_0 pairs with all Qwen3.8-27B-*). All candidates land in the launch form: the "DsparkDraft" field cycles them with ← → (Q8_0 first — drafts want speed, and bf16 doesn't run on e.g. Tesla V100; measured ~x2 in chat mode with Q8_0 on a V100). Launched with --spec-type draft-dspark --spec-draft-model --spec-draft-ngl <backend ngl>; n-max defaults to 7 (upstream block size). MTP/dflash/DSpark are mutually exclusive (one --spec-type, one draft model): a paired draft defaults dspark ON with Mtp off, flipping one toggle auto-disables the others, and the argv builder enforces the invariant (priority dspark > mtp > dflash). The draft's weights are included in the VRAM estimate while enabled;
  • thinking amount: ThinkBudget → --reasoning-budget N (-1 unlimited, 0 = stop thinking at once, N>0 = token budget — also works for always-thinking models like Qwen3.5/3.6), and Reason.Effort (xhigh/medium/low) for models whose chat template supports the reasoning_effort kwarg (Qwen3.8), passed via --chat-template-kwargs;
  • Forced chat template: models that ship without (or with a wrong) tokenizer.chat_template can be launched with a custom .jinja file (defaults: Gemma 4 family). A toggle in the form enables it;
  • free-form Env (KEY=VAL …, e.g. LLAMA_SERVER_SLOTS_DEBUG=1) and ExtraArgs;
  • optional per-backend vram_gb budget for a "fits?" check.
  • The form scrolls on small screens (phone SSH): fields outside the viewport are hidden with ↑ N above / ↓ N below indicators.
• Remembers each model's last-used parameters per slot (slots are different GPUs — context and cache-quant fits differ) and pre-fills them next time.
• Crash recovery — units run with Restart=always (+ RestartSec, StartLimit); restarts are detected and appended to ~/.config/llamactrl/crashes.log, with crash-filtered journal views.
• Logs overlay with vertical + horizontal scrolling (long lines don't wrap — handy on narrow / phone terminals).

Requirements

• Go 1.23+ (builds against the latest Bubble Tea / bubbles / lipgloss).
• A systemd user session (systemctl --user works; enable lingering if you want units to survive logout).
• Your llama.cpp builds + models on disk (paths are configurable).

Build

go build -o llamactrl .     # or: ./build.sh

Usage

./llamactrl            # launch the TUI (default)
./llamactrl init       # write an example config to ~/.config/llamactrl/config.toml
./llamactrl list       # list discovered models
./llamactrl status     # show current slot/unit status
./llamactrl crashes    # show crash log + journal scan
./llamactrl restore    # relaunch the recorded server set (also runs at boot)

TUI keys

Key	Action
1 / 2 / Tab	focus Instances / Models pane
R	refresh status now
? / h	help
q / Ctrl+C	quit
Instances
↑ ↓	select slot
s	jump to model list (auto-filtered to this slot)
x / r	stop / restart selected slot
e	edit params of the running model & relaunch
n / E / D	add / edit / delete a slot
l / c	open logs / crash-filtered logs
Models
f	cycle backend filter (all / CUDA-capable / ROCmFPX)
/ or s	start name search (type, backspace edits, Esc exits)
Enter	launch (opens the form)
i	model info
Launch form
Tab / ↑ ↓	move field
b	cycle target slot
a	auto-fill cram from context size
p	switch which on-disk copy (local/NFS) to run
← →	toggle/cycle fields (MTP, Mmproj, ForceTpl, DsparkDraft)
Enter / Esc	launch / cancel
Slot form
Tab / ↑ ↓	move field (Name → Backend → Port → Devices → Split → Main)
← →	cycle backend / split mode / main GPU
Enter on Devices	open the GPU checklist (Space/Enter toggle, Esc done)
Enter / Esc	save / cancel
Logs
↑ ↓ / ← →	vertical / horizontal scroll
f c r	follow / crashes-only / refresh

Configuration

Config is optional — sensible defaults are baked in. Edit ~/.config/llamactrl/config.toml (run llamactrl init to generate a commented example). Key sections:

• [defaults] — global base for context, parallel, reasoning, spec_p_min, cache_type, cram, cache_reuse, sampling, restart_sec…
• [[slots]] — launch targets {name, backend, port, devices, split_mode, main_gpu} (managed from the TUI; devices = checked GPUs for --device, split_mode = --split-mode for multi-GPU slots, main_gpu = the main card for --main-gpu).
• [scan] — dirs to scan for models; rocmfpx_dirs (models under these lock to the rocmfpx backend).
• [backends.<name>] — binary, device, host, port, slot, ngl, mtp_n_max, spec_p_min, cache_type, cram, cache_reuse, sampling, optional vram_gb, env, extra_args.
• [[overrides]] — per-model tweaks (id = path relative to a scan dir, or basename): alias, backend, context, mtp, cache type, cram, cache_reuse, env, …
• [[chat_templates]] — maps a model-name substring to a .jinja file for forced template injection. Every matching entry becomes a selectable option in the launch form's ForceTpl cycle (first listed = default), shown by its full file name; several entries may share one match. Optional name per entry overrides the display label. Default: gemma-4 / gemma4 → ~/models/gemma-4-chat-template.jinja; Qwen 3.5/3.6/3.8 offer two templates (qwen3-sharp…, qwen3-v22.2…). Add entries for other model families that ship without a correct tokenizer.chat_template.

All paths support ~ expansion and can also be set via environment variables (which override the file):

LLAMACTRL_CUDA_BINARY, LLAMACTRL_VULKAN_BINARY, LLAMACTRL_ROCMFPX_BINARY,
LLAMACTRL_SCAN_DIRS, LLAMACTRL_ROCMFPX_DIRS   # colon-separated

State lives next to the config: state.json (running units + per-model last-used params) and crashes.log.

How it works

llamactrl never runs llama-server itself — it creates transient user units with systemd-run --user --unit=llamactrl-<slot> … (Restart=always, journald logging, --collect). systemd owns process supervision, restart-on- crash and logging; llamactrl builds the argv, starts/stops/restarts units and reads their status. That means models keep running (and auto-restart after OOM / segfault) even when the TUI is closed.

Restore after reboot

Transient units die with the session, so after a reboot nothing is left of them — but state.json still records exactly what every slot was serving. The enabled user unit llamactrl-restore.service (installed/refreshed on every llamactrl start; runs llamactrl restore at login via WantedBy=default.target) relaunches all recorded-but-inactive units with their recorded parameters. The TUI performs the same restore at startup, and r (restart) in the instances pane relaunches from the recorded entry — so it also works on a crashed/collected unit where plain systemctl restart would fail with "unit not found". Entries are removed only by an explicit stop, so "restore" never resurrects servers you stopped.

Project layout

main.go                  entrypoint + CLI subcommands
internal/config          backends/slots/defaults + TOML + env overrides
internal/catalog         scan, classify, dedup, GGUF metadata, projector pairing
internal/state           persist running units + last-used params (state.json)
internal/systemd         argv builder, transient units, status, journal/crash scan
internal/tui             Bubble Tea dashboard: panes + logs/form/help overlays

Before committing, run: gofmt -w . && go vet ./... && go build ./...