Skills DirectorySearch all skills ↗AI agent skill · by @vasilyu1983
Ai Local Model Ops
Runs local and self-hosted LLM workflows with Ollama, LM Studio, MLX, Open WebUI, llamafile, and adapters. Use when operating private model stacks.
Archived 2026-10-23 · Source updated 2026-09-02 · License: MIT. Check the original repository for the latest version.
SKILL.md instructions
# Local Model Operations Use this skill to choose and operate local or self-hosted LLM workflows when privacy, offline access, or low-friction experimentation matter more than large-cluster serving. This skill covers: - local runtime choice for laptops, workstations, and small self-hosted setups - team-facing local or private chat surfaces - single-binary or minimal-dependency model packaging - lightweight adaptation paths before full training or cluster-scale serving - evaluation and escalation rules before a local stack becomes a product dependency ## ASCII Flow ```text local/private model need | v constraint privacy | offline | cost | hardware | demo portability | team chat | v runtime selection Ollama | LM Studio | MLX | Microsoft Foundry Local | Open WebUI | llamafile | lightweight adapter workflow | v local operating contract pinned model + quantization + eval set + storage/privacy boundary + optimization levers: KV-cache quant | speculative decoding | NPU tier | v use or escalate local workflow OR hand off to inference/MLOps for production serving ``` ## Quick Reference | Need | Default path | Notes | |------|--------------|-------| | Run a local model quickly | Ollama | Lowest-friction day-0 local runtime for experiments and private workflows | | Share a self-hosted chat UI | Open WebUI | Best fit when a team needs a ChatGPT-like local or private interface | | Ship a no-install demo or portable binary | llamafile | Useful for single-file distribution and low-ops delivery | | Apple Silicon on-device inference at framework level | MLX (mlx-lm) | Primary path for Metal-native inference and LoRA fine-tune on Mac; verify at https://github.com/ml-explore/mlx-lm | | GUI model browser and switcher (non-technical users) | LM Studio | Supports GGUF and MLX; good for rapid model comparisons | | Windows / enterprise SDK-first local inference | Microsoft Foundry Local | Curated Microsoft catalog; SDK + REST; verify at https://learn.microsoft.com/en-us/ai/foundry-local | | Fine-tune or adapt cheaply | Unsloth + `../ai-llm/SKILL.md` | Good for lightweight adaptation, not a substitute for full training ops | | Optimize throughput or production serving | `../ai-llm-inference/SKILL.md` | Use this skill for local ops; use `ai-llm-inference` for deeper serving engineering | ## Runtime Selection | Situation | Best fit | |-----------|----------| | Solo developer or analyst on one machine | Ollama | | Internal team chat with local or self-hosted models | Open WebUI | | Portable model demo or offline executable distribution | llamafile | | Fast adapter and fine-tuning iteration on limited hardware | Unsloth | | Apple Silicon, framework-level inference or LoRA fine-tune | MLX (mlx-lm) | | Non-technical user, GUI model management | LM Studio | | Windows-primary, SDK-first, Microsoft model catalog | Microsoft Foundry Local | See [references/desktop-runtime-landscape.md](references/desktop-runtime-landscape.md) for a detailed Ollama / LM Studio / Foundry Local comparison. ## Local vs Hosted API: Judgment, Not Reflex Do not default to "local" just because privacy or cost was mentioned once. Decide with a real eval set and a real cost model: - Local tends to win on data residency/offline requirements, steady high-volume traffic that amortizes hardware cost, sub-100ms latency needs, and narrow tasks where a well-evaluated 7–32B open model already matches frontier quality. - Hosted API tends to win on frontier-tier reasoning or long-context needs that no locally-runnable model size covers yet, spiky/low-volume traffic, or when local ops overhead (drivers, quant regressions, capacity planning) would cost more engineering time than the API bill. - Full frontier-scale open weights (DeepSeek-V3/R1-class, Mistral Large 3, Llama 4 Maverick) do not fit on a single consumer GPU or single H100 at usable quant — "open-weight" does not mean "runs on your laptop." Check the model-sizing matrix before promising local feasibility. - Local *inference* feasibility does not imply local *training* or fine-tuning feasibility. Training amplifies memory, bandwidth, and especially energy per sample well beyond inference, so the binding constraint often shifts from VRAM capacity to thermal/power budget. See [references/model-sizing-matrix.md](references/model-sizing-matrix.md#local-training-is-a-different-feasibility-question) before promising on-device adaptation. See [references/model-sizing-matrix.md](references/model-sizing-matrix.md#local-vs-api-the-real-tradeoff) for the full tradeoff and current family-by-family sizing (Llama, Qwen, DeepSeek, Mistral, Gemma, GPT-OSS). ## Default Workflow 1. Define the real constraint first: privacy, offline use, cost ceiling, hardware ceiling, or demo portability. 2. Pick the runtime or UI layer that matches that constraint. 3. Pin model IDs, quantization choice, and prompt/eval set before broader rollout. 4. Decide whether the stack is only for local use or will become part of a product or team workflow. 5. If it needs stronger serving, routing, or monitoring, hand off to the adjacent skills instead of stretching a local-first setup too far. ## Operational Rules - Keep model IDs and quantization choices explicit and versioned. - Treat local and self-hosted endpoints as sensitive services, not casual defaults for internet exposure. - Measure quality on a small real eval set before swapping local models into a user-facing workflow. - Separate runtime selection from product integration. Running a model locally is not the same thing as shipping a good AI feature. ## Known Traps - Treating a laptop prototype as proof that a workflow is production-ready. Latency, uptime, auth, and observability requirements change immediately once real users appear. - Leaving model IDs, quant levels, and system prompts implicit. Local stacks drift quickly when operators rely on tags like `latest`. - Exposing Ollama, Open WebUI, or ad hoc reverse proxies without an explicit threat model and access controls. - Assuming a polished chat UI solves governance. UI convenience does not replace logging, retention policy, or approval paths. - Using lightweight local adaptation as a substitute for evaluation discipline. Faster iteration is useful only if the eval loop is real. - Using vLLM V0 features — V0 is fully deprecated as of 2026. Use the V1 engine (see https://docs.vllm.ai/en/stable/usage/v1_guide/). - Citing vendor speedup figures (e.g., Microsoft Foundry Local vs cloud) as neutral benchmarks — always measure on your own workload. - Assuming speculative decoding helps at high concurrency — the benefit is concentrated at batch size 1. - Running NPU inference without confirming the accelerator is being used — fallback to CPU/GPU is silent in some runtimes. ## Common Anti-Patterns - Installing several local runtimes at once before deciding which constraint actually matters: privacy, portability, cost ceiling, or offline access. - Treating local models as drop-in replacements for hosted models without rechecking tool use, structured outputs, and long-context behavior. - Shipping a team workflow on consumer hardware with no capacity envelope, backup path, or restart procedure. - Using "local" as the only justification for a stack choice when a small self-hosted or managed setup would be operationally safer. ## Escalation Boundaries Use adjacent skills when: - you need cluster-scale serving or throughput tuning -> [ai-llm-inference](../ai-llm-inference/SKILL.md) - you need full fine-tuning strategy, dataset design, or evaluation -> [ai-llm](../ai-llm/SKILL.md) - you need product UX, streaming chat, or structured output in an app -> [software-ai-integration](../software-ai-integration/SKILL.md) - you need deployment, monitoring, or operational governance -> [ai-mlops](../ai-mlops/SKILL.md) ## When To Use This Skill Use this skill when the user asks: - "Should I use Ollama or something else locally?" - "How do I run private models on my machine or a small server?" - "What should I use for a self-hosted ChatGPT-like interface?" - "How do I package a model into a low-friction local demo?" - "When should I stay local vs move to a real serving stack?" - "What's the best way to run models on Apple Silicon / Mac?" - "Should I use MLX or Ollama on my M-series Mac?" - "What small models can I run locally? Gemma / Phi / Qwen / DeepSeek / Mistral?" - "How do I speed up local inference with speculative decoding or KV-cache quantization?" - "What is LM Studio / Microsoft Foundry Local for?" - "Should I run this locally or just call a hosted API?" ## Navigation **References** - [references/runtime-selection.md](references/runtime-selection.md) - runtime, UI, and packaging choice rules - [references/desktop-runtime-landscape.md](references/desktop-runtime-landscape.md) - Ollama vs LM Studio vs Microsoft Foundry Local: detailed comparison, decision heuristic, anti-patterns - [references/adaptation-and-packaging.md](references/adaptation-and-packaging.md) - lightweight adaptation, portable delivery, and evaluation handoff rules - [references/model-sizing-matrix.md](references/model-sizing-matrix.md) - Llama 4, Mixtral, Qwen 3, GPT-OSS across M3/M4/4090/H100 hardware tiers; MLX and Ollama Apple Silicon paths; throughput estimates; why local training feasibility differs from local inference feasibility - [references/quantization-format-table.md](references/quantization-format-table.md) - GGUF Q4_K_M/Q5_K_M/Q8_0, AWQ, GPTQ, FP8, EXL2; KV-cache quantization; speculative decoding; NPU/accelerator tier; decision tree - [references/small-model-tier-table.md](references/small-model-tier-table.md) - Gemma, Phi, Qwen small-instruct capability tiers: footprint, strengths, selection heuristic (versions hedged — verify current release) - [data/sources.json](data/sources.json) - local-model tooling sources from the curated repo list **Templates** - [assets/templates/ollama-setup-recipe.md](assets/templates/ollama-setup-recipe.md) - end-to-end Ollama install, model pull, OpenAI-compatible API usage, Modelfile pinning - [assets/templates/openwebui-deployment-recipe.md](assets/templates/openwebui-deployment-recipe.md) - Docker / Compose deployment of Open WebUI over Ollama, env vars, reverse proxy, upgrades **Related Skills** - [../ai-llm-inference/SKILL.md](../ai-llm-inference/SKILL.md) - serving, quantization, routing, and throughput tuning - [../ai-llm/SKILL.md](../ai-llm/SKILL.md) - full LLM lifecycle and fine-tuning strategy - [../ai-mlops/SKILL.md](../ai-mlops/SKILL.md) - deployment, monitoring, and governance - [../software-ai-integration/SKILL.md](../software-ai-integration/SKILL.md) - product integration and AI UX ## Fact-Checking - Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact. - Start from `data/sources.json` for local-model tooling references. - Verify current model support, install steps, and hardware caveats before giving time-sensitive recommendations. - If web access is unavailable, mark runtime-specific claims as unverified. ## Learnings Loop Before applying this skill on a non-trivial task, read `learnings.consolidated.md` in this directory (and `learnings.md` if present). After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to `learnings.md` via `agents-skills-feedback-loop/scripts/append_learning.py`. Do not modify `SKILL.md` itself.