MarkMyAss

Illustration: a confident pirate navigator at the ship's wheel beside a large brass compass, sailing clear open water

Sail your own stack.

Run Models Locally#

A practical guide to avoiding provider-side provenance at the source#

MarkMyAss's main tool cleans up supported metadata and provenance signals after a file already exists — it inspects, cleans, and verifies what a hosted AI provider (or any tool) left behind. That's useful, but it's downstream of the problem.

There's another path upstream of it: run the model yourself. If you generate text or images with an open-weight model under your own inference stack, you are no longer dependent on a hosted provider's own generation pipeline, and whatever that provider's service adds at generation time or file-export time simply never happens in the first place.

This page is a practical, no-hype guide to that path: what "open-weight" actually means, what's realistic to run given your hardware and budget, which current models and tools are worth using, and when renting a GPU beats buying one. It's written for developers, not for people looking for a five-minute miracle.

← Back to the MarkMyAss cleaner

Hosted vs. local: what actually changes#

Hosted closed models (Anthropic Claude, OpenAI GPT, Google Gemini, and similar)#

These are usually the best all-around choice for most people: frontier quality, no setup, no hardware, and the provider handles scaling and reliability for you. The tradeoff is that you are calling someone else's pipeline. The provider controls how your request is processed, what's logged, and what (if anything) is embedded in what comes back -- including any provenance or watermarking signal the provider chooses to attach, such as C2PA Content Credentials on generated images, or a statistical bias in how the model samples tokens for text (see MarkMyAss's main page for why that specific mechanism can't currently be verified by anyone outside the provider).

Open-weight models, run yourself#

"Open-weight" means the trained model's parameters are published for anyone to download and run -- on your own laptop, your own server, or a rented GPU. You control the entire stack: the inference engine, the runtime, and the output pipeline. Nothing is added to your output that you didn't put there yourself, because there is no hosted service in the loop at generation time.

Open-weight is not the same as "open source." Some releases (DeepSeek, Qwen, Mistral's newer models, Microsoft's Phi, OpenAI's gpt-oss) use the permissive, OSI-recognized Apache 2.0 or MIT licenses. Others (Meta's Llama family, Google's Gemma) ship under custom terms that the Open Source Initiative does not consider open source -- they typically restrict use above a certain scale or add other conditions. Always read the specific model's license before shipping a product on top of it; a link is provided for every model below.

Can you run the best closed models locally?#

No. The frontier closed models from Anthropic, OpenAI, and Google are not published as downloadable weights, and there is no legitimate way to run them outside those providers' own infrastructure. What you can do is run the strongest open-weight models, which have closed most of the practical quality gap for everyday engineering, writing, and reasoning work -- but a fair comparison still generally favors the current frontier closed models on the hardest reasoning and long-context tasks. Judge for your own use case; don't take anyone's marketing claim (including this page's) at face value -- benchmarks move fast and are easy to cherry-pick.

Decision matrix#

Option Privacy Cost Setup difficulty Performance Best for
Hosted closed model (API) Provider sees your data Pay per token, low upfront Trivial Frontier-level Most products, fastest path to shipping
Local small model (laptop/CPU) Fully private Free after hardware you already own Easy (Ollama/LM Studio) Basic assistant tasks Drafting, simple Q&A, offline use, learning
Local consumer GPU (8-24GB) Fully private One-time hardware cost Moderate Good 7B-30B-class models Coding assistants, private chat, iteration
Self-hosted server/workstation Fully private (your infra) Higher upfront, ongoing power/space Higher (multi-GPU, serving stack) Strong, larger models Small teams, persistent internal tools
Rented cloud GPU (hourly) Depends on provider's policies Pay per hour, no upfront cost Moderate (image/template based) Up to frontier-class open models Occasional heavy workloads, experimentation, large models

Treat this as a starting point, not a verdict -- the right column changes based on how often you actually need the compute.

Recommendations by hardware and budget#

A. No dedicated GPU (laptop, integrated graphics only)#

Realistic today: small models in the 1B-4B parameter range, quantized (4-bit/Q4), run on CPU via Ollama or LM Studio. Expect noticeably slower generation than a hosted API and weaker reasoning than larger models. Good for: drafting, simple rewriting, offline note-taking assistants, and learning the tooling before investing in hardware. Not a realistic substitute for a frontier hosted model on hard tasks.

B. Consumer GPU#

VRAM is the constraint that matters most -- not the GPU's raw compute. As a rule of thumb for 4-bit (Q4) quantized models: roughly 0.5-0.7 GB of VRAM per billion parameters, plus some headroom for context/KV cache. That rule of thumb is approximate and varies by quantization scheme and context length -- see llama.cpp's quantization docs for specifics.

VRAM Realistic model class (Q4) Notes
8 GB 7B-8B dense models Comfortable at moderate context; tight above ~8K tokens
12 GB 8B-14B dense models Good sweet spot for coding assistants
16 GB up to ~20B dense, or small MoE (e.g. gpt-oss-20b-class) Solid daily-driver tier
24 GB up to ~30-34B dense, larger MoE Best consumer tier for serious local coding work

C. Higher-end local workstation (32GB+ system RAM, prosumer/multi-GPU)#

Multi-GPU setups (e.g. dual 24GB cards) or a single 48-80GB professional card open up 70B-class dense models and larger mixture-of-experts (MoE) models at usable quantization. This is where you can realistically run models like Qwen3 at large sizes or gpt-oss-120b (OpenAI states it runs on a single 80GB GPU). Setup complexity rises: you're usually moving from Ollama/LM Studio to vLLM or llama.cpp directly for proper multi-GPU support and serving throughput.

D. Mini PCs / compact local setups#

Mini PCs with unified memory (e.g. Apple Silicon Mac mini/Studio, or GPU-equipped mini workstations) can be a genuinely good fit when your bottleneck is memory capacity rather than raw throughput -- Apple Silicon's unified memory lets a modest-looking machine hold a surprisingly large quantized model, at the cost of lower tokens/second than a discrete high-end GPU. They make less sense if you need high-throughput multi-user serving or training/fine-tuning workloads -- that's better served by a discrete-GPU workstation or a rented instance.

E. Rent instead of buy#

Renting a cloud GPU by the hour makes more sense than buying hardware when: you need a large model (70B+ dense, or big MoE) only occasionally; you're experimenting and don't yet know what you need long-term; you want to avoid the capital cost and depreciation of GPUs that will be obsolete in 2-3 years; or your workload is bursty (batch jobs, agent runs, fine-tuning) rather than constant. See the dedicated section below.

Model rankings and exact point releases change every few months -- that's normal and expected. Rather than chase a leaderboard snapshot, this list favors model families with stable, official homes you can check for the current best release yourself. Verify licenses before commercial use; links go to the official source.

Coding#

General reasoning#

Lightweight / consumer-hardware-first#

Larger / server-grade deployment#

Tools, runtimes, and installation paths#

Tool Best for License Link
Ollama Easiest way to get a model running locally with an OpenAI-compatible API MIT ollama.com
LM Studio GUI-first, model browser with hardware-aware quantization suggestions Free proprietary app (built on open runtimes) lmstudio.ai
llama.cpp The engine underneath most of the local-AI ecosystem; best for maximum control and unusual hardware (Apple Silicon, CPU-only, etc.) MIT github.com/ggml-org/llama.cpp
vLLM Production/multi-user serving on NVIDIA or AMD GPUs, much higher concurrent throughput than the single-user tools above Apache 2.0 github.com/vllm-project/vllm
text-generation-webui (now "TextGen") Full-featured local desktop app: chat UI, multiple backends, fine-tuning AGPL-3.0 github.com/oobabooga/textgen

Quick starts (illustrative -- check each project's own docs for the current install method)#

Ollama (macOS/Linux/Windows):

# Install from https://ollama.com/download, then:
ollama run qwen3

llama.cpp (build from source, works everywhere including CPU-only):

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release

vLLM (Python, NVIDIA/AMD GPU serving):

pip install vllm
vllm serve <model-name-or-path>

These commands are illustrative starting points, not guaranteed to be exactly current -- always follow the linked project's own install instructions, which are updated far more often than this page can be.

Renting GPUs instead of buying#

Renting makes the most sense for occasional heavy workloads: experimenting with a model class you don't yet own hardware for, running a large model (70B+ dense or big MoE) for a batch job, agent runs that spike compute briefly, or fine-tuning -- all without the capital cost of GPUs that depreciate quickly.

Reputable, commonly used options as of this writing:

On pricing: hourly GPU rental prices change often and vary a lot by GPU model, region, and provider tier (and, for marketplaces like Vast.ai, by which specific host you pick). Rather than freeze numbers here that will be stale in weeks, check each provider's own pricing page directly before committing.

Tradeoffs versus buying hardware:

Suggested paths by user type#

If you just want something easy: install Ollama and run a mid-size model from the Qwen3 or Llama family. Minutes to working, no GPU required (though one helps a lot with speed).

If you want the cheapest local option: a small quantized model (Gemma 3 or Qwen3 in the 1B-4B range) via Ollama or llama.cpp on hardware you already own.

If you want the best coding performance under a reasonable budget: a consumer GPU with 16-24GB VRAM running a Qwen3-Coder or gpt-oss-20b class model via Ollama, LM Studio, or llama.cpp directly.

If you want stronger privacy with minimal hardware investment: a small model (Gemma 3, Phi, or Ministral 3) on a laptop CPU or modest GPU -- fully local, no rented infrastructure involved at all.

If you need bigger models but don't want to buy a GPU: rent one -- start with RunPod or Vast.ai for occasional use, or Lambda if reliability and support matter more to you than the lowest hourly rate.

What this page does not claim#

Sources#

Is something outdated or inaccurate?#

This page covers a fast-moving ecosystem and we'd rather correct it than leave it stale. Open an issue or submit a pull request against src/ghostmark/web/content/run_local.md on GitHub.

Last reviewed: 2026-08-13