Last modified: 2026-09-13 00:11:34 (PDT)
Cloud model providers offer state-of-the-art capability, but privacy constraints, HIPAA data handling, air-gapped environments, and cost considerations often require running models locally or offline. This chapter covers running coding agents without outside network calls, connecting open harnesses to local Ollama and OpenRouter endpoints, steering Claude Code with non-Anthropic models, and evaluating small local models for autonomous coding loops.
Some environments restrict or prohibit internet access—high-performance computing (HPC) clusters, hospital networks, or air-gapped research servers may block connections to cloud AI providers. Running a local AI model lets you use coding assistance in these settings without sending code to external servers, which also addresses data-privacy concerns when working with sensitive or confidential data.
Trade-offs of Local vs. Cloud Models
Local models work best with a GPU (roughly 8 GB VRAM or more for smaller models); CPU-only inference is possible but significantly slower, and hardware needs vary with model size and quantization. They are generally less capable than frontier cloud models, and may produce lower-quality results on complex tasks. For routine work in fully connected environments, cloud-based agents remain the better choice. Use local models when network access or data-privacy policies require it.
Ollama is a common way to run open-weight AI models locally. It packages models and a simple API server into a single tool and is available for macOS, Linux, and Windows.
Install Ollama:
Caution
Before running any remote install script, review it first. The real risk is piping curl straight into sh/bash (curl ... | sh), which executes unreviewed code. Save the script, read it, then run it: curl -fsSL https://ollama.com/install.sh -o install.sh && less install.sh (paging the saved file rather than piping curl into less, which can behave oddly in some terminal emulators). Alternatively, use your system package manager (e.g., brew install ollama on macOS) or follow the manual installation steps on the Ollama releases page.
On Windows, download the installer from https://ollama.com/download.
Pull a code-focused model:
The VRAM each model needs depends on its size and quantization and changes as models are re-quantized—check the Ollama model library for current requirements. As a rough guide, smaller (7B) models run on consumer GPUs with around 8 GB of VRAM, while larger (32B and 70B) models need substantially more and may not fit on a single GPU.
A good completion model is not necessarily a usable agent
The models above are strong at writing code when you ask them to. That is a different skill from tool calling — emitting a well-formed request to read a file or run a command, and then using the result. Inline completion and chat need only the first. Anything autonomous needs the second, because an agent that cannot call a tool cannot read your repository at all.
The two come apart in practice, and the failure is subtler than a model simply refusing. Tested against a single-function tool schema on a 24 GB M2, qwen2.5-coder:14b returned finish_reason: stop with an empty tool_calls field on four attempts out of four. It had not ignored the request: it wrote a correct tool call as ordinary prose in the content field,
which is the right JSON in the wrong place. A harness looks for tool_calls, finds nothing, and treats the turn as a plain reply, so the tool never runs. granite4:7b-a1b-h, at half the parameter count, returned a well-formed call in the tool_calls field three times out of three and completed a full multi-turn round trip.
An advertised tools capability is necessary but not sufficient, so do not settle the question with ollama show. On the same machine qwen2.5-coder:14b lists tools among its capabilities and still cannot be driven by a harness, for the reason above. Test it yourself with one request before building a loop on it:
RESP=$(curl -s -w '\n%{http_code}' \
http://localhost:11434/v1/chat/completions -H 'content-type: application/json' -d '{
"model": "granite4:7b-a1b-h", "stream": false,
"messages": [{"role": "user", "content": "What is in src/main.py? Use the tool."}],
"tools": [{"type": "function", "function": {"name": "read_file",
"parameters": {"type": "object", "properties": {"path": {"type": "string"}},
"required": ["path"]}}}]}')
CODE=$(printf '%s' "$RESP" | tail -1)
BODY=$(printf '%s' "$RESP" | sed '$d')
if [ "$CODE" != "200" ]; then
echo "request failed (HTTP $CODE): $BODY" # bad tag, tools unsupported, server down
elif printf '%s' "$BODY" | grep -q '"tool_calls"'; then
echo "usable as an agent"
else
echo "no tool call; completion model only"
printf '%s' "$BODY" | grep -o '"content":"[^"]*"' | head -c 200
fiCheck the status code separately from the result. A request that simply errored — a mistyped tag, a model the server rejects for tool use, a server that is not running — produces the same silence as a model that declined to call the tool, and only one of those is a fact about the model. Printing the content field on the no-call branch is what distinguishes a model that ignored the tools from one that described the call in prose instead of emitting it, which is the qwen2.5-coder case above.
Start the Ollama server:
By default the server listens at http://localhost:11434.
Positron supports Ollama natively through the OpenAI-compatible API endpoint that Ollama exposes.
Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows/Linux).http://localhost:11434/v1 and leave the API key blank, or, if the client rejects an empty field, enter any placeholder value such as ollama.qwen2.5-coder:7b).Once configured, Positron Assistant will send requests to your local Ollama server instead of a cloud provider.
Continue is an open-source VS Code extension that supports Ollama and many other local and cloud backends.
Continue provides inline completions and a chat panel, similar to GitHub Copilot, but routed entirely to your local model.
The editor integrations above provide inline completion and chat. For an autonomous agent that reads your files, proposes edits across a whole repository, and commits them to git—the local counterpart to a cloud coding agent—aider works directly against Ollama.
Install it in an isolated environment so its dependencies do not collide with other tools:
Point it at Ollama and choose a model with the ollama_chat/ prefix, which gives better results in aider than the plain ollama/ prefix:
Raise the Ollama context window
Ollama’s default context window is small, which silently truncates your code and makes the model look far less capable than it is. This is the single most common mistake when pairing aider with Ollama.
The default is not a fixed number. Ollama picks it from the memory it detects, as its own ollama serve --help states:
OLLAMA_CONTEXT_LENGTH Context length to use unless otherwise specified
(default: 4k/32k/256k based on VRAM)Do not assume you landed in a generous tier. A 24 GB Apple-silicon machine gets 4096 tokens, not the 32k its total memory suggests, because only about 75% of unified memory is addressable by the GPU and the tier boundary sits above that share. Check what you actually got rather than inferring it — ollama ps prints the context of each loaded model:
There are three ways to raise it, and they differ in which clients they reach:
OLLAMA_CONTEXT_LENGTH on the server, which sets the default for everything. Note that the macOS menu-bar app starts the server with its own environment, so exporting the variable in your shell does not reach it; this route applies when you run ollama serve yourself.
A num_ctx parameter sent per request, which is what aider does through ~/.aider.model.settings.yml:
A Modelfile that bakes the context into a derived model, which is the only one of the three that reaches clients that cannot send num_ctx themselves:
The derived model shares weight blobs with its base, so it costs no extra disk.
Context is not free. Raising a 14B model from 4k to 32k on a 24 GB M2 took its resident size from 9.5 GB to 15 GB, about 5.5 GB of key-value cache, so pick the largest value that still leaves the weights and the cache in GPU memory and confirm with ollama ps that PROCESSOR still reads 100% GPU.
To avoid passing flags every time, set defaults in a config file at ~/.aider.conf.yml:
Keep instruction files lean for small local models
While cloud frontier models handle tens of thousands of tokens of system prompt easily, small local models (1.5B–7B) lose speed and instruction fidelity when loaded with large instruction files. A multi-page documentation bundle (such as a 95 KB CLAUDE.md) will consume most of an SLM’s active context window and cause severe prompt-ingestion delays. For local SLMs, point read at a concise, focused summary file (such as AGENTS.md or a project-specific rules snippet) rather than the full multi-tool configuration suite.
While aider is primarily used from the command line, it includes a built-in browser-based GUI:
This launches a local web application in your default browser with:
You can bring Aider directly into VS Code or Positron through three workflows:
aider in the built-in terminal (Ctrl+`). Edits and Git commits made by Aider immediately reflect in your editor tabs.aider --gui in the terminal, open the Command Palette (Cmd+Shift+P), and run Simple Browser: Show with http://localhost:8501 to dock the Aider interface side-by-side with your code.aider can also split the work between two models in “architect” mode: a larger model plans the change (the architect), and a second model applies the edits (the editor).
This can improve results on multi-step changes, but on a machine without a strong GPU it roughly doubles the time per turn, because the two models take turns and their weights are swapped in and out of memory. Reserve it for genuinely tricky changes; for small edits, a single model is faster.
Set edit_format: whole for a small model
aider asks the model to express an edit either as a SEARCH/REPLACE block (diff, the default for most models) or by rewriting the file (whole). Producing an exact SEARCH/REPLACE block is a demanding format, and small models are unreliable at it.
Measured on a 24 GB M2 with granite4:7b-a1b-h at 32k context, same prompt each time:
| File | edit_format: diff |
edit_format: whole |
|---|---|---|
| One function | 3 of 3 correct | 3 of 3 correct |
| Two functions, one to be left alone | 0 of 3 correct | 3 of 3 correct |
The two-function failure is worth dwelling on, because it is not the failure you would expect. The model did not refuse the edit or produce a broken file. In all three diff runs it fixed the target function correctly and silently deleted the other one, then committed with a message naming only the intended fix. Nothing in the commit message, the exit status, or the model’s own summary mentioned the deletion.
That is the hazard of an unattended loop in its most concrete form: a step that reports success while destroying work, leaving a wrong state as the premise for every step after it. It is also why committing after every step matters — git held the original, so the damage was one git revert away rather than lost.
Set the format per model:
whole costs more tokens per edit, since the model rewrites the whole file, which is a real cost on large files. Larger models generally handle diff correctly; re-measure rather than assuming either way.
Ollama also serves an Anthropic-compatible endpoint at /v1/messages, alongside the OpenAI-compatible one used above. Any client that speaks the Anthropic API can therefore be pointed at a local model, including Claude Code itself, with no proxy in between. Check that the endpoint answers before wiring anything to it:
Point Claude Code at it with three environment variables:
Set these in a wrapper script rather than in your shell profile, so that plain claude keeps using the cloud and a separate command uses the local model.
A heavyweight harness can defeat a small model
This works, but expect worse results than the same model gives through aider, and understand why before blaming the model.
A harness spends context before your task does. Claude Code sends a long system prompt and a schema for every tool it exposes, and any Model Context Protocol (MCP) servers you have configured add their own schemas on top. Against a 32k local context that overhead is a large fraction of the budget, and a small model handles it poorly.
Observed on a 24 GB M2 with granite4:7b-a1b-h, asking only that it read one file and comment on one function:
Read, Grep, and Glob, it still ignored the question and asked what it should work on.The same model, same context, through aider, fixed a real bug and committed it in 17 seconds. Swapping in a 12B model produced no answer at all in ten minutes, because processing that much prompt at 10 tokens per second is simply slow.
Two practical rules follow. Shrink the tool surface a local model is shown — --strict-mcp-config --mcp-config '{"mcpServers":{}}' loads no MCP servers, and --allowed-tools narrows the built-ins. And tell the harness the real context size, since Claude Code assumes a 200k window for a model it does not recognize and would let the conversation grow far past what the model can hold:
For autonomous work on a small local model, prefer a light harness such as aider. Reserve this route for using a familiar interface offline, not for getting the best out of the hardware.
Air-gapped work aside, the common case is a laptop that is usually online but sometimes is not—on a plane, behind a flaky hospital network, or temporarily rate-limited by a cloud provider. You can keep a coding agent working across these gaps by putting a cloud model and a local model behind one endpoint and falling back automatically.
LiteLLM runs a small local proxy that presents a single OpenAI-compatible endpoint. You give it a primary model and one or more fallbacks; when the primary fails with a retryable error—a rate-limit response (HTTP 429), or a connection error when you are offline—it retries the request on the next model. Pointing your agent at the proxy instead of directly at a provider makes the cloud-to-local switch automatic and invisible to the tool.
Install the proxy:
Create a config file (for example, ~/.litellm/config.yaml) with a cloud primary and a local fallback:
model_list:
# Cloud primary --- replace with your provider and a current model id
- model_name: coder
litellm_params:
model: anthropic/YOUR-MODEL-ID
api_key: os.environ/ANTHROPIC_API_KEY
# Local fallback, served by Ollama
- model_name: coder-local
litellm_params:
model: ollama_chat/qwen2.5-coder:7b
api_base: http://localhost:11434
litellm_settings:
num_retries: 2
fallbacks:
- coder: ["coder-local"]Run the proxy, which listens on http://localhost:4000:
Then point your agent at the proxy. Because the proxy speaks the OpenAI API, most tools accept it as a custom endpoint. For aider:
Requests now go to the cloud model when it is reachable and fall back to the local model on a rate limit or when you are offline.
A cloud API key is separate from a chat subscription
The cloud side needs an API key billed per token, issued from the provider’s developer console. A chat subscription such as Claude Pro or a Copilot seat is not an API key and cannot be used here. Omit the cloud entry entirely to run local-only, or add the key later to enable the hybrid.
Keep the proxy on loopback
By default the proxy binds to localhost, which is what you want. Do not expose it on 0.0.0.0 on a shared or untrusted network without authentication, because anyone who can reach the port can spend your cloud API key.
Many HPC clusters do not have outbound internet access on compute nodes but do allow access on login nodes.
Install the Ollama binary on the cluster first
Ollama must be installed on the cluster (on the host that will run ollama serve) before any of the steps below. On most HPC systems you do not have root access, so the curl | sh installer may fail or install to the wrong place. Instead, check whether your cluster already provides it (e.g., module load ollama), ask your HPC administrators, or download a static binary from the Ollama releases page and place it on your PATH.
A useful pattern:
Pre-pull models on a login node or a machine with internet access, then copy the model files to the cluster:
Watch your home-directory quota
Model files are large—qwen2.5-coder:7b is ~4 GB and qwen2.5-coder:32b is ~20 GB—and most HPC home directories have tight quotas (often 10–50 GB). Filling your home directory can break other jobs. Redirect model storage to a scratch or project filesystem with OLLAMA_MODELS and rsync to that path instead:
Set the same OLLAMA_MODELS value before running ollama serve so the server finds the models.
Start Ollama on a compute node (or an interactive session) using the pre-downloaded model files—no internet required. Set OLLAMA_HOST=0.0.0.0 so the SSH tunnel from the login node can reach the port:
If you are on a shared compute node, be aware that binding to 0.0.0.0 exposes the Ollama port to other users on that host. Scheduler policies vary by site and job type, so confirm whether your job has exclusive node access (request it explicitly when in doubt—e.g., --exclusive in SLURM), or bind only to loopback (OLLAMA_HOST=127.0.0.1:11434) and tunnel from the login node when the node is shared.
Forward the port to your local machine to use your editor’s Ollama integration. Because Ollama is running on a compute node (e.g., gpu-node-01), forward through the login node to that specific host:
This terminal must stay open for as long as you use the editor’s Ollama integration—closing it tears down the tunnel and silently drops the connection. Alternatively, start the tunnel in the background (non-interactive) so it does not occupy a terminal:
(-N runs no remote command, -f backgrounds ssh after authenticating.) To stop the tunnel later, match the full SSH command rather than a bare port string—pkill -f "ssh.*-N.*11434:gpu-node-01"—so you don’t accidentally kill unrelated processes whose command line happens to contain that port. Safer still, note the PID when you start it (pgrep -f "11434:gpu-node-01") and kill that PID directly.
Then configure your editor to use http://localhost:11434/v1 as the base URL.
SLURM and GPU Allocation
If running Ollama on a SLURM-managed cluster, request a GPU node with enough VRAM for your chosen model and load any required CUDA modules before starting ollama serve. See the UCD-SERG Lab Manual’s SLURM chapter for guidance on requesting GPU resources.
Running a model locally ensures that your code and prompts never leave your machine or cluster. This is important when working with:
Even with local models, avoid including raw sensitive data in prompts. Work with anonymized or synthetic data wherever possible.
“Local” is not automatic—some model tags route to a cloud
Running Ollama does not by itself guarantee that a prompt stays on your machine. Ollama can serve cloud-hosted models alongside local ones, and those are the models too large to run on a laptop at all, which is exactly when a tag is tempting. A cloud-routed tag looks much like a local one in everyday use.
Two habits keep this honest, and they matter most in precisely the settings that motivated running locally:
Pull and reference explicitly local tags, and treat a tag with no listed download size as cloud-routed until you check its own page in the Ollama model library.
Disable the cloud path outright when the data is regulated, so the guarantee does not depend on remembering which tag is which:
Verify rather than trust either one. Cut the machine off the network, or block outbound traffic, and confirm the agent still completes a real task — the check described under Verifying you are genuinely offline below. A setup that quietly depended on a cloud endpoint fails that test immediately.
A local setup that has never been tested without a network is a local setup you are guessing about. Cutting the machine off entirely is the honest test. A lighter one that does not disturb the rest of your session is to make outbound traffic fail for a single command, while leaving localhost reachable:
export HTTPS_PROXY=http://127.0.0.1:9 HTTP_PROXY=http://127.0.0.1:9
export NO_PROXY=localhost,127.0.0.1
# Confirm the block is real before trusting the result:
curl -s -m 5 -o /dev/null -w '%{http_code}\n' https://example.com # 000 = blocked
curl -s -m 5 -o /dev/null -w '%{http_code}\n' http://localhost:11434/api/version # 200 = local
aider --yes --message "Fix the off-by-one error in mean()." stats.pyCheck the block itself first, as above. A test that passes because the proxy was never applied tells you nothing, and looks exactly like success.
Issue #46 asked whether two GitHub projects, Aeris and Graft, have a place in the lab’s AI workflow. The notes below reflect each repository’s README and metadata as read on 2026-09-09.
Cedrick-Coto/Aeris (Coto 2026) describes itself as a deterministic cognitive simulation engine with emergent narrative, whose README describes an entity-component-system (ECS) architecture in C# (.NET 10). Its design premise is that the language model “verbalizes, never thinks”: a deterministic simulation core computes perception, attention, memory, affect, and goals each tick, and the LLM only turns that state into narrative or dialogue, never modifying it. The target application is a simulated character in a Pokemon world.
Repository facts (measured 2026-09-09):
.csproj project files on main, but no .cs source files, so the “complete” engine and “210 tests” that the README roadmap reports are not present in the public repositoryThe project is a research design document for LLM-backed simulated agents, not a tool for working with AI coding assistants. Nothing in it targets writing, reviewing, or running code.
Aeris does not overlap with any lab workflow. Its one transferable idea, keeping the LLM out of state changes and confining it to presentation, is already how this site recommends treating agents: the responsibility-for-validation policy and the review guidance put the deterministic checks and the accountability with the human and the CI pipeline, not the model. The GPL-3.0 license would also complicate reuse of any code in lab packages, though at the reading date there was no code to reuse. No further action is warranted.
trailhq/Graft (NanoNets and Trail contributors 2026) (formerly NanoNets/Graft; the old URL redirected at the reading date) is an open-source “context layer” for coding agents. Its aim is to stop an agent re-exploring a repository at the start of every task by building a persistent map of the codebase once and feeding the relevant parts of that map into each prompt.
Repository facts (measured 2026-09-09):
@nanonets/graft (0.16.0 on npm; package.json on main reads 0.17.0), requiring Node 20 or laterHow it works, per the README:
graft build parses the repository with tree-sitter into a per-symbol code graph (functions, classes, call and import edges). This layer is deterministic, runs locally, and never calls a model, so it needs no API key.
graft build --deep adds an LLM layer: a short summary per file, grouped into a few dozen Markdown “nodes” (one per subsystem or concept) with typed links between them and a “crux” excerpt of the lines that carry the logic. The model runs under your own key through one of:
graft init wires the graph into an agent. For Claude Code it writes a skill file, hooks, and a status line. For the other hosts it adds a fenced section to AGENTS.md, GEMINI.md, or .github/copilot-instructions.md:
It also registers an MCP server exposing six tools (find code, file API, trace callers, regex search, repo map, freshness check).
The graph is a git-ignored local cache, rebuilt incrementally and refreshed before each query; only the small wiring is committed.
R is one of the “full-fidelity” languages, with support for plain functions, S3, S4, and R6 classes and methods, roxygen @export tags, and library() and source() imports.
The README reports benchmarks from the project’s own harness (42% fewer tokens and 60% less latency with equal correctness on 162 runs across two repositories) and a 50-instance SWE-bench Verified run, Claude Code with Claude Sonnet 5 on both arms (33 of 50 resolved with Graft against 27 without, using 23% fewer tokens). These are vendor-run measurements on the vendor’s chosen tasks, so treat them as an upper bound until reproduced. The package sends an anonymous batched usage ping by default; the README states the ping carries no code, paths, or queries, and that graft telemetry disable or DO_NOT_TRACK=1 turns it off.
Graft addresses a cost the lab pays constantly: each new agent session re-reads an R package to rebuild the same picture of its structure. Three things make it a good fit for a trial:
Check two things before adopting it lab-wide. First, the benchmark claims are vendor-run, so measure the token and tool-call counts on one of our own packages before and after graft init. Second, the --deep layer spends model tokens on every changed file and the init step for Codex writes to user-level config outside the repository (~/.codex/; skip with --no-global), so start with the structural layer only and the Claude Code wiring only. The graft init --dry-run flag lists every file it would touch, which is the right first command. Graft is complementary to the memory tooling in Magic Context: that section covers remembering what happened across sessions, whereas Graft covers what the code is, and the two do not overlap.
OpenCode is an open-source coding agent that runs in your terminal, reads your project, edits files, and runs commands. It supports local models through OpenAI-compatible providers.
This section assumes Ollama is already installed and that you have pulled a code-focused model — see Section 1 for both, including the Linux and Windows install paths.
Verify the server is running:
On macOS, brew services start ollama registers a launchd agent so the server comes back automatically at login rather than needing a manual start each session.
Install the model-discovery plugin:
Rather than hand-coding each model into opencode.json, use the opencode-local-ollama plugin, which discovers your local Ollama models automatically on startup:
This writes to ~/.config/opencode/opencode.json:
Restart OpenCode and run /models to see your local models listed alongside any cloud providers. The plugin reads from Ollama’s /api/tags and /api/show endpoints, so newly pulled models appear on the next restart with no config edits.
Alternative: multi-backend local provider
If you also run LM Studio, llama.cpp, or vLLM alongside Ollama, the opencode-local-provider plugin auto-detects all of them under a single local provider and probes each at runtime for loaded models. Install it with opencode plugin --global opencode-local-provider.
A lightweight hand-written provider block in your project’s opencode.json still works if you prefer explicit control over model names and context limits, but the plugin removes the need to keep that list in sync with ollama pull.
OpenRouter is a gateway that exposes hundreds of hosted models — Claude, GPT, Gemini, DeepSeek, Qwen, Kimi, Llama, and more — behind a single API key and billing account. OpenCode treats it as a built-in provider, so its catalog appears in the /models picker alongside local models (Section 3). The catalog changes frequently; model IDs below were verified against it in August 2026.
Connect an API key:
:free ID suffix and cost nothing, at the price of tight rate limits./connect, select OpenRouter, and paste the key. The CLI command opencode auth login does the same thing outside the TUI. Either way the key is stored in ~/.local/share/opencode/auth.json, never in opencode.json./models, filter for openrouter, and pick a model.Pick a model that supports tool calls:
Coding agents drive every action — reading files, editing, running commands — through tool calls. Image-generation, speech, and embedding models have no endpoints that support tool use, so an agent session fails on them immediately with No endpoints found that support tool use. Prefer chat or coder variants such as anthropic/claude-sonnet-4.5, deepseek/deepseek-chat, or qwen/qwen3-coder.
One trap worth naming: on OpenRouter, Google lists Gemini 3 Pro only as image-output variants (google/gemini-3-pro-image, google/gemini-3-pro-image-preview) — there is no plain google/gemini-3-pro entry — so those image models are easy to pick by mistake. For tool-calling work, use one of the Gemini Flash chat variants instead, such as google/gemini-3-flash-preview.
Models are addressed as openrouter/<vendor>/<model>, for example openrouter/deepseek/deepseek-chat.
Optional configuration in ~/.config/opencode/opencode.json (or the project-level file):
model pins the session default; without it, OpenCode starts each session on its own built-in default
small_model sends housekeeping tasks (session titles, summaries) to a cheap model instead of a frontier one
entries under provider.openrouter.models add models that are not preloaded, or pin routing: OpenRouter load-balances across upstream hosts by default, and order restricts requests to named providers (provider-selection docs)
Config loads at startup, so restart OpenCode after editing it.
Claude Code is built around the Anthropic Messages API, and its documentation describes how to point the harness at any endpoint that speaks that format (measured 2026-09-01). That mechanism is what makes it possible to run Claude Code against models Anthropic does not make, and the same documentation says plainly that doing so is unsupported. This section summarizes:
A fast-moving snapshot
The configuration claims in this section were checked against the Claude Code documentation on that date. The community practice and the policy history come from the research summary in issue #96, compiled from press coverage and forum discussion up to August 2026; Anthropic revised the subscription rules more than once during 2026. Re-read the current gateway documentation and the consumer terms before relying on any of it.
Anthropic’s gateway documentation is written for organizations that run their own gateway in front of Claude models, so that credentials, usage tracking, cost controls, and audit logging live in one place. It states that “any gateway that exposes a supported API format works,” and in the same paragraph that Anthropic “doesn’t endorse, maintain, or audit third-party gateway products, and doesn’t support routing Claude Code to non-Claude models through any gateway.”
The supported non-default backends are all Claude models on other clouds:
CLAUDE_CODE_USE_BEDROCK=1)CLAUDE_CODE_USE_VERTEX=1)CLAUDE_CODE_USE_FOUNDRY=1)Routing to a different model family is the unsupported case. It works because the gateway mechanism is provider-agnostic, not because Anthropic tests it.
The variables that matter, from the connection guide, the model configuration page, and the environment-variable reference:
ANTHROPIC_BASE_URL points the harness at a gateway. The documentation is explicit that it “changes where requests are sent, not which model answers them.”ANTHROPIC_AUTH_TOKEN (sent as Authorization: Bearer) or ANTHROPIC_API_KEY (sent as x-api-key) carries the credential. ANTHROPIC_AUTH_TOKEN takes precedence over a saved claude.ai login immediately; ANTHROPIC_API_KEY takes over after a one-time approval prompt in interactive mode.ANTHROPIC_MODEL, and the ANTHROPIC_DEFAULT_SONNET_MODEL, ANTHROPIC_DEFAULT_OPUS_MODEL, ANTHROPIC_DEFAULT_HAIKU_MODEL, and ANTHROPIC_DEFAULT_FABLE_MODEL family, map the built-in aliases to whatever model IDs the backend accepts.CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1 makes Claude Code query the gateway’s /v1/models endpoint at startup and add the results to the /model picker.ANTHROPIC_CUSTOM_MODEL_OPTION (with optional _NAME and _DESCRIPTION) adds a single custom row to the picker; Claude Code skips validation for that ID, so any string the endpoint accepts works.CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 stops Claude Code from sending pre-release request fields and beta headers. The connection guide lists it as the fix for 400 errors naming context_management or Extra inputs are not permitted, which is what a non-Anthropic upstream returns when it rejects Claude-specific fields.CLAUDE_CODE_MAX_CONTEXT_TOKENS declares the context window for a model ID Claude Code does not recognize; Section 1 shows it in use for a local model.Claude Code reads these at startup, either from the shell or from the env block of ~/.claude/settings.json.
The forum reports collected in the tracking issue fall into a few shapes, from fewest moving parts to most:
ANTHROPIC_BASE_URL plus the vendor’s key. It is the route those reports describe most often, and the one bundled with the vendors’ “coding plan” subscriptions.Treat any self-hosted gateway or community router as infrastructure: pin its version, and keep the official claude launcher as a fallback. The tracking issue records a 2026 incident in which malicious LiteLLM releases were briefly published to PyPI, which is the concrete reason for pinning.
The forum reports in the tracking issue agree on three seams:
400 errors and the CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS remedy in the configuration list above are the documented form of this, and the model configuration page adds that the context window Claude Code assumes for an unrecognized model ID may not match the real one.The summary also notes that benchmark gaps between the strongest open-weight coding models and Claude narrowed during 2026, while cautioning that the open-weight figures it saw were largely the vendors’ own launch numbers. Harness-level reliability is a separate question from benchmark score in any case.
Two rules from the gateway documentation settle most questions:
apiKeyHelper is active, “a developer’s claude.ai subscription isn’t used … and the subscription’s usage limits don’t apply.” That traffic is billed per token to whoever owns the credential the gateway forwards.ANTHROPIC_BASE_URL alone, without a gateway credential, does not replace the subscription; the saved login stays the active credential and its limits apply. The documented case for that is an organization’s own gateway forwarding to Anthropic, not a third-party harness serving other models.The policy history in the tracking issue concerns the other direction, using a Claude subscription login to drive third-party harnesses or proxies. As that summary records it, Anthropic announced in April 2026 that subscriptions would stop covering usage in third-party tools from 2026-04-04, with such usage drawing on separately purchased extra usage or an API key instead, while provisioning your own API keys or provider credentials, billed to the key owner, stayed allowed. The summary also notes that the terms were reworded afterwards, so read it as a pointer to what to check rather than as the current text.
| Goal | Suggested route |
|---|---|
| Model flexibility is the priority | A model-agnostic harness (OpenCode, Aider, Cline), which is built for it; see Section 3 and Section 4 |
| Claude Code’s skills and plugins with one cheaper model | The vendor’s Anthropic-compatible endpoint with your own key |
| Several models with team governance | A self-hosted gateway such as LiteLLM, or an enterprise gateway product, pinned as infrastructure |
| Per-request-type cost routing | Claude Code Router, pinned as infrastructure |
| Claude models at subscription prices | The official Claude Code CLI signed in with the subscription, which remains fully supported |
Whichever route you pick:
Section 1 covers the mechanics of running a model on your own hardware: installing Ollama, wiring up an editor, and driving aider against a local endpoint. This section is about a narrower and harder question sitting on top of that setup: which local model to pick, and how to let it work autonomously — making a sequence of edits, commits, and tool calls with no human approving each step — without the loop quietly going wrong.
A fast-moving, opinionated snapshot
As of August 2026, the open-weight coding-model landscape changes monthly: new releases, new quantizations, and new benchmark numbers appear faster than any static page can track. The model names, sizes, and figures below were verified against each model’s own listing at the time this section was written, not against benchmark round-ups, and they will drift. Re-check the source links before choosing a model for a new project, and re-benchmark on your own tasks rather than trusting a published score — your repository’s mix of languages and idioms is not the benchmark’s.
Small models make more per-step mistakes than frontier cloud models: a slightly wrong function signature, a hallucinated package, a test edited to pass instead of a bug fixed. A human working alongside a small model catches most of these immediately. An autonomous loop does not have that human in it, so a small error on step 3 becomes the premise for steps 4 through 40, and the mistakes compound rather than cancel out.
This is the reason small + local + fully autonomous is the hardest combination to run safely, and the reason this section spends most of its length on structure rather than on model selection. The fix is not a bigger local model — that only raises the error rate at which the same compounding problem starts to bite. The fix is bounding the loop so that a compounding error is caught and stopped early, covered under Guardrails for autonomy below.
Prefer a model explicitly trained for tool calling and agentic use over a general chat or plain code-completion model: an agentic loop depends on the model reliably emitting well-formed tool calls and stopping when it has finished a step, not only on writing plausible code. A handful of open-weight families currently fit that description well enough to run an autonomous loop against:
| Family | Sizes worth running locally | License | Best for |
|---|---|---|---|
| Llama 3.2 | 3B (2.0 GB), 1B (1.3 GB) dense | Llama 3.2 Community License | High-speed local subagents (\(<1\text{s}\) action turnaround); native structured tool calling |
| Qwen3-Coder | 30B-A3B mixture-of-experts (smallest tag, 19 GB) | Apache 2.0 | General-purpose agentic coding across languages |
| Qwen2.5-Coder | 1.5B, 3B, 7B, 14B, 32B dense | Apache 2.0 | Fast code generation (3B at \(\sim 45\text{ tok/s}\)). Verify tool-call formatting in your harness |
| Phi-4-mini | 3.8B (2.4 GB) dense | MIT | Strong multi-step reasoning in a compact footprint |
| Granite 4 | 7B-A1B mixture-of-experts (4.2 GB), 32B-A9B | Apache 2.0 | A small, fast tool-caller that fits where the tiers above do not |
| Granite 3.2 | 2B (1.5 GB) dense | Apache 2.0 | Ultra-lightweight IBM tool-calling model for single-task triage |
| Devstral Small | 24B | Apache 2.0 | Purpose-built for coding agents (multi-file edits, tool use) |
| Codestral | 22B | Mistral AI Non-Production License | Fill-in-the-middle completion, not redistribution in a product |
| DeepSeek-Coder-V2 | 16B (Lite) mixture-of-experts | DeepSeek Model License (commercial use permitted, own terms) | A capable, low-VRAM mixture-of-experts option |
| GLM-4.x | Flagship models are large MoE (over 100B total parameters) | MIT | Strong agentic benchmarks, but sized for a workstation or rented GPU, not a laptop |
Qwen3-Coder’s 30B-A3B tag is a mixture-of-experts model: 30B total parameters, but only about 3.3B active per token. VRAM at rest is set by the total, not the active count — every expert has to stay resident in memory even though only a fraction fires on any given token — which is why the tag still needs roughly 19 GB at 4-bit quantization, in line with its 30B total rather than its 3.3B active count. What the small active count buys is speed: inference runs closer to a 3–4B model’s pace despite the larger memory footprint. Codestral’s license is worth reading before you rely on it: Mistral’s Non-Production License permits local evaluation but not production or commercial deployment — fine for trying it out, not fine for a lab pipeline that runs unattended.
As a practical floor, treat the 24–32B tier at 4-bit quantization as the smallest size that holds up across a multi-step autonomous loop without frequent tool-call errors. Below that, a model is still useful as an assistant you supervise turn by turn (Section 1 covers exactly that setup), but it is not yet a safe choice to leave unattended.
Parameter count is the wrong first question
Treat that floor as a statement about sustained multi-step loops, not as a filter to apply before anything else. Size predicts tool-calling ability poorly enough that checking it first will mislead you.
Measured on a 24 GB M2 against a single-function tool schema, qwen2.5-coder:14b returned an empty tool_calls field on four attempts out of four, with finish_reason: stop each time. It wrote a correct tool call as prose in the content field instead, which no harness will act on. The 4.2 GB granite4:7b-a1b-h, at half the parameter count, put a well-formed call in tool_calls three times out of three and completed a multi-turn round trip using the result. The smaller model was usable as an agent where the larger one was not, and no amount of context or prompting fixes a model whose calls never reach the field a harness reads.
The advertised capability list does not settle it either. ollama show qwen2.5-coder:14b lists tools, and the model still cannot be driven by a harness, so treat the tag as necessary rather than sufficient.
So order the questions this way:
ollama ps.A small model that clears the first two is worth measuring on your own tasks before concluding it cannot be left unattended, because the guardrails below, not the parameter count, are what actually bound the damage from a bad step.
| VRAM (or unified memory) | Model tier | Autonomy |
|---|---|---|
| ~8 GB | 7–8B | Assistant only — keep a human reviewing every step |
| ~12–16 GB | 14–24B | Entry point for a bounded autonomous loop |
| ~24 GB+ | 30–32B, with context headroom | Comfortable autonomy at the practical floor above |
| Apple-silicon unified memory (32 GB+) | Same tiers as above, generally slower per token | Well suited to an overnight batch job where wall-clock time matters less |
These are rough guides, not guarantees: VRAM headroom for context length matters as much as VRAM for the weights themselves, and a long-running agentic loop accumulates a long conversation history that eats into that headroom as it runs. Check the current requirements on the model’s own listing (the Ollama model library states them per tag) rather than a rule of thumb, since quantization schemes change.
Apple unified memory does not map onto the VRAM column
Read the unified-memory row as its own scale rather than as the VRAM figures with a speed penalty attached. Two deductions come off the headline number before any model loads:
Together those rule out the 30–32B tier on a 24 GB Mac, even though the headline number matches the VRAM column. The smallest qwen3-coder tag is 19 GB, which exceeds the addressable ceiling on its own, leaving nothing for context. There is no smaller variant of it to fall back to.
The practical ceiling on 24 GB of unified memory is a 12–14B dense model at 32k context, or a mixture-of-experts model of similar footprint. Confirm with ollama ps after loading: PROCESSOR reading 100% GPU means it fits, and anything less means part of the model is on the CPU and the loop will be far slower than the tier table suggests.
When designing an interactive agent or subagent workflow, turn turnaround time dictates whether the tool feels responsive or unusable. In an autonomous or semi-autonomous loop, latency per action is governed by two phases:
\[\text{Action Latency} = \text{Time to First Token (Prompt Ingestion)} + \text{Tool-Call Generation (Decode)}\]
On consumer unified memory hardware (such as Apple M-series chips with \(\sim 100\text{ GB/s}\) memory bandwidth), the memory bus sets a hard theoretical ceiling on token generation speeds:
Prompt caching preserves latency across turns
In a multi-turn agent loop, the system prompt and accumulated history are resent on every iteration. Ensuring the inference server keeps the model resident in memory (keep_alive: -1 in Ollama) and maintains prompt KV-cache reuse drops Time-To-First-Token on subsequent turns from several seconds to under \(50\text{ ms}\).
A common stumbling block when running local models is typing agent instructions directly into ollama run:
In plain ollama run, the model is running in an isolated conversational REPL with no tool schemas, no file access, and no shell or Git access. Because it cannot actually query GitHub or inspect your repository, it will fabricate a fictional issue (e.g. cpython/issues/1234), write fictional code in prose, and describe a non-existent commit. When asked “did you push the PR?”, it will correctly admit that it is a text-only assistant without execution capabilities.
An agent requires a harness (such as aider or a programmatic tool runner) that:
tools parameter).tool_calls payloads.Default local model tags are configured for open-ended conversation (temperature \(0.8\), small \(4\text{k}\) context). For agentic tool use, bake a dedicated model tag via a Modelfile to enforce deterministic schema compliance:
Create the derived model:
This ensures:
0.0): Prevents hallucinated JSON keys or invalid tool parameters.16384): Accommodates multi-turn tool outputs and file snippets without silent truncation.Section 1 already shows the mechanics of splitting a task between two local models with aider --architect: a larger model plans the change, and a smaller one applies the edits. The same split has a name in the research literature and a stronger motivating argument than “it’s faster”: Belcak and NVIDIA’s small-language-model research group argue that most of what an agent does in a loop is “a small number of specialized tasks repetitively and with little variation” — reading a diff, running a test, formatting a commit message — and that a small model is “sufficiently powerful, inherently more suitable, and necessarily more economical” for that work (Belcak et al. 2025). A large model earns its cost only on the steps that genuinely need broad, general reasoning: deciding what to change and why.
Two shapes of this pattern are worth knowing:
The same split is the main cost lever in a public field report from a non-programmer (“Vibe coded this game in four months”, r/ClaudeCode, 2026-08-22; summarized in issue #98), who built a browser racing game over four months with coding agents. The report names three levers:
The reported total was roughly $200 in project-specific subscriptions over the four months, on top of a general-purpose subscription the author already held. The lab’s machine-facing configuration, Morrison-Lab/ai-config, states the same routing rule for agents.
None of these routing choices substitutes for the guardrails below. A well-chosen planner still hands off to an executor that can make a per-step mistake, and the loop still needs a way to catch that.
Errors compound in an unattended loop
A frontier cloud model makes fewer per-step mistakes than a small local one, but the risk that matters here is not the per-step error rate on its own — it is that autonomous mode removes the human who would otherwise catch a mistake before it becomes the premise for the next ten steps. Structure the loop so that a mistake is caught by something other than a human watching in real time, or do not leave a small model fully unattended.
The mitigation for a higher per-step error rate is not a better model; it is a loop that cannot silently drift far from a known-good state. Five patterns do most of the work, and each is deliberately mechanical rather than judgment-based — the whole point is that they do not depend on the model noticing its own mistake:
quarto render, a linter, R CMD check — after each edit, and treat a non-zero exit as a hard stop for that step, not a suggestion. The environment’s pass/fail signal is the judge, never the model’s own claim that it “should work now.”worktree feature), and commit after every step that passes its gate — never after a batch of several. A commit-per-green-step history means the worst outcome of a bad step is one commit to roll back, not an unreviewable pile of changes.aider.These are the guardrails as a reader-facing rationale. The concrete gate wiring — an ai-config skill that launches a capped, worktree-isolated local loop, and a gha reusable workflow that runs one against a pull request using this repository’s own lint, spellcheck, and render checks as its gates — is tracked separately; see Companion work below.
This site’s own stack — R, Python, Quarto, Julia, GitHub Actions YAML, and Markdown — is largely about producing verifiable artifacts: a script that runs, a document that renders, a workflow that passes. That is exactly the property that makes a narrow, checkable sub-task safe for an autonomous small-model loop, but the safety margin is not the same across languages:
| Language / format | Autonomy dial | Notes |
|---|---|---|
| Python, Markdown, GitHub Actions YAML | Loosest leash | Well represented in training data; a syntax or lint check is a strong gate on its own |
| R | Tighter gates | Watch for non-tidyverse idioms and unfamiliar use of S4 or Reference (R5) classes; a model trained mostly on Python code can default to non-idiomatic R |
Quarto (.qmd) |
quarto render as the pass/fail judge |
Have the model edit a known-good _quarto.yml rather than authoring one from scratch — a render failure is a strong, cheap gate |
| Julia | Shortest leash, strongest model, tightest test gate | The weakest training coverage of this stack’s languages, so treat any unattended Julia change as higher risk by default |
None of this changes the guardrails above; it changes how tightly you set them — a smaller step size, a lower consecutive-failure cap, or simply keeping a human in the loop for Julia while letting a Markdown fix run unattended.
A local model’s non-idiomatic R or Julia, noted in the stack-specific table above, is a training-data problem rather than a capability problem: the model has seen far less R and Julia than Python, not that it is incapable of writing either. Two lighter options are worth trying before fine-tuning anything:
CLAUDE.md and .github/copilot-instructions.md are examples of exactly that.When those are not enough, LoRA (Low-Rank Adaptation) and its 4-bit variant QLoRA are the standard way to close an idiom gap without retraining a whole model. Both freeze the pretrained weights and train a small set of additional low-rank matrices on top, which cuts the trainable parameter count by orders of magnitude compared to full fine-tuning (Hu et al. 2021). QLoRA adds 4-bit quantization of the frozen weights on top of that, which is what actually shrinks the memory footprint enough to fine-tune a mid-sized model on a single consumer GPU (Dettmers et al. 2023). Hugging Face’s PEFT library is the common tooling entry point; the specifics of running it against this lab’s own repositories belong in ai-config, not here.
Whatever you fine-tune on, keep a held-out evaluation set of real tasks from your own codebase that the training data never touched, and re-check it after every fine-tuning run — a model that has memorized its training examples will look better on paper than it performs on the next genuinely new task.
This page explains the reasoning; it does not implement a launcher or a CI gate. Two companion issues carry the runnable parts, each linking back here for rationale:
ai-config #1292: a skill that configures and launches a local autonomous loop — model choice, an Ollama or llama.cpp endpoint, and the guardrail caps above.
gha #436: a reusable workflow, a sibling to this repository’s own claude.yml, that runs a small/self-hosted-model agent against a pull request, wiring this site’s existing checks as the loop’s verification gates:
Inkling (Thinking Machines Lab 2026a) is a pair of open-weight, multimodal foundation models from Thinking Machines Lab (measured 2026-09-09). This is a model release, not a coding agent or a chat product: the weights are downloadable, and the company’s own hosted surfaces for it are a fine-tuning API and a playground rather than an end-user assistant. This section summarizes the release and asks whether it matters for our workflow.
Thinking Machines Lab is a San Francisco AI startup founded in February 2025 by Mira Murati, formerly OpenAI’s chief technology officer, with John Schulman, an OpenAI co-founder, as chief scientist (Wikipedia contributors 2026). It raised $2 billion at a $12 billion valuation in July 2025, and two founding members left for OpenAI in January 2026 (Wikipedia contributors 2026). Its first product, Tinker (Thinking Machines Lab 2026f), released in October 2025, is a LoRA fine-tuning API for open-weight models where the customer writes the training loop in Python and Thinking Machines runs the GPUs (Wikipedia contributors 2026). Inkling is the company’s first in-house model, released on 2026-07-15 (Thinking Machines Lab 2026c).
Two models share the name (measured 2026-09-09) (Thinking Machines Lab 2026a):
Both accept text, image, and audio input and produce text only (Thinking Machines Lab 2026b, 2026d). Both are released under the Apache 2.0 license (Thinking Machines Lab 2026b, 2026d), and both expose an effort setting that trades reasoning tokens for score; the reported evaluations are run at effort=0.99 (Thinking Machines Lab 2026c).
The announcement positions Inkling as a broad base for customization rather than a leaderboard winner (Thinking Machines Lab 2026c). Its claims, and the evidence offered for them:
These are the vendor’s own numbers on the vendor’s own runs; we have not reproduced any of them. Inkling-Small’s announcement reports 80.2% on SWE-Bench Verified and 31.6% on Humanity’s Last Exam, against Inkling’s 77.6% and 29.7% (Thinking Machines Lab 2026e).
Access routes (measured 2026-09-09):
NVFP4 checkpoints (Thinking Machines Lab 2026b, 2026d). Even the smaller model has 276 billion parameters, so neither fits any hardware the lab owns.llama.cpp among supported inference engines (Thinking Machines Lab 2026c). Our quota-aware defaults for the Databricks endpoint are already in
Thinking Machines’ stated bet is that organizations will want to fine-tune a capable open model on their own data rather than rent a closed frontier model, and Inkling exists to give Tinker a strong first-party base (Thinking Machines Lab 2026c). Tinker’s own pitch lists specialized agents, forecasting, continual learning, and AI research as the intended uses (Thinking Machines Lab 2026f). The audio and image inputs make it a candidate for transcription-plus-reasoning pipelines as well as coding.
As a coding-agent backend, Inkling is one more open-weight option alongside the models in the agent catalog, reachable through Databricks (custom model endpoints) or, when a host lists it there, OpenRouter (Section 4). Its reported SWE-Bench and Terminal Bench scores are competitive but not ahead of the closed models we already pay for, and the 8,192-token output cap on the Databricks endpoint is a real constraint for agentic edits. Self-hosting is out: the lab has no machine within an order of magnitude of the memory required.
The more interesting angle is research. Tinker lets a student run a fine-tuning or reinforcement-learning experiment on a frontier-class open model with a Python training loop and no cluster administration, and universities can ask for wider access (Thinking Machines Lab 2026f). If a project ever needs to adapt a model to epidemiological text or lab-specific coding conventions, Tinker with Inkling-Small is a cheap place to try it. Until such a project appears, no action is needed beyond keeping the Databricks endpoint in our defaults table.