Measured on real hardware · 2026-08-03
Which local models actually do agent work on 8 GB
Every “best local LLM” list ranks models by tokens per second. That tells you how fast a model finishes a sentence. An agent reads files, runs commands and chains steps — so we measured that instead, on one board, with tests a plausible-sounding answer cannot pass.
The short version: you don’t need a bigger box — you need a smaller, better model. The best agentic model we tested is granite4:micro-h at just 1.9 GB. It passed every task in every run, and it was also the fastest — 16.4s per full pass, leaving around 6 GB free for context.
NVIDIA Jetson Orin Nano Super · 8 GB unified · 67 TOPS · JetPack R36.5 / Ubuntu 22.04 · MAXN_SUPER (jetson_clocks pinned) · Ollama. Freshly booted with nothing else running — a loaded board produces meaningless numbers. Everything here reproduces on any Orin Nano Super.
The tests
Five tasks, each with an objective oracle, run 5 times per model. Scores are pass rates. That matters: an earlier single-run version of this benchmark reached the opposite conclusion and did not replicate.
- 1Read. Read a file containing an unguessable random token and report it back. A model that guesses cannot pass.
- 2Write. Write exact contents to a file. Checked on disk, not taken from what the model claims it did.
- 3Chain. Read numbers from one file, add them, write the total to another. Three steps where each depends on the last.
- 4Shell. Run a command whose answer only the hardware knows.
- 5Restraint. A trivial question where reaching for a tool at all is the failure. Knowing when not to act counts.
Results
| Model | Size | Score | Read | Write | Chain | Shell | Restraint | Median run |
|---|---|---|---|---|---|---|---|---|
granite4:micro-hour default | 1.9 GB | 5.0/ 5 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 16.4s |
granite4:tiny-h | 4.2 GB | 5.0/ 5 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 16.8s |
cogito:3b32K ctx | 2.2 GB | 5.0/ 5 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 41.2s |
qwen3.5:2b | 2.7 GB | 5.0/ 5 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 45.9s |
qwen3.5:4b | 3.4 GB | 5.0/ 5 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 88.2s |
nemotron-3-nano:4b | 2.8 GB | 4.8/ 5 | 4/5 | 5/5 | 5/5 | 5/5 | 5/5 | 49.8s |
llama3.2:3b32K ctx | 2 GB | 3.0/ 5 | 5/5 | 5/5 | 0/5 | 5/5 | 0/5 | 20.8s |
hermes3:3b32K ctx | 2 GB | 1.0/ 5 | 0/5 | 0/5 | 0/5 | 0/5 | 5/5 | 21.8s |
Median run = wall-clock for one complete five-task pass, every tool round-trip included. Models marked 32K ctx could not be loaded at 64K on 8 GB — see the notes below.
5 of 8
models cleared every task, every run
1.9 GB
the winning model’s footprint
16.4s
its median full-task run
What the numbers say
5 of 8 models cleared every task, every time. Read that as a floor, not a ceiling: these are foundational tool calls, and the useful finding is that reliable agentic work on an 8 GB board is now routine rather than marginal. Eighteen months ago it was not.
It also means the interesting question has moved. When most models pass, choosing between them is about footprint and latency, not capability — which is why a 1.9 GB model finishing in 16 seconds beats a 3.4 GB one that takes 88 seconds to reach the identical result.
granite4:micro-h— Perfect score, smallest model tested, and the fastest. This is what we ship.llama3.2:3b— Handles single tool calls, never completed the three-step chain, and reached for a tool on the question that did not need one.hermes3:3b— Writes the tool call into its reply as prose instead of emitting a structured call, so nothing can execute it. Its one pass is the restraint task — which it passes by accident, because it cannot call a tool anyway.
For agents, the “efficient” power mode is the slow one
The best published throughput sweep of this board — eight models across all four power modes — finds 25 W is the sweet spot, delivering 35–47% more tokens/sec than 15 W and better tokens/joule than MAXN. So we re-ran this entire suite at 25 W to check whether our MAXN numbers were leaving performance unclaimed.
They weren’t. For agentic work 25 W came out 13% slower, on every model:
| Model | MAXN_SUPER | 25 W | Penalty |
|---|---|---|---|
granite4:micro-h | 16.4s | 18.4s | +12% |
granite4:tiny-h | 16.8s | 18.7s | +11% |
qwen3.5:2b | 45.9s | 50.7s | +10% |
qwen3.5:4b | 88.2s | 105.4s | +20% |
nemotron-3-nano:4b | 49.8s | 54.8s | +10% |
Both results are right — they measure different shapes of work. A throughput sweep measures decode at fixed prompt and generation lengths, and decode is bound by memory bandwidth, where 25 W’s efficiency wins. An agent loop is the opposite: every tool result re-processes a growing context, so it is prefill-heavy with short generations — and prefill is compute-bound, which is precisely what MAXN’s higher clocks buy.
Scores were identical in both modes. Power mode changes how long the work takes, not whether the model gets it right. If you are running an agent on this board, use MAXN.
What this measures — and what it doesn’t
Several models score 100% here while purpose-built tool-calling suites put capable small models nearer 40–55%. That isn’t a contradiction, and it isn’t us marking our own homework — it’s a different question. Worth reading before you quote any number above.
These are foundational tasks, not a difficulty ceiling
Four of the five are single tool calls; one is a two-hop chain. Purpose-built suites such as the Berkeley Function Calling Leaderboard score capable small models around 40-55% because they include parallel calls, deeply nested arguments and long multi-turn state. A perfect score here means a model clears the basics reliably — not that it matches a frontier model.
A pass rate is not an accuracy score
We report how often a model completed a task across five runs, with an oracle check. We do not score partial credit, argument quality or recovery after a bad call. A model that fails once in five is doing something different from one that fails every time, and the per-task columns are there so you can see which.
One board, and the power mode was checked rather than assumed
Everything was measured on a single Orin Nano Super at MAXN_SUPER with clocks pinned. We re-ran the whole suite at 25W because published throughput sweeps of this board favour it — for agent work it turned out 12% slower, for the reason explained above. Results still move with quantisation, runtime version and board.
Ollama costs throughput, and we used it anyway
Independent testing on this same board found llama.cpp ahead of Ollama by 36-74% on throughput for small models. We measured through Ollama because that is what ships on the box and what most people actually run. Treat the seconds as realistic rather than optimal — the ranking is unaffected, since every model faced identical conditions.
Notes that will save you an afternoon
At 64K context, Llama-3.2-derived models run out of memory on 8 GB
llama3.2:3b, cogito:3b and hermes3:3b all fail with cudaMalloc failed while allocating a 2.28 GB KV cache. They run fine at 32K. Granite-4 (hybrid attention) and Qwen-3.5 never hit this. When a model will not load, check the context length before blaming its size.
gemma3:4b cannot call tools at all
Not "does it badly" — Ollama refuses outright: "does not support tools". It is a capable chat model and cannot be an agent. Several popular "best small model" lists still recommend it for agent work.
Nanbeige4.2-3B needs a newer llama.cpp than Ollama ships
Stock Ollama fails with unknown model architecture: nanbeige. The GGUFs are fine. The model uses weight-shared depth loops — 22 layers run twice rather than 44 stacked — which needs its own architecture handler. llama.cpp merged support on 27 July 2026; Ollama v0.32.5 shipped the same day without it. Built from current llama.cpp source it loads and runs.
Model size predicts almost nothing
The 1.9 GB winner beat the 4.2 GB model on speed and matched its score, and finished five times faster than the 3.4 GB one for the same result. Pick on measured behaviour, not on parameter count.
Run it yourself
The harness is published. Point it at any Ollama endpoint and it reports a pass rate per task — and when something fails, why: no tool call at all, the wrong tool, or a claim of success with nothing on disk. Those are different problems needing different fixes.
python3 agentic_bench.py --models granite4:micro-h,qwen3.5:2b --runs 5Full results and the test harness →
If these numbers don’t reproduce on your hardware, we’d rather hear it than not. An earlier version of this benchmark was wrong, and a reader pointing that out is worth more to us than a tidy table.
Common questions
What is the best local model for an agent on 8 GB?
granite4:micro-h. It passed all 5 tasks in all 5 runs, is only 1.9 GB, and was also the fastest at 16.4 seconds per full pass — leaving roughly 6 GB free for context.
Can an 8 GB Jetson Orin Nano really run an AI agent?
Yes, for everyday tool use. 5 of the 8 models tested completed every task in every run, including multi-step chaining, on a board drawing about 25 W. These are foundational operations rather than a hard benchmark, so read it as "this works reliably", not "this matches a frontier model".
Why measure tool calling instead of tokens per second?
Tokens per second measures how fast a model finishes a sentence. An agent reads files, runs commands and chains steps, so what matters is whether those calls are well-formed and land. The two rankings differ.
Does a bigger model do better agent work?
Not on this evidence. The 1.9 GB winner matched the 4.2 GB model on score while being faster, and finished five times quicker than a 3.4 GB model that reached the same result.
We measured this on the box we sell
ClawBox is an NVIDIA Jetson Orin Nano Super with the agent stack already installed and granite4:micro-h set up as the local model. Same hardware, same numbers, nothing to configure.