Field Report 03

Status: Complete · Date: 2026-08-07 · Cost: $0

The short version

Field Report 01 built a deterministic evaluation gate and benchmarked four models against it. This report answers the question that benchmark left open: do you need a hosted model, or is a model on your own hardware good enough?

On 449 real BearingBrain queries: a hosted DeepSeek model scored 0.950. An 8B model running locally on a Mac scored 0.917 — within 3 points — at the same median latency (2.34 s vs 2.51 s), with zero per-query cost and no data leaving the building. A 1B model was 3.5× faster but its accuracy collapsed to 0.640 on the same workload.

Why this matters

Most small manufacturers are told they need a hosted AI service to get good results. That comes with per-query costs, network dependency, and sending production data to a third party. The alternative — a model on your own machine — has a reputation for being "not good enough."

The gate from FR1 gives us a way to test that reputation with real numbers instead of opinions. Same 449 queries, same deterministic scoring, same JSON schema.

The results

All runs: temperature 0, deterministic decoding, identical scoring. Hosted runs used the free tier — no inference spend. Local runs used Ollama on a Mac M4 Pro with 48 GB unified memory.

Where the 1B model breaks

The 1B model is not uniformly worse — it is specifically bad at structured decision traffic. Its scores by query source:

The failure is not "small models can't do this." It's that a 1B model cannot reliably follow the structured JSON schema when the input resembles a partial form entry. For a pure chat-assistant workload, 1B is surprisingly capable — 0.888 on unlabeled chat, with JSON validity at 0.44.

What this means in practice

• An 8B local model is a credible production optionfor this workload — within 3 points of the best hosted model on a hard gate, at the same latency, with zero per-query cost.
• Data never leaves the building. For a manufacturer with quote history, customer lists, or internal part data, this is the difference between "we can try this" and "we cannot."
• Size selection is workload-specific. The same 1B model that collapses on structured extraction is fine at conversational chat. Model choice should follow the task, not a general "bigger is better" rule.

Notes and limitations

• Latency is median wall-clock per query on a single machine. An 8B model on older hardware, or without unified memory, will be slower; a GPU workstation will be faster.
• The hosted DeepSeek run used a free tier with rate limits — its median includes queue time, not pure generation. Local latency is generation time only, which makes the comparison conservative in the local model's favor.
• Accuracy was measured on one workload (bearing-quote parsing). "Good enough" is a property of the task, not the model.

Cost

• Hosted DeepSeek free tier (449 queries): $0
• Local Qwen3-8B via Ollama (449 queries): $0
• Local Llama-3.2-1B via Ollama (449 queries): $0
• Hardware: Mac M4 Pro, already owned
• TOTAL: $0