Field Report 01

Status: Complete · Date: 2026-08-07 · Cost: $0

The short version

BearingBrain is a production bearing-quote system. We built a decision-grading loop around it: every customer query becomes a structured JSON decision (intent, part number, specs), scored deterministically against production labels. A reproducible evaluation gate is the precondition for any RL training — you cannot improve what you cannot measure.

We then ran hosted RL training on Prime Intellect — a 2.1B open model, our own private evaluation environment, 18 minutes, $0 — and re-ran the exact same gate: 0.954 → 0.962 overall.

The dataset

449 real queries from BearingBrain production logs (225 hand-labeled, 224 raw chat). Each query is scored against an 11-field structured decision schema: intent, part number, manufacturer, bearing type, seal type, bore/OD/width, environment, speed, load. Unknown fields are null.

The grader is deterministic — no LLM judge. Per-field matching with tolerance: exact enum match for intent, normalized string match for part number, ±0.5 mm numeric tolerance for specs, parse check for JSON validity. Weighted score: intent 25%, part number 25%, spec 30%, manufacturer 10%, JSON validity 10%.

Baseline: one gate, four models

The spread is the training opportunity: a capable hosted model scores 0.95, a small local model 0.64. The small model's weakness is concentrated in the structured-decision load — exactly what the gate measures (JSON validity 0.51, decision-event accuracy 0.19).

What the gate exposed

Label quality, not model quality, was the bottleneck. Three findings:

• Production-label noise. 6 of 11 low-scoring traces were label errors: whole phrases stored as part numbers ("6204-2RS CROSS REFERENCE"), gibberish ("Jgj"/"test") labeled as part lookups, and a full sentence labeled as a part number. The model was right; the label was wrong.
• Duplicate query, contradictory labels. "NSK 6204DDU replacement" appears twice labeled "crossref"; the model said "part_lookup" both times. Same input, two defensible answers — the gate caught a dataset inconsistency.
• Taxonomy ambiguity. Conversational queries ("My 6204 keeps spinning in the housing, what should I do?") are labeled part_lookup/spec_search but route to chat. Both are defensible; the production heuristic parser defaults bearing_type to deep_groove on everything.

The hosted RL training run

We built a private evaluation environment on Prime Intellect and ran hosted RL training on a free 2.1B open model (poolside/Laguna-XS-2.1). 30 steps, 18 minutes, $0. On the platform's own gate, the trained policy moved 0.903 → 0.921 at the run's sampling temperature. JSON validity and manufacturer hit 1.0 by step 10 and stayed there; spec accuracy climbed 0.952 → 0.979; part-number 0.838 → 0.875.

The after-number: same gate, temp 0, full 449 tasks

Same gate, same tasks, deterministic decoding. The trained policy gains +0.8 points overall, driven by spec accuracy (+2.9pt) and part-number (+1.0pt) — the fields the grader rewards. Manufacturer dips −0.8pt, noise-level on a near-saturated key. The delta is modest because the free 2.1B base is already strong on this gate; the durable win is the loop itself: data → environment → gate → train → eval now runs end-to-end unattended.

Notes and limitations

• The free 2.1B base is already strong on this gate — a +0.8pt move on a saturated scorecard is real but modest.
• The platform cannot serve poolside/Laguna-XS-2.1 for inference (training-platform-only), so the after-number came from warm-started re-eval runs. ~10% of rollouts hit transient free-pool 503s (counted as 0 in "all", excluded from "effective").
• Adjudicated label accuracy across the production rows is ≈ 0.95–0.96 — the gate's ceiling is partly set by label quality, which is exactly the finding.

Cost

• 40-task baseline (DeepSeek-V4-Flash, free tier): $0
• 449-task full run (DeepSeek): $0
• Prime hosted RL training (30 steps, free Laguna-XS-2.1, private env): $0
• Deterministic-gate re-evals (base control + 3 trained runs): $0
• Local Qwen3-8B full-449 gate (Ollama): $0