Outcome
This was a full one-epoch supervised fine-tuning run. “Supervised” means the model was repeatedly shown an input and the desired next assistant action, then adjusted to make that desired output more likely. The run completed without out-of-memory errors, numerical instability, truncation, or checkpoint recovery.
| Signal | Before | After V4 | What changed |
|---|---|---|---|
| Held-out loss | 1.8875 | 0.1030 | 94.5% lower |
| Held-out token accuracy | 65.18% | 96.96% | +31.78 points |
| Pi tool-loop tasks | 0 / 3 | 1 / 3 | One new pass |
| Strict static-clean answers | 0 / 10 | 0 / 10 | No gate change |
| Compiler-clean answers | 3 / 10 | 5 / 10 | +2 answers |
| Modern-Svelte policy clean | 1 / 10 | 6 / 10 | +5 answers |
| Standalone quality rubric | 1.58 / 5 | 1.57 / 5 | Essentially flat |
What the model trained on
The V4 export contains coding-agent trajectories rather than only final snippets. A trajectory records the sequence an agent should follow: inspect a file, make a change, run validation, respond to failures, and finish only after checking the real result. That makes it possible to teach both code and workflow.
- Training examples
- 3,543 assistant actions
- Validation examples
- 377 assistant actions kept out of training
- Source trajectories
- 1,118 complete coding-agent trajectories
- Action coverage
- 566 reads, 566 edits, 552 writes, 1,118 validation phases, and 1,118 final responses
- Sequence lengths
- 1,393 minimum; 4,868 median; 8,519 at the 95th percentile; 12,195 maximum
- Training ceiling
- 16,384 tokens, with zero oversized or truncated rows
If a long example is cut off, the model may see “run validation” without the result, or an error without the repair. Every V4 example fit under the token ceiling, so each supervision target remained intact.
How QLoRA works
The base model has about 8.28 billion parameters. Updating all of them would require much more GPU memory. QLoRA keeps the base weights frozen and stores them in a compact 4-bit form, then trains small low-rank adapter matrices alongside selected layers. At inference time, the adapter steers the frozen base model toward the new behavior.
The 167 MB adapter is not a standalone 8B model. It must be loaded with the exact Nemotron Cascade base revision used for training. Different adapters can share that same base.
The training recipe, translated
| Setting | Value | Plain-language meaning |
|---|---|---|
| Epochs | 1 | The trainer made one pass over all 3,543 training examples. |
| Quantization | 4-bit NF4 + double quantization | The frozen base weights used a memory-efficient representation designed for normally distributed neural weights. |
| Compute type | bfloat16 | Math used a wider-range 16-bit format that is generally stable for modern training. |
| LoRA rank | 32 | Each adapter update is represented through a compact 32-dimensional bottleneck. |
| LoRA alpha | 64 | This scales the adapter contribution; with rank 32, the nominal scale is 64/32 = 2. |
| Dropout | 0.05 | Five percent of adapter activations are randomly omitted during training to reduce memorization. |
| Batch size | 1 × 8 accumulated | One long example fits at a time; gradients from eight examples are combined before each optimizer update. |
| Learning rate | 0.0001 with decay | Updates start larger, then become progressively smaller so the run settles near the end. |
| Loss mask | Completion only | The prompt provides context, but the penalty applies to the assistant action the model is supposed to produce. |
| Seed | 42 | Fixes controllable randomness so reruns are easier to compare. |
What happened during training
The trainer measured the 377 held-out examples before the first update and after every 50 steps, plus once at the end. Held-out measurements matter because those examples were not used to update the adapter.
Behavior-level evaluation
Real Pi coding-agent loops
Both models ran through Pi 0.79.10 with the same read, bash, edit, and write tools. Each final file was independently checked after the model stopped, so a confident final message could not substitute for a real pass.
| Task | Base | V4 | Observed behavior |
|---|---|---|---|
| Read, exact edit, validate | Fail | Pass | V4 used read → edit → bash, then inspected two clean validator results. |
| Recover after injected failure | Fail | Fail | V4 diagnosed the missing button type but malformed its follow-up edit tool call. |
| Create and validate component | Fail | Fail | V4 wrote, checked, and attempted repairs, but left an invalid CSS value. |
Ten-prompt raw component suite
The strict “static-clean” result is 0/10 for both models because every answer had at least one failure across output shape, modern-Svelte policy, parsing, type checking, or automated best-practice review. The subchecks show that V4 still moved in the desired direction.
Blinded pairwise review
Claude Fable compared base and V4 on each prompt twice, reversing which answer appeared first. The stable verdicts split 4–4. Two prompts were order-sensitive, meaning the judge preferred whichever candidate appeared in a particular position; those were excluded from the win counts.
How to interpret the result
A model can accurately imitate the next token when shown a correct trajectory yet still drift when it must generate many steps, consume tool results, recover from an error, and decide when to stop. Small mistakes compound in an autonomous loop.
What clearly improved
- The held-out imitation objective improved dramatically and remained stable through the end of the epoch.
- V4 emitted valid structured tool arguments in the serving stack; the base often double-encoded them.
- V4 completed one full Pi read-edit-validate loop where the base completed none.
- Modern-Svelte policy compliance rose from 1/10 to 6/10, output-shape compliance from 1/10 to 6/10, and compiler-clean output from 3/10 to 5/10.
What did not improve enough
- Two of three agent tasks still failed after malformed tool syntax or a bad repair.
- No raw answer passed every strict static check.
- The standalone rubric mean stayed essentially flat, and the direct pairwise judge found no overall winner.
- The model can still generate corrupted identifiers or CSS values during longer answers.
What I would test next
- Add more short, clean recovery trajectories where validation fails once and the next tool call repairs exactly that error.
- Increase examples that finish immediately after two clean validators, reducing over-generation after success.
- Run a larger agent-loop benchmark with repeated seeds so a single malformed call has less influence.
- Track “valid tool call,” “successful edit,” “validation attempted,” and “final validation clean” as separate metrics during future iterations.
- Consider filtering or reweighting very long generated code trajectories that contain corrupted text, while retaining high-quality tool-oriented examples.
Reproducibility and artifacts
- Base model
- nvidia/Nemotron-Cascade-8B
- Base revision
276a779fe0c1fe2d63a68fd573fec7b4b0a1907e- Repository commit
dd29dc73e60b68fa4f3b4fbed91ccfe069a94b86- Training data SHA-256
4fcdd0a7360d7049ae5333635403f5b8f9c2be2eaab9f83e27bc69734635324a- Validation data SHA-256
98e1aa13b80cb092e01ce3d269fc14fdac12b4d2c475413042f52be377682d20- Environment
- PyTorch 2.13.0 + CUDA 13.0; Transformers 5.14.1; TRL 1.8.0; PEFT 0.19.1
- Adapter size
- 167 MB model weights
- Agent harness
- Pi 0.79.10 with read, bash, edit, and write tools
- Serving
- vLLM 0.25.1; Qwen3 reasoning parser; Hermes JSON tool-call parser
The run metadata, TensorBoard events, final adapter, raw benchmark answers, static scores, Pi event streams, and judge records remain in the native Morningstar checkout. No training examples, credentials, or private prompts are embedded in this public report.