Run and recovery
The adapter trained on 1,080 rows from 270 specs, with 120 validation rows from 30 disjoint specs. Token lengths ranged from 872 to 10,784; the 12,288-token ceiling truncated zero examples.
The machine restarted after step 105. Checkpoint 100 was complete, so training resumed from its adapter, optimizer, scheduler, and RNG state. Five uncheckpointed steps were lost. The resumed job ran at lowered CPU and I/O priority with the noisy vLLM Studio GPU polling service disabled, then completed all 270 optimizer steps and exactly two epochs.
- Base model
- nvidia/Nemotron-Cascade-8B
- Training method
- 4-bit NF4 QLoRA, rank 32, all linear modules
- Final adapter size
- 349,243,752 bytes
- Adapter SHA-256
1389c83c1b6b81806f572761682176a4460818ef258529dba0c4ea91c0fcc514
Learning curve
| Step | Epoch | Loss | Token accuracy |
|---|---|---|---|
| 0 | 0.000 | 1.0588 | 0.7855 |
| 25 | 0.185 | 0.6245 | 0.8318 |
| 100 | 0.741 | 0.4753 | 0.8608 |
| 175 | 1.296 | 0.4489 | 0.8680 |
| 250 | 1.852 | 0.4402 | 0.8704 |
| 270 | 2.000 | 0.4401 | 0.8704 |
The held-out curve improved through the final sweep. This was not a failed optimizer run; the model learned the training distribution successfully.
Frozen benchmark
| Model | Static clean | Rubric average | Direct evidence |
|---|---|---|---|
| Base | 0 / 10 | 1.30 / 5 | Baseline |
| V1 | 2 / 10 | 1.83 / 5 | Best |
| V2 | 1 / 10 | 1.67 / 5 | Beats V3, 6–4 |
| V3 | 0 / 10 | 1.72 / 5 | Loses to V1, 2–7 |
V3 produced 10/10 correctly shaped raw answers, but only 7/10 passed policy, 4/10 compiled, 2/10 had zero svelte-check errors, and 1/10 passed the autofixer. The all-or-nothing static score was therefore 0/10. One response hit the 8,192-token generation cap; truncation does not explain the other failures.
Open the interactive per-prompt report and code playground links.
V4 implications
The key lesson is that low held-out language-model loss did not predict executable Svelte quality. V4 should be gated primarily by fresh generated-code validity and prompt fidelity, not by training loss alone.
- Keep the frozen 10-prompt suite unchanged for direct comparability.
- Inspect target completion length and document-like verbosity; V3 generated several malformed or over-elaborate components.
- Favor examples whose behavior is fully specified and validated, while excluding portability-biased contract labels from reward decisions.
- Run an early checkpoint benchmark before committing to the full schedule; stop if compile-clean rate regresses.