Plain-English overview
V6 changes the job. V5 tried to operate as a coding agent. V6 is trained to inspect one Svelte component and return a compact review: pass or fail, a summary, and grounded findings in a fixed JSON shape. This is a narrower task for an 8B model, which makes the model's limited capacity much more useful.
The starting point was NVIDIA's Nemotron-Cascade-8B. QLoRA left the original eight-billion-parameter model unchanged and learned a 167 MiB adapter containing small corrective weight matrices. The base and adapter can therefore be served together and compared with exactly the same request.
- Base model
- nvidia/Nemotron-Cascade-8B
- Frozen base revision
- 276a779fe0c1fe2d63a68fd573fec7b4b0a1907e
- Method
- 4-bit NF4 QLoRA, rank 32, alpha 64, dropout 0.05
- Effective batch size
- 8 records: one at a time, accumulated across eight forward passes
- Training context
- 12,288 tokens; longest rendered record was 10,703 tokens
- Reasoning mode
- Disabled in both training and evaluation
Dataset and frozen split
The corpus contains 2,046 reviewer records: 1,641 for training, 203 for validation, and 202 for the final test. Training records change the adapter. Validation records guide recipe decisions while training is visible. Test records remain untouched until the end, so they provide the fairest estimate of behavior on unseen component families.
| Split | Records | Used for |
|---|---|---|
| Training | 1,641 | Updating the QLoRA adapter |
| Validation | 203 | Measuring generalization during the run |
| Test | 202 | One final base-versus-V6 comparison |
The test set spans 30 complete specification lineages. Leakage checks found zero shared specification IDs and zero exact source hashes between train/validation and test. That is important: putting close variants from one component family on both sides of the split would make the score look better without proving real transfer.
Passing examples teach restraint. Failing examples teach a canonical rule, severity, source span, evidence, and explanation. Controlled mutants make one intentional behavioral defect in an otherwise good component, creating a clean contrast for the model to learn.
How the training run went
The run completed all 412 optimizer steps over two epochs. Validation loss dropped sharply in the first 50 steps, then improved gradually through step 400. The final step landed at 0.10743, only 0.00001 above the best scheduled checkpoint—effectively identical—so the served final adapter is the correct simple choice.
System health
A separate monitor sampled Morningstar every 30 seconds and was authorized to stop the run after two readings at or above 84 °C, below 4 GiB available host memory, or below 20 GiB free disk. None of those guardrails fired.
| Resource | Observed | Interpretation |
|---|---|---|
| GPU temperature | 73.2 °C avg / 76 °C peak | 8 °C below stop threshold |
| GPU power | 558 W avg / 579 W peak | High sustained utilization with brief sensor-level transients |
| GPU utilization | 94.8% average | Compute stayed productive |
| GPU memory | 19,916 / 32,607 MiB peak | About 12.4 GiB headroom |
| Host memory available | 40.9 GiB minimum | No pressure; zero swap |
| Disk available | 666.7 GiB minimum | No storage pressure |
Before and after
Both models received the same 202 frozen test prompts, temperature zero, reasoning off, one attempt per case, and the same strict JSON schema. The test includes 69 failing and 133 passing components.
| Metric | Base | V6 strict | Reading |
|---|---|---|---|
| Verdict F1 | 69.4% | 82.1% | +12.7 points |
| Fail precision | 78.2% | 100% | Every strict V6 failure verdict was correct |
| Fail recall | 62.3% | 69.6% | V6 still misses some real defects |
| False rejection of good code | 9.0% | 0% | No valid test component rejected |
| Exact canonical findings | 0 / 98 | 48 / 98 | Base invented rule names; V6 learned the contract |
| Controlled-mutant detection | 0% | 81.4% | 48 of 59 under strict parsing |
| Reference reranking | 28.6% | 100% | 28 of 28 eligible lineages |
| Strict schema validity | 100% | 94.6% | 11 evidence strings contain raw tabs |
The verdict-F1 improvement is positive under a 10,000-resample lineage bootstrap: +2.5 to +23.4 percentage points at 95% confidence. In plain English, the gain is large enough that it is unlikely to be an accident of this particular test mix.
The JSON tab edge case
Eleven otherwise-correct controlled-mutant answers copied a leading tab into the
evidence string as a literal control character. The outer API response is
valid, but parsing the inner reviewer JSON fails. The strict score correctly counts those as
failures. If the serving layer escapes those tabs before parsing, all 11 recover: verdict F1
becomes 92.2%, fail recall 85.5%, canonical findings 59/98, and
controlled-mutant detection 59/59.
The recoverable score shows model knowledge, not current end-to-end reliability. The 82.1% strict score remains the headline until serving guarantees JSON-safe evidence.
Two benchmark coverage notes
- Only one rule family has at least five gold findings, so the current test cannot support a broad “all Svelte rules” promotion claim.
- Two of 30 lineages lack a complete one-reference/two-mutant trio and are excluded from reranking. Their ordinary verdict and finding results still count in the 202-case score.
How the model feels
On the task it was redesigned for, V6 feels dramatically more coherent than the base. It
usually returns a terse pass for good code and, for an intentional behavioral mutation,
names spec/behavior-requirement, quotes the relevant source, and explains the
broken requirement. It no longer sprays plausible-sounding custom rule names.
It is especially strong as a relative judge: across every eligible trio, it ranked the clean reference above the two mutants. That suggests a practical near-term use as a candidate filter or review assistant where a human still reads the explanation.
The weaknesses are equally clear. It missed all of the small held-out
dom, html, ssr, and quality finding
groups, including a non-deterministic hydration ID. Its line numbers are often offset even
when the quoted evidence is correct: only 12.5% of exact-rule matches overlap the gold line
span, and just 2.1% of predicted evidence passes the scorer's strict line-and-source
grounding check. It should not yet auto-block a pull request or apply fixes unattended.
Key concepts
- Token
- A small piece of text predicted by the model. Code punctuation and identifiers may be split across several tokens.
- Loss
- A penalty for assigning too little probability to the correct next token. Lower is better only when comparing the same task and held-out data.
- Token accuracy
- The fraction of supervised positions where the most likely next token matches the target. It is not the same as complete-answer accuracy.
- Epoch
- One pass through all training records. V6 completed two passes.
- Validation set
- Examples excluded from weight updates but checked during training to reveal generalization or overfitting.
- Test set
- A final frozen set used only after training. Repeatedly tuning against it would turn it into another validation set.
- Precision
- Of the components V6 called bad, the fraction that really were bad. V6 achieved 100% under the strict run.
- Recall
- Of all truly bad components, the fraction V6 caught. Strict recall was 69.6%, so misses remain the main verdict problem.
- F1
- A balance of precision and recall. It prevents a model from looking good merely by calling everything pass or everything fail.
- QLoRA adapter
- A compact learned layer over a quantized frozen model. It makes fine-tuning an 8B model practical on Morningstar's single 32 GiB GPU.
Using the model
Morningstar is serving an OpenAI-compatible endpoint over Tailscale. Use model ID
svelte-reviewer-v6 with reasoning disabled:
Base URL: https://morningstar.otter-hawksbill.ts.net:10001/v1
Model: svelte-reviewer-v6
API key: any non-empty local placeholder
This adapter is a reviewer, not a Pi coding agent. Give it the V6 reviewer system prompt, a
task specification when behavioral review is desired, and the component source. Expect one
svelte-review-v1 JSON object in response.
The frozen dataset can be inspected in the existing V6 dataset review page.
Recommendation
Keep V6 running as an experimental review assistant. Before calling it a production Svelte gate, make a focused V6.1 corpus that line-numbers every source line, normalizes evidence whitespace, adds far more static-only failures across each rule family, and includes adversarial passing examples near those rules. Re-run this exact frozen benchmark afterward without changing the test labels.
The core decision is encouraging: the 8B model has enough capacity for this narrower classification-and-explanation task. Moving the same cleaned reviewer corpus to a larger Qwen model may improve recall and grounding later, but V6 already proves that task specialization matters more than simply asking the small model to be a full coding agent.