Nemotron Pi Agent V5 training report

What was trained, what the curves mean, how the machine behaved, and what the model can actually do inside Pi.

Training examples

4,119

436 held out for validation

Optimizer steps

515

One complete pass through the data

Final validation loss

0.0635

Down from 0.9423 before training

Pi agent success

3 / 9

Base model: 0 / 9

Bottom line

V5 is measurably better than the base model at completing a real Pi tool loop, but it is still unreliable. It can read, edit, validate, and report correctly; intermittent double-encoded tool arguments and runaway generation still make it unsuitable as an unattended coding agent.

Plain-English overview

The starting point was NVIDIA's Nemotron Cascade 8B model. Instead of retraining all eight billion parameters, this run used QLoRA: the base model was loaded in a memory-efficient 4-bit form and a small set of trainable adapter weights was attached to it. The adapter is only about 178 MiB, while the underlying model remains unchanged.

The model saw examples of complete coding-agent conversations rendered with Pi 0.82.0's exact system prompt and tool schemas. That matters because an agent must learn two things at once: how to write good code and how to express a tool call in the exact machine-readable shape the host expects.

Base model: nvidia/Nemotron-Cascade-8B at revision 276a779fe0c1fe2d63a68fd573fec7b4b0a1907e
Method: 4-bit NF4 QLoRA, rank 32, alpha 64, dropout 0.05
Context limit during training: 20,480 tokens
Serving context: 32,768 tokens
Reasoning mode: Off; the model is served with Hermes tool parsing only

The V5 dataset

V5 contains 1,118 accepted trajectories and 4,555 assistant actions. Exporting those conversations produced 4,119 training records and 436 validation records. The validation set is kept out of gradient updates so it can measure whether the model learned patterns that generalize beyond the examples used to change its weights.

A full tokenizer audit rendered every record before training: 18,953,799 prompt tokens and 3,438,933 supervised assistant tokens. No prompt token was accidentally included in the loss, no hidden <think> content appeared in the target labels, and the longest sequence was 19,082 tokens—safely below the 20,480-token training cap.

Why this is stronger than V4

The trajectories were pinned to the installed Pi version and its exact read, edit, write, and bash schemas. Fifty lower-quality Codex Spark trajectories were replaced with higher-quality alternatives before export.

How the training run went

The run completed all 515 optimizer steps in about 5 hours 25 minutes. Evaluation ran before training and every 50 steps. Loss fell sharply by step 50, then continued improving slowly until step 500. The final step was only 0.0000168 worse than the best checkpoint—far too small to claim a meaningful difference—so the simpler final adapter is the served model while checkpoint 500 remains archived.

Validation loss, shown on a logarithmic scale so the smaller late-run improvements remain visible.

Validation mean-token accuracy. This measures next-token imitation, not whether a complete coding task succeeds.

System health

A separate guard process sampled the machine every 30 seconds for 656 samples. It was configured to stop training after two readings at or above 84 °C, if available host memory fell below 4 GiB, or if free disk fell below 20 GiB. None of those guardrails fired.

Observed resource envelope
Resource	Average	Peak / minimum	Interpretation
GPU temperature	66.5 °C	76 °C peak	8 °C below stop threshold
GPU power	404 W	577 W peak	Within the card's nominal 575 W operating envelope
GPU memory	—	32,149 / 32,607 MiB	Very full, but no OOM
Host memory available	—	42.5 GiB minimum	Large safety margin
Swap used	0	0	No memory pressure
Disk available	—	669 GiB minimum	No storage pressure

Some unusually long sequences took one to five minutes because they forced more memory movement across PCIe. They recovered normally; there were no out-of-memory errors, thermal slowdowns, hardware error flags, or crashes.

Before and after

Concise base-model versus V5 comparison
Test	Base	V5	Reading
Complete Pi agent loops	0 / 9	3 / 9	V5 improvement
Expected code change made	0 / 9	3 / 9	V5 can act; base never completed a requested change
Used requested validation command	0 / 9	5 / 9	Workflow adherence improved
Finished with a response	5 / 9	6 / 9	Both can still run away on creation tasks
Frozen raw Svelte: compile	4 / 10	1 / 10	One-shot code quality regressed
Frozen raw Svelte: policy clean	3 / 10	5 / 10	V5 uses modern conventions more often
Frozen raw Svelte: fully clean	0 / 10	0 / 10	Neither model is dependable without a repair loop

What a successful V5 run looked like

Called read first with a valid object argument.
Called edit with an exact replacement that preserved behavior.
Called bash with the exact requested svelte-utils checks.
Inspected the clean result and then summarized the verified change.

The main failure mode was not ignorance of the workflow. The model sometimes produced the right JSON text but wrapped it as a string, causing Pi's schema validator to reject it. The create-from-scratch case also triggered runaway output large enough to exceed the benchmark's 50 MiB capture limit in every trial.

What the training terms mean

Token: A small unit of text. Code punctuation, identifiers, and words are split into tokens the model predicts one at a time.
Loss: A penalty for assigning too little probability to the correct next token. Lower is better when comparing the same model on the same held-out dataset.
Token accuracy: The fraction of supervised positions where the most likely next token exactly matched the target. A single wrong token can still break a JSON tool call, so 98% token accuracy does not mean 98% successful agent tasks.
Epoch: One pass through the training dataset. This run used one epoch to reduce the risk of memorizing the examples.
Batch and gradient accumulation: The GPU processed one example at a time and accumulated gradients across eight examples before each optimizer update, giving an effective batch size of eight.
Validation set: Examples not used to update weights. They detect whether the model improves beyond the exact training records, but only on the same kind of next-token imitation task.
QLoRA adapter: A compact set of learned corrections layered over a quantized base model. It makes fine-tuning feasible on one 32 GiB GPU and leaves the original model intact.

Trying V5 in Pi

The model is served on Morningstar through an OpenAI-compatible endpoint. Pi should use the provider ID local-svelte-agent, model ID pi-svelte-agent-v5, and reasoning disabled.

pi --model local-svelte-agent/pi-svelte-agent-v5

The provider base URL is https://morningstar.otter-hawksbill.ts.net:10001/v1. The checked-in example configuration is training/pi-model-v5.example.json.

Use it interactively

Watch each tool call and keep approval enabled. The benchmark shows that the model may need a retry and should not yet be trusted to make broad unattended changes.

Recommendation

Keep V5 as the best Nemotron Pi experiment, but do not promote it as a reliable daily driver. The next dataset should target the observed boundary directly: many short examples contrasting object-valued tool arguments with accidentally stringified JSON, more create-file trajectories that terminate promptly, and negative examples that stop verbose internal planning from entering normal output.

After that cleanup, repeat this exact nine-case Pi benchmark before moving the recipe to the larger Qwen model. The larger model should improve capacity, but only if the dataset and serving stack agree on the same tool-call representation.