Follow-up to the first shuffle-transfer run. The same aggressive transformation (permute all 12 MLPs and all 144 attention heads, reset the tied embedding, discard the optimizer) was applied to snapshots of one source model at 500, 1,500, 3,000, and 10,000 training steps. Each transformed model was retrained for 3,000 steps against the same scratch trajectory.
The two most-trained sources are the closest thing to "converged" here, and both lose: the 3,000-step source (val loss 3.47, at the run's loss plateau) finished +0.0404 worse than scratch, and the 10,000-step source (val loss 3.15) finished +0.0858 worse and trailed at every one of the 13 matched checkpoints. Both penalties are Holm-significant, and the damage grows with source training. Positive transfer appeared only from a barely trained 500-step source (−0.0055) — the opposite of a converged model. Caveats: even 10k steps is not fully converged (loss was still falling), and this is one seed and one shuffle permutation.
Components shuffled from a 500-step source beat scratch at all 13 matched checkpoints and finished −0.00548 validation loss better. A 1,500-step source gave the largest early speedup (−0.160 at retrain step 500) but crossed behind scratch and finished slightly worse. The 3,000-step source finished +0.0404 worse, and the 10,000-step source was worse at every checkpoint (+0.0858 final). All four final differences are significant after Holm correction on paired validation batches. Interpretation: partially trained components carry broadly reusable structure, but continued training specializes them to their original layer/head context — after shuffling, that specialization becomes a liability. Single source trajectory and one shuffle seed, so cross-seed generality is untested.
Final validation loss after 3,000 retraining steps (1.47B tokens), versus scratch at 3.4720. Lower is better.
Darker blue = older source model. The ordering is clean and monotonic in age: every added unit of source training makes the shuffled components less useful to the retrained model, even though the source itself keeps improving (5.27 → 3.15 validation loss from step 500 to 10,000).
Linear axes · 13 matched checkpoints per condition · identical LR schedule (20k-step cosine horizon)
Transfer − scratch validation loss · log-x · below zero = transfer ahead
After 3,000 retrain steps · whiskers = paired 95% CI (tiny) · log-x
Validation-loss AUC − scratch AUC · integrates the early speedup · log-x
Retrain steps 250–3,000 · all five runs
EMA (α 0.15) of per-10-step loss · retrain phase
One uninterrupted 10,000-step scratch run · log–log · marked points are the four snapshot ages
Each checkpoint compares the same 100 validation batches, paired per batch: 20,000 bootstrap resamples for the 95% CI and a normal paired test with Holm correction across all 52 condition/checkpoint comparisons. This measures evaluation-sample uncertainty only — not variance across training seeds or shuffle permutations.
| Source steps | Source val loss | Final val loss | Final Δ vs scratch | Paired 95% CI | Holm p | AUC Δ |
|---|---|---|---|---|---|---|
| 500 | 5.2708 | 3.4666 | −0.00548 | [−0.00717, −0.00379] | 2.1e−09 | −0.03548 |
| 1,500 | 3.8015 | 3.4851 | +0.01306 | [+0.01140, +0.01476] | 1.1e−50 | −0.01983 |
| 3,000 | 3.4720 | 3.5124 | +0.04039 | [+0.03870, +0.04213] | ≈0 | +0.03932 |
| 10,000 | 3.1530 | 3.5578 | +0.08580 | [+0.08382, +0.08783] | ≈0 | +0.13483 |
Validation loss difference to scratch at each retrain checkpoint. Green = transfer ahead of scratch.
| Step | Scratch val | Δ src 500 | Δ src 1,500 | Δ src 3,000 | Δ src 10,000 |
|---|
Every snapshot received the identical transformation (same realized permutations and embedding reset, seed 20260727), isolating source-training age as the only variable.
Automated verification (audit_ablation.py) of the completed artifacts.