Experiment report · nanoGPT · single seed

Do shuffled GPT-2 components transfer useful structure?

A trained GPT-2-small had its 12 MLP modules and all 144 attention heads independently permuted across layers, its tied token embedding reinitialized, and its optimizer state discarded — then it was retrained for the same 3,000 steps as a from-scratch baseline.

host: ubuntu@sun path: /home/ubuntu/gpt2-shuffle-transfer run date: 2026-07-27 seed: 20260727

Result: no evidence of useful transfer.

The shuffled model learned at almost the same rate as the scratch baseline but finished slightly worse — final validation loss +0.0381 higher and validation-loss AUC +0.0266 higher. It led at only 2 of 13 matched checkpoints (steps 500 and 750) and trailed at every checkpoint from step 1,000 onward. Single-seed result: variance across initializations and permutations is not estimated.

Headline numbers

Both phases: 3,000 AdamW steps on OpenWebText, global batch 491,520 tokens (1.47B tokens per phase). Lower loss is better.

Final val loss · baseline

3.4710

trained from scratch

Final val loss · shuffled

3.5091

+0.0381 vs baseline

Val-loss AUC gap

+0.0266

integrated over 3,000 steps

Checkpoints shuffled led

2 / 13

steps 500 and 750 only

Learning curves

Matched evaluation every 250 steps (100 batches each); training loss logged every 10 steps. The log–log views are the scaling-law lens: if shuffled components carried reusable structure, the orange curve would sit durably below blue or bend earlier. It does neither — after a brief mid-warmup lead around steps 500–750, it settles into a thin, persistent penalty.

Validation loss vs optimizer step

Baseline (scratch) Shuffled transfer

Linear axes · full trajectory from the reinitialized start

Validation loss · log–log

Steps 250–3,000 (step 0 has no log position)

Training loss · log–log

Every 10 steps · thin = raw · bold = EMA (α 0.15)

Checkpoint gap: shuffled − baseline

worse better

Validation loss difference · y clipped to ±0.10 (step 0 is +0.273)

Relative gap over training

(shuffled − baseline) / baseline · matched checkpoints · log-x

Training-loss gap · log-x

EMA(shuffled) − EMA(baseline) every 10 steps

Learning-rate schedule

Identical for both phases · 1,000-step warmup, then cosine decay toward 6e-5 (barely begun by step 3,000)

Hover any panel for exact values. One optimizer step = 491,520 tokens, so step 1,000 ≈ 0.49B tokens and step 3,000 ≈ 1.47B tokens.

What was shuffled

The baseline checkpoint (val loss 3.4710 at step 3,000) was transformed with seed 20260727, then retrained under identical config with a fresh optimizer.

1MLP bag. All 12 complete MLP modules permuted across layers without replacement; both linear layers of each MLP travel together.

2Attention-head bag. All 144 (layer, head) pairs permuted across every head slot without replacement; each head carries its Q, K, V rows plus matching output-projection columns.

3Embedding reset. The tied token-embedding / LM-decoder matrix reinitialized once (normal, σ = 0.02); the two uses remain tied.

4Optimizer reset. AdamW state discarded entirely; retraining starts with fresh moments and step count.

–Preserved: positional embeddings, per-block layer norms, final layer norm, and layer-level structure.

Training configuration

Model	GPT-2-small · 123.59M params · 12L / 12H / 768d
Context / vocab	1,024 tokens / 50,304
Global batch	491,520 tokens per step (12 seq/GPU × 40 accum)
Steps / tokens per phase	3,000 / 1.47456B
Optimizer	fused AdamW · β (0.9, 0.95) · wd 0.1
Learning rate	6e-4 peak → 6e-5 min · 1,000-step warmup · cosine
Precision / hardware	bfloat16 · 4 × RTX 4090 D · DDP · ~31% MFU
Data	OpenWebText · 9.04B train / 4.43M val tokens (GPT-2 BPE)
Evaluation	100 batches every 250 steps

All matched checkpoints

Step	Baseline val loss	Shuffled val loss	Shuffled − baseline	Relative

Integrity audit

An automated audit (audit_results.py) verified the completed artifacts against the experiment’s invariants.

status: pass

12 MLP permutation without replacement
144 attention-head permutation without replacement
Q/K/V rows and output-projection columns moved together
Tied token-embedding/decoder matrix reinitialized
Position embeddings preserved
Source optimizer state absent at retraining boundary
Fresh optimizer state present after retraining
Matched 3,000-step metrics and plot present

Source: ubuntu@sun:/home/ubuntu/gpt2-shuffle-transfer — see README.md, runs/loss_comparison.csv, runs/*/metrics.jsonl, runs/audit.json, and shuffle_checkpoint.py for the transformation implementation. Based on nanoGPT 3adf61e1. Reproduce with ./run_all.sh (requires exactly 4 GPUs).