DAILY RESEARCH REPORTBACK TO MASTER SHEET2026-04-01

val_bpb · 211 of 855 · 0 warnings
audit summary
{
  "date": "2026-04-01",
  "fatal": 0,
  "headline_metric": "val_bpb",
  "raw_records": 855,
  "render_records": 211,
  "test_rows": 6,
  "warnings": 0
}
153 commits on 2026-04-01
  1. 2e445b72 05:00:03 parameter-golf-lab Accept custom hopper FA3 path while keeping strict SOTA gating
  2. 2e445b72 05:00:03 sota_crawler Accept custom hopper FA3 path while keeping strict SOTA gating
  3. 2e445b72 05:00:03 SOTA_FINAL Accept custom hopper FA3 path while keeping strict SOTA gating
  4. 0ed1ea30 05:04:00 parameter-golf-lab Prefer working FA3 provider and fail clearly on ABI mismatch
  5. 0ed1ea30 05:04:00 sota_crawler Prefer working FA3 provider and fail clearly on ABI mismatch
  6. 0ed1ea30 05:04:00 SOTA_FINAL Prefer working FA3 provider and fail clearly on ABI mismatch
  7. 580bf081 05:06:03 parameter-golf-lab Disable FA3 wheel installs by default for custom-head pods
  8. 580bf081 05:06:03 sota_crawler Disable FA3 wheel installs by default for custom-head pods
  9. 580bf081 05:06:03 SOTA_FINAL Disable FA3 wheel installs by default for custom-head pods
  10. 3203ed9c 05:24:29 parameter-golf-lab Restore custom FA3 discovery in pod_setup.sh
  11. 3203ed9c 05:24:29 sota_crawler Restore custom FA3 discovery in pod_setup.sh
  12. 3203ed9c 05:24:29 SOTA_FINAL Restore custom FA3 discovery in pod_setup.sh
  13. 03698c1b 05:26:39 parameter-golf-lab Lock pod_setup.sh snapshot to vault (2026-04-01, cu124, custom FA3)
  14. 03698c1b 05:26:39 sota_crawler Lock pod_setup.sh snapshot to vault (2026-04-01, cu124, custom FA3)
  15. 03698c1b 05:26:39 SOTA_FINAL Lock pod_setup.sh snapshot to vault (2026-04-01, cu124, custom FA3)
  16. e7a5944b 05:30:34 parameter-golf-lab Re-enable FA3 wheel install by default (restore cu124 Hopper FA3)
  17. e7a5944b 05:30:34 sota_crawler Re-enable FA3 wheel install by default (restore cu124 Hopper FA3)
  18. e7a5944b 05:30:34 SOTA_FINAL Re-enable FA3 wheel install by default (restore cu124 Hopper FA3)
  19. 0ae805f8 05:30:58 parameter-golf-lab Update vault pod_setup snapshot to e7a5944 (FA3 wheel default restored)
  20. 0ae805f8 05:30:58 sota_crawler Update vault pod_setup snapshot to e7a5944 (FA3 wheel default restored)
  21. 0ae805f8 05:30:58 SOTA_FINAL Update vault pod_setup snapshot to e7a5944 (FA3 wheel default restored)
  22. 49c544fc 05:40:01 parameter-golf-lab Fix FA3 install: replace dead pytorch WHL URL with GitHub releases
  23. 49c544fc 05:40:01 sota_crawler Fix FA3 install: replace dead pytorch WHL URL with GitHub releases
  24. 49c544fc 05:40:01 SOTA_FINAL Fix FA3 install: replace dead pytorch WHL URL with GitHub releases
  25. f825d029 05:42:41 parameter-golf-lab Add FRESHPOD_OF_BELAIR.sh — original working pod setup (fa91fda)
  26. f825d029 05:42:41 sota_crawler Add FRESHPOD_OF_BELAIR.sh — original working pod setup (fa91fda)
  27. f825d029 05:42:41 SOTA_FINAL Add FRESHPOD_OF_BELAIR.sh — original working pod setup (fa91fda)
  28. 4fd4c94c 05:44:33 parameter-golf-lab Restore vault pod_setup to fa91fda original (FRESHPOD_OF_BELAIR)
  29. 4fd4c94c 05:44:33 sota_crawler Restore vault pod_setup to fa91fda original (FRESHPOD_OF_BELAIR)
  30. 4fd4c94c 05:44:33 SOTA_FINAL Restore vault pod_setup to fa91fda original (FRESHPOD_OF_BELAIR)
  31. c81ca06b 05:46:28 parameter-golf-lab 11L GQA U-Net body: strip agent-injected torch/CUDA version gate
  32. c81ca06b 05:46:28 sota_crawler 11L GQA U-Net body: strip agent-injected torch/CUDA version gate
  33. c81ca06b 05:46:28 SOTA_FINAL 11L GQA U-Net body: strip agent-injected torch/CUDA version gate
  34. cc1fc254 06:05:38 parameter-golf-lab Add install_fa3_cu124.sh — FA3 wheel installer for cu124/torch2.4
  35. cc1fc254 06:05:38 sota_crawler Add install_fa3_cu124.sh — FA3 wheel installer for cu124/torch2.4
  36. cc1fc254 06:05:38 SOTA_FINAL Add install_fa3_cu124.sh — FA3 wheel installer for cu124/torch2.4
  37. 6bc1d593 06:09:43 parameter-golf-lab Add setup_pod_cu130.sh — install torch cu130, verify FA3
  38. 6bc1d593 06:09:43 sota_crawler Add setup_pod_cu130.sh — install torch cu130, verify FA3
  39. 6bc1d593 06:09:43 SOTA_FINAL Add setup_pod_cu130.sh — install torch cu130, verify FA3
  40. da6c41c3 06:58:12 parameter-golf-lab Harden pod_setup.sh + restore run.sh version gate
  41. da6c41c3 06:58:12 sota_crawler Harden pod_setup.sh + restore run.sh version gate
  42. da6c41c3 06:58:12 SOTA_FINAL Harden pod_setup.sh + restore run.sh version gate
  43. bc8a34bd 06:59:23 parameter-golf-lab Vault-lock pod_setup (cu124, sympy pin, hard assertions)
  44. bc8a34bd 06:59:23 sota_crawler Vault-lock pod_setup (cu124, sympy pin, hard assertions)
  45. bc8a34bd 06:59:23 SOTA_FINAL Vault-lock pod_setup (cu124, sympy pin, hard assertions)
  46. 4ffaf675 07:01:49 parameter-golf-lab pod_setup: add source-build fallback for FA3 when wheel 404s
  47. 4ffaf675 07:01:49 sota_crawler pod_setup: add source-build fallback for FA3 when wheel 404s
  48. 4ffaf675 07:01:49 SOTA_FINAL pod_setup: add source-build fallback for FA3 when wheel 404s
  49. bffbd34d 07:02:06 parameter-golf-lab pod_setup: add wheel to pip deps (needed for FA3 source build)
  50. bffbd34d 07:02:06 sota_crawler pod_setup: add wheel to pip deps (needed for FA3 source build)
  51. bffbd34d 07:02:06 SOTA_FINAL pod_setup: add wheel to pip deps (needed for FA3 source build)
  52. ad809f2c 07:03:21 parameter-golf-lab pod_setup: add 2.8.3 wheels (cp312 support), fix cross-device fallback
  53. ad809f2c 07:03:21 sota_crawler pod_setup: add 2.8.3 wheels (cp312 support), fix cross-device fallback
  54. ad809f2c 07:03:21 SOTA_FINAL pod_setup: add 2.8.3 wheels (cp312 support), fix cross-device fallback
  55. 8743330c 07:43:04 parameter-golf-lab Add fresh pod wrapper for 11L GQA U-Net body III SLOT
  56. 8743330c 07:43:04 sota_crawler Add fresh pod wrapper for 11L GQA U-Net body III SLOT
  57. 8743330c 07:43:04 SOTA_FINAL Add fresh pod wrapper for 11L GQA U-Net body III SLOT
  58. 4421a8d9 11:46:18 parameter-golf-lab Record: 4096-Vocab + Larger Model + High WD + Simplifications — val_bpb 1.09785 (3-seed mean)
  59. 4421a8d9 11:46:18 sota_crawler Record: 4096-Vocab + Larger Model + High WD + Simplifications — val_bpb 1.09785 (3-seed mean)
  60. 4421a8d9 11:46:18 SOTA_FINAL Record: 4096-Vocab + Larger Model + High WD + Simplifications — val_bpb 1.09785 (3-seed mean)
  61. 0622c325 15:56:56 parameter-golf-lab 11L GQA U-Net body III SLOT copy: fix SLOT batch gating and pack int6 to 6-bit
  62. 0622c325 15:56:56 sota_crawler 11L GQA U-Net body III SLOT copy: fix SLOT batch gating and pack int6 to 6-bit
  63. 0622c325 15:56:56 SOTA_FINAL 11L GQA U-Net body III SLOT copy: fix SLOT batch gating and pack int6 to 6-bit
  64. 7c98509d 16:53:45 parameter-golf-lab Restore pod_setup.sh to pre-storm version (1ea0901)
  65. 7c98509d 16:53:45 sota_crawler Restore pod_setup.sh to pre-storm version (1ea0901)
  66. 7c98509d 16:53:45 SOTA_FINAL Restore pod_setup.sh to pre-storm version (1ea0901)
  67. 1f0ab87f 16:56:15 parameter-golf-lab Restore all run scripts to pre-storm versions (9332369)
  68. 1f0ab87f 16:56:15 sota_crawler Restore all run scripts to pre-storm versions (9332369)
  69. 1f0ab87f 16:56:15 SOTA_FINAL Restore all run scripts to pre-storm versions (9332369)
  70. 9e0905bf 16:57:05 parameter-golf-lab Remove storm debris scripts
  71. 9e0905bf 16:57:05 sota_crawler Remove storm debris scripts
  72. 9e0905bf 16:57:05 SOTA_FINAL Remove storm debris scripts
  73. f435825b 17:01:47 parameter-golf-lab Relax CUDA gate in 11L GQA U-Net body III SLOT run.sh: 12.4* → 12.*
  74. f435825b 17:01:47 sota_crawler Relax CUDA gate in 11L GQA U-Net body III SLOT run.sh: 12.4* → 12.*
  75. f435825b 17:01:47 SOTA_FINAL Relax CUDA gate in 11L GQA U-Net body III SLOT run.sh: 12.4* → 12.*
  76. fa693615 17:09:03 parameter-golf-lab Add 11L GQA U-Net body: SLOT + true 6-bit int6 packing
  77. fa693615 17:09:03 sota_crawler Add 11L GQA U-Net body: SLOT + true 6-bit int6 packing
  78. fa693615 17:09:03 SOTA_FINAL Add 11L GQA U-Net body: SLOT + true 6-bit int6 packing
  79. cd8825de 17:38:46 parameter-golf-lab 11L GQA U-Net body III SLOT: add seed300 run log (2026-04-01 17:20)
  80. cd8825de 17:38:46 sota_crawler 11L GQA U-Net body III SLOT: add seed300 run log (2026-04-01 17:20)
  81. cd8825de 17:38:46 SOTA_FINAL 11L GQA U-Net body III SLOT: add seed300 run log (2026-04-01 17:20)
  82. 53724322 17:39:26 parameter-golf-lab pod_setup: smart FA3 install — Dao-AILab wheels + system discovery
  83. 53724322 17:39:26 sota_crawler pod_setup: smart FA3 install — Dao-AILab wheels + system discovery
  84. 53724322 17:39:26 SOTA_FINAL pod_setup: smart FA3 install — Dao-AILab wheels + system discovery
  85. aa1a24cf 18:30:17 parameter-golf-lab Add setup_cu124_fa3_venv.sh — automated cu124+FA3 bridge setup
  86. aa1a24cf 18:30:17 sota_crawler Add setup_cu124_fa3_venv.sh — automated cu124+FA3 bridge setup
  87. aa1a24cf 18:30:17 SOTA_FINAL Add setup_cu124_fa3_venv.sh — automated cu124+FA3 bridge setup
  88. dd48eac2 18:34:41 parameter-golf-lab setup_cu124_fa3_venv: symlink .so instead of copy, add disk cleanup
  89. dd48eac2 18:34:41 sota_crawler setup_cu124_fa3_venv: symlink .so instead of copy, add disk cleanup
  90. dd48eac2 18:34:41 SOTA_FINAL setup_cu124_fa3_venv: symlink .so instead of copy, add disk cleanup
  91. abd7f317 20:20:01 parameter-golf-lab crawler: add crawler variant 2k interaction ablation sequence
  92. abd7f317 20:20:01 sota_crawler crawler: add crawler variant 2k interaction ablation sequence
  93. abd7f317 20:20:01 SOTA_FINAL crawler: add crawler variant 2k interaction ablation sequence
  94. 8d806f9f 21:10:22 parameter-golf-lab crawler: record crawler variant results and add crawler variant 4x interaction series
  95. 8d806f9f 21:10:22 sota_crawler crawler: record crawler variant results and add crawler variant 4x interaction series
  96. 8d806f9f 21:10:22 SOTA_FINAL crawler: record crawler variant results and add crawler variant 4x interaction series
  97. fabad3a8 21:10:59 parameter-golf-lab pod_setup: enforce true FA3 runtime (remove FA2 wheel fallback)
  98. fabad3a8 21:10:59 sota_crawler pod_setup: enforce true FA3 runtime (remove FA2 wheel fallback)
  99. fabad3a8 21:10:59 SOTA_FINAL pod_setup: enforce true FA3 runtime (remove FA2 wheel fallback)
  100. 6929756f 21:14:38 parameter-golf-lab scripts: add fast trimmed-kernel FA3 installer; remove broken cu124 bridge
  101. 6929756f 21:14:38 sota_crawler scripts: add fast trimmed-kernel FA3 installer; remove broken cu124 bridge
  102. 6929756f 21:14:38 SOTA_FINAL scripts: add fast trimmed-kernel FA3 installer; remove broken cu124 bridge
  103. 822bf2fd 21:16:11 parameter-golf-lab setup: vault fast FA3 installer; remove best-effort FA3 bootstrap path
  104. 822bf2fd 21:16:11 sota_crawler setup: vault fast FA3 installer; remove best-effort FA3 bootstrap path
  105. 822bf2fd 21:16:11 SOTA_FINAL setup: vault fast FA3 installer; remove best-effort FA3 bootstrap path
  106. 41fe618a 21:22:46 parameter-golf-lab fa3: canonical quick build uses --no-build-isolation
  107. 41fe618a 21:22:46 sota_crawler fa3: canonical quick build uses --no-build-isolation
  108. 41fe618a 21:22:46 SOTA_FINAL fa3: canonical quick build uses --no-build-isolation
  109. 1fcaa699 21:37:29 parameter-golf-lab 11L GQA U-Net body: disable torch._dynamo optimize_ddp by default to avoid inductor NaN compile failure
  110. 1fcaa699 21:37:29 sota_crawler 11L GQA U-Net body: disable torch._dynamo optimize_ddp by default to avoid inductor NaN compile failure
  111. 1fcaa699 21:37:29 SOTA_FINAL 11L GQA U-Net body: disable torch._dynamo optimize_ddp by default to avoid inductor NaN compile failure
  112. 9807abe7 21:38:35 parameter-golf-lab revert: restore 11L GQA U-Net body III SLOT train_gpt_slot.py exactly (no agent edits)
  113. 9807abe7 21:38:35 sota_crawler revert: restore 11L GQA U-Net body III SLOT train_gpt_slot.py exactly (no agent edits)
  114. 9807abe7 21:38:35 SOTA_FINAL revert: restore 11L GQA U-Net body III SLOT train_gpt_slot.py exactly (no agent edits)
  115. 192b22f3 21:49:32 parameter-golf-lab hotfix: add sitecustomize to disable dynamo optimize_ddp and suppress compile crashes
  116. 192b22f3 21:49:32 sota_crawler hotfix: add sitecustomize to disable dynamo optimize_ddp and suppress compile crashes
  117. 192b22f3 21:49:32 SOTA_FINAL hotfix: add sitecustomize to disable dynamo optimize_ddp and suppress compile crashes
  118. 4e18a407 21:54:11 parameter-golf-lab crawler: record crawler variant interaction ablation results
  119. 4e18a407 21:54:11 sota_crawler crawler: record crawler variant interaction ablation results
  120. 4e18a407 21:54:11 SOTA_FINAL crawler: record crawler variant interaction ablation results
  121. 210c59c8 22:08:56 parameter-golf-lab fa3: revert installer flags to historical 4-flag baseline
  122. 210c59c8 22:08:56 sota_crawler fa3: revert installer flags to historical 4-flag baseline
  123. 210c59c8 22:08:56 SOTA_FINAL fa3: revert installer flags to historical 4-flag baseline
  124. a0dda133 22:13:17 parameter-golf-lab crawler: add crawler variant big-swing 2k ablation sequence
  125. a0dda133 22:13:17 sota_crawler crawler: add crawler variant big-swing 2k ablation sequence
  126. a0dda133 22:13:17 SOTA_FINAL crawler: add crawler variant big-swing 2k ablation sequence
  127. 5b32740b 22:32:07 parameter-golf-lab fa3: add portable cu124 wheel build/install scripts
  128. 5b32740b 22:32:07 sota_crawler fa3: add portable cu124 wheel build/install scripts
  129. 5b32740b 22:32:07 SOTA_FINAL fa3: add portable cu124 wheel build/install scripts
  130. ed9222bc 01:30:45 parameter-golf-lab neural: add 2k same-checkpoint SLOT H2H probe
  131. ed9222bc 01:30:45 sota_crawler neural: add 2k same-checkpoint SLOT H2H probe
  132. ed9222bc 01:30:45 SOTA_FINAL neural: add 2k same-checkpoint SLOT H2H probe
  133. f0edf59e 01:41:25 parameter-golf-lab neural: relax h2h runner stack gate
  134. f0edf59e 01:41:25 sota_crawler neural: relax h2h runner stack gate
  135. f0edf59e 01:41:25 SOTA_FINAL neural: relax h2h runner stack gate
  136. fae52c12 01:51:25 parameter-golf-lab crawler: unify all ablations into single 4x runner
  137. fae52c12 01:51:25 sota_crawler crawler: unify all ablations into single 4x runner
  138. fae52c12 01:51:25 SOTA_FINAL crawler: unify all ablations into single 4x runner
  139. b96ad3a7 01:58:32 parameter-golf-lab neural: add 11L GQA U-Net body ii 2k comparison runner
  140. b96ad3a7 01:58:32 sota_crawler neural: add 11L GQA U-Net body ii 2k comparison runner
  141. b96ad3a7 01:58:32 SOTA_FINAL neural: add 11L GQA U-Net body ii 2k comparison runner
  142. f029757b 02:21:05 parameter-golf-lab scripts: add locked cu124 fa3 pod installer
  143. f029757b 02:21:05 sota_crawler scripts: add locked cu124 fa3 pod installer
  144. f029757b 02:21:05 SOTA_FINAL scripts: add locked cu124 fa3 pod installer
  145. dbd0b07a 02:45:20 parameter-golf-lab scripts: add clean pre-storm pod_setup copy (1ea0901)
  146. dbd0b07a 02:45:20 sota_crawler scripts: add clean pre-storm pod_setup copy (1ea0901)
  147. dbd0b07a 02:45:20 SOTA_FINAL scripts: add clean pre-storm pod_setup copy (1ea0901)
  148. 3736faa7 04:01:12 parameter-golf-lab crawler: add crawler variant depth sweep for NUM_FLAT_LAYERS 6-11
  149. 3736faa7 04:01:12 sota_crawler crawler: add crawler variant depth sweep for NUM_FLAT_LAYERS 6-11
  150. 3736faa7 04:01:12 SOTA_FINAL crawler: add crawler variant depth sweep for NUM_FLAT_LAYERS 6-11
  151. 833a9bcd 04:34:56 parameter-golf-lab vault: extract user-tested SOTA file from 99b790d (d70ec518, 2468 lines)
  152. 833a9bcd 04:34:56 sota_crawler vault: extract user-tested SOTA file from 99b790d (d70ec518, 2468 lines)
  153. 833a9bcd 04:34:56 SOTA_FINAL vault: extract user-tested SOTA file from 99b790d (d70ec518, 2468 lines)
DATE2026-04-01
DAY015
PRIMARY METRICVAL_BPB
TESTS RUN6 unique (6 valid / 0 TTT leakage)
ARCHITECTURE FAMILYWIDENED CRAWLER BOTTLENECK (×4) · SLOT-TUNED TRANSFORMER BODY (×2)
CLASSIFICATIONEXPERIMENTAL 100% (6)
ARCHITECTURE RECIPE
WIDENED CRAWLER BOTTLENECKx4

Widened crawler bottleneck body: 5 flat blocks, 1 shared crawler block, 5 flat blocks.

BODYMODEL_DIM=512 · NUM_FLAT_LAYERS=5 · NUM_CRAWLER_LAYERS=1 · CRAWLER_LOOPS=3 · USE_CRAWLER=1 · num_heads=8 · num_kv_heads=4
CRAWLERCRAWLER_MLP_MULT=6.0 · INST_DIM=32 · CRAWLER_QUANT_INT8=1 · CRAWLER_LOOP_ROPE_SCALES=9,1,1 · CRAWLER_TAP_DIM=32 · CRAWLER_TAP_LAYERS=all
PACKAGINGBIGRAM_VOCAB_SIZE=2048 · XSA_LAST_N=11 · ROPE_DIMS=16 · SKIP_EMA=1 · SKIP_GPTQ=1 · MTP_NUM_HEADS=0

SECTION 1.0 — TRAJECTORY

VAL_BPB for 2026-04-01
csv data
VAL_BPB SPREADnminmedianmaxdownload
WIDENED CRAWLER BOTTLENECK41.17491.17631.1768download
SLOT-TUNED TRANSFORMER BODY21.14391.14411.1442—
ALL VALID61.14391.17551.1768—

SECTION 2.0 — TEST RUNS

EXPERIMENTAL TEST RUNS — 6

#TIMEHWABLATIONVAL_BPBSTATUSΔ %OPEN
113:468×H100train_seed300.log1.1749OK↑10.1%
training_log
213:468×H100train_seed4.log1.1768OK↑10.3%
training_log
313:46—widened crawler bottleneck1.1761OK↑10.2%
details
413:468×H100train_seed444.log1.1765OK↑10.2%
training_log
515:248×H100 (runpod)slot_seed300_runpod_20260401_torch291_OVERSIZED.log1.1442OK↑7.2%
training_log
617:208×H100slot_f_seed300_20260401_172030.log1.1439OK↑7.2%
training_log