THE CRAWLER — DEFINITIONFrugendorff → hybrid
A recursive weight-sharing transformer module. A small set of K unique blocks is reused N times in sequence to produce effective depth K·N from a smaller stored parameter set; the saved budget is reinvested into wider hidden states. Inherited from the Frugendorff line, hardened into a hybrid form for int6 quantization survival.
Origin · 2026-03-19First introduced as NUM_LOOPS in commit “feat: add fractal weight sharing via NUM_LOOPS env var.” When NUM_LOOPS > 1 the model reuses num_layers blocks in a loop instead of the U-Net skip pattern. A learned per-loop position embedding p_loop disambiguates which pass the shared weights are on.
Forward · Frugendorffhi+1 = TransformerBlockθ[i mod K](hi + p_loop⌊i / K⌋)
i ∈ {0, 1, …, K·N − 1}
i ∈ {0, 1, …, K·N − 1}
Origin · ResultProof of concept at 3 × 3, 864-dim hit −7.1% BPB vs the unique-layer baseline at fewer parameters. Gravity losses on early loops converged to ≈ [0.13, 0.13, 0.70] — the model learned to suppress early loop contributions.
Failure modesAt competition budgets, pure recursion surfaced two compounding issues. (1) Shared-weight gradient conflict — the same θ must simultaneously map early → mid and mid → late representations, and short budgets cannot find the compromise. (2) Quantization catastrophe — int6 rounding error compounds multiplicatively across loop iterations (post-quant BPB > 5× pre-quant at 3+ loops). Frugendorff was declared dead 2026-03-27, then rebuilt as a hybrid 2026-03-29.
Final hybrid formA flat stack frames a small shared core. NUM_FLAT_LAYERS standard transformer blocks process the embedded input; NUM_CRAWLER_LAYERS shared-weight blocks then loop CRAWLER_LOOPS times. Each pass k uses a distinct RoPE scale from CRAWLER_LOOP_ROPE_SCALES and is conditioned by a content-derived instruction vector π(h) of dimension INST_DIM. Loop-aware GPTQ, Hessian collection on the shared block, and int8 / int6 sliding-window export keep the model under the 16 MB cap.
Forward · Hybridh0 = StackFlatθf(embed(x))
for k = 1 … K: hk = CrawlerBlockθc(hk−1, π(hk−1), RoPEk)
y = Head(hK)
for k = 1 … K: hk = CrawlerBlockθc(hk−1, π(hk−1), RoPEk)
y = Head(hK)
Goals(1) effective depth from weight reuse, not from new parameters. (2) reinvest the saved parameter budget into wider hidden states. (3) differentiate each pass via loop-position RoPE and a content-derived instruction, so the same θ does different jobs each loop. (4) survive int6 quantization at the competition cap. (5) act as the recurrent core of a flat-around-crawler topology, so the model has both standard transformer expressivity and recurrent depth.