THE CRAWLER — DEFINITIONFrugendorff → hybrid
A recursive weight-sharing transformer module. A small set of K unique blocks is reused N times in sequence to produce effective depth K·N from a smaller stored parameter set; the saved budget is reinvested into wider hidden states. Inherited from the Frugendorff line, hardened into a hybrid form for int6 quantization survival.
Origin · 2026-03-19First introduced as NUM_LOOPS in commit “feat: add fractal weight sharing via NUM_LOOPS env var.” When NUM_LOOPS > 1 the model reuses num_layers blocks in a loop instead of the U-Net skip pattern. A learned per-loop position embedding p_loop disambiguates which pass the shared weights are on.
Forward · Frugendorffhi+1 = TransformerBlockθ[i mod K](hi + p_loop⌊i / K⌋)
i ∈ {0, 1, …, K·N − 1}
Origin · ResultProof of concept at 3 × 3, 864-dim hit −7.1% BPB vs the unique-layer baseline at fewer parameters. Gravity losses on early loops converged to ≈ [0.13, 0.13, 0.70] — the model learned to suppress early loop contributions.
Failure modesAt competition budgets, pure recursion surfaced two compounding issues. (1) Shared-weight gradient conflict — the same θ must simultaneously map early → mid and mid → late representations, and short budgets cannot find the compromise. (2) Quantization catastrophe — int6 rounding error compounds multiplicatively across loop iterations (post-quant BPB > 5× pre-quant at 3+ loops). Frugendorff was declared dead 2026-03-27, then rebuilt as a hybrid 2026-03-29.
Final hybrid formA flat stack frames a small shared core. NUM_FLAT_LAYERS standard transformer blocks process the embedded input; NUM_CRAWLER_LAYERS shared-weight blocks then loop CRAWLER_LOOPS times. Each pass k uses a distinct RoPE scale from CRAWLER_LOOP_ROPE_SCALES and is conditioned by a content-derived instruction vector π(h) of dimension INST_DIM. Loop-aware GPTQ, Hessian collection on the shared block, and int8 / int6 sliding-window export keep the model under the 16 MB cap.
Forward · Hybridh0 = StackFlatθf(embed(x))
for k = 1 … K: hk = CrawlerBlockθc(hk−1, π(hk−1), RoPEk)
y = Head(hK)
Goals(1) effective depth from weight reuse, not from new parameters. (2) reinvest the saved parameter budget into wider hidden states. (3) differentiate each pass via loop-position RoPE and a content-derived instruction, so the same θ does different jobs each loop. (4) survive int6 quantization at the competition cap. (5) act as the recurrent core of a flat-around-crawler topology, so the model has both standard transformer expressivity and recurrent depth.
CRAWLER-SAFE TRAIN/EVAL SURFACEx299
Crawler-safe training/eval surface around a widened shared-crawler stack with final int6 and int8 checks.
BODYvocab_size=1024 · model_dim=512 · num_heads=8 · num_kv_heads=4 · shared crawler block · looped crawler passes
FEATURESBigramHash · XSA · RoPE · crawler loop rope scales · instruction bottleneck
QUANT / EVALint6 roundtrip · int6 sliding-window eval · int8 zlib roundtrip · 8xH100 · 600s wallclock
INSTRUCTED CRAWLER RECURRENCE FAMILYx20
Crawler-oriented recurrence experiments that tested instruction-conditioned loop behavior and DeltaNet-adjacent crawler variants.
BODYshared crawler recurrence path · content-derived loop instructions · crawler loop rope scaling · DeltaNet crawler recurrence variants
TEST SURFACEGB10 micro probes · H100/runpod follow-up logs · commit-level architecture entries · crawler branch retained as the strongest follow-on direction
WIDENED CRAWLER BOTTLENECKx15
Widened crawler bottleneck body: 5 flat blocks, 1 shared crawler block, 5 flat blocks.
BODYMODEL_DIM=512 · NUM_FLAT_LAYERS=5 · NUM_CRAWLER_LAYERS=1 · CRAWLER_LOOPS=3 · USE_CRAWLER=1 · num_heads=8 · num_kv_heads=4
CRAWLERCRAWLER_MLP_MULT=6.0 · INST_DIM=32 · CRAWLER_QUANT_INT8=1 · CRAWLER_LOOP_ROPE_SCALES=9,1,1 · CRAWLER_TAP_DIM=32 · CRAWLER_TAP_LAYERS=all
PACKAGINGBIGRAM_VOCAB_SIZE=2048 · XSA_LAST_N=11 · ROPE_DIMS=16 · SKIP_EMA=1 · SKIP_GPTQ=1 · MTP_NUM_HEADS=0
LOOP-INSTRUCTED DELTANET CRAWLER BODYx14
Crawler-capable GQA body with loop instruction projection, optional DeltaNet memory, and loop-aware GPTQ experiments.
BASE BODYvocab_size=1024 · num_layers=11 · model_dim=512 · num_heads=8 · num_kv_heads=4 · mlp_mult=3.0 · BigramHash 2048x128 · XSA last_n=11 · rope_dims=16
CRAWLER BODYflat transformer blocks around shared crawler block · crawler_mlp_mult=4.0 default · inst_dim=32 · loop instruction projection · crawler_loop_rope_scales=1,1,1 default · optional DeltaNet memory
QUANT PATHloop-aware GPTQ option · crawler Hessian collection · EMA / no-EMA ablations · int6 sliding-window export path
12L GQA MIXED-BIT TOWERx11
12-layer GQA tower: 8/4 GQA, RoPE-16, Bigram 2048, XSA on the last 11 layers, and mixed-bit packaging.
BODYnum_layers=12 · num_heads=8 · num_kv_heads=4 · rope_dims=16 · xsa_last_n=11 · BigramHash vocab=2048
QUANT / EVALattention int5 · mlp int6 · aux int6 · embed int8 · other int8 · Brotli artifact · 16 MB artifact cap
RUN SHAPE8xH100 · train_seq_len=2048 · eval_seq_len=2048 · 600s wallclock
CRAWLER-ONLY NGRAM EXPORT BODYx8
Crawler body with the fast ngram finish: crawler architecture retained, CPU-heavy phrase/regime paths removed.
MODEL BODYcrawler-only instructed-recurrence body · DELTA_NET_HEADS=0 · model_dim=512 · num_heads=8 · num_kv_heads=4 · int6+zstd target under 11 MB
EVAL FINISHngram9 eval stack · NGRAM_DIRICHLET=1 · remove NGRAM_CHUNK_TOKENS=65536 · remove PHRASE_CACHE · remove REGIME_TRACKER
RECURRENT GREEN-BRANCH PROTOTYPEx8
Fast prototype line around green-branch recurrence, complement mixing, and residual/value-routing variants before later crawler consolidation.
BODYgreen-branch recurrent path · value-residual variants · complement-mix alpha variants · high-speed prototype iteration
SWEEPSremote fast A/B runs · synthetic branch variants · green v7 follow-up · warmdown and SWA schedule probes
11L GQA U-NET BODYx7
11-layer dim-512 GQA U-Net transformer base with XSA, BigramHash, value embedding, and SmearGate.
BODYvocab_size=1024 · num_layers=11 · model_dim=512 · num_heads=8 · num_kv_heads=4 · head_dim=64 · mlp_mult=3.0 · tie_embeddings=1
STRUCTUREencoder_layers=5 · decoder_layers=6 · skip_weights=5 · GQA attention · RMSNorm · relu_sq MLP · Muon matrix optimizer path
FEATURESBigramHash vocab=2048 dim=128 · XSA last_n=11 · rope_dims=16 · VE dim=128 layers=9,10 · SmearGate · logit_softcap=30
9-FLAT + 1-CRAWLER LOOP BODYx5
9-flat plus 1-crawler loop body: flat U-Net context around a shared crawler block fired twice.
BODYvocab_size=1024 · model_dim=512 · num_heads=8 · num_kv_heads=4 · NUM_FLAT_LAYERS=9 · NUM_CRAWLER_LAYERS=1 · CRAWLER_LOOPS=2 · USE_CRAWLER=1
CRAWLERCRAWLER_MLP_MULT=6.0 · INST_DIM=32 · CRAWLER_MLP_CHOKE_DIM=0 · CRAWLER_LOOP_ROPE_SCALES=9,1,1 · CRAWLER_MLP_LEAKY_SLOPE=0.5
PACKAGINGLOOP_AWARE_GPTQ=1 · BigramHash 2048x128 · XSA last_n=11 · rope_dims=16 · VE dim=128 layers=9,10
HYBRID GATEDDELTANET-ATTENTION BODYx2
Hybrid GreenRod branch combining GatedDeltaNet-style memory with attention-body changes as part of the March 27 exploratory campaign.
BODYGatedDeltaNet-attention hybrid · X1 experimental branch · H100/runpod measurement path
DIM-576 ATTENTION-KERNEL BODYx1
A 12-layer dim-576 model body plus the custom attention-kernel benchmark shape used against FA3.
MODEL BODYnum_layers=12 · model_dim=576 · embedding_dim=576 · matrix_bits=5 · xsa_last_n=11 · parallel_residual_start=8 · model_params=48551136
KERNEL SHAPEbatch=2 · seq_len=2048 · q_heads=8 · kv_heads=4 · head_dim=64 · custom attention path compared directly to FA3
PROMOTED KERNELDQ=128,64,8,5 · DKDV=128,64,8,3 · DKDV_MAXNREG=256 · QMR=1984 for 2x primary · QMR=1664 single-GPU anchor
