Configuration Parsing Warning:In UNKNOWN_FILENAME: "auto_map.AutoTokenizer" must be a string

Cognica-PoE-v1.0-1.3B-stage-chat

Paper: Product of Experts as Scalable Local Learning: Modular Construction at 1.3B Parameters (Jeong, 2026)

Not a production chat model. This release is empirical validation of PoE's dual-head specialist construction — paper §6.5.

A 107 M-parameter SFT specialist stage trained on top of the frozen Cognica-PoE-v1.0-1.3B-base. At this parameter scale (1.3 B base + 107 M specialist, d26 total), chat behavior is bounded by capacity, not by the specialist construction. What this artifact does demonstrate — reliably — is:

1. Base preservation is bit-identical. The base lm_head and layers 0–23 are frozen during SFT. Base-style continuation (<|bos|> + raw text) remains strong and coherent after specialist training — empirically verified on four scientific-knowledge probes (Sun, photosynthesis, quantum mechanics, language-learning advice). This is the live-model evidence for the paper's Δlogit = 0.0000 claim across 12 checkpoints.
2. Post-hoc specialist composition works. The specialist delta (300 MB, 16 tensors) can be loaded on demand and composed additively with the base (logits = lm_head_base(x) + lm_head_stage(x)). The loader supports arbitrary-depth chaining, so this stage is a building block for math / code / domain specialists that stack on top.
3. Delta-only distribution scales. The shipped stage is a 300 MB delta, not a 2.6 GB full-model copy. The cascade loader pulls the base once and assembles the d26 model in memory.

What this artifact does not demonstrate, and should not be compared against:

• Instruction following in the GPT-3.5 / GPT-4 sense.
• Arithmetic or multi-step reasoning. At 1.3 B, these are capacity-limited regardless of SFT data mix; including GSM8K train does not change this.
• Multi-turn context tracking across many turns.
• Strict format constraints ("answer yes or no", "exactly 3 bullets").

If you want a production chat model, this is not it. If you are evaluating PoE's architectural claims — whether a frozen-base + trainable-specialist construction can produce a composable SFT delta without degrading base capabilities — this is the artifact to look at.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "cognica/Cognica-PoE-v1.0-1.3B-stage-chat"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
 model_id,
 trust_remote_code=True,
 torch_dtype=torch.bfloat16,
 device_map="auto",
)
model.eval()

# Chat-style inference uses the SmolTalk special-token protocol:
# <|bos|><|user_start|> USER <|user_end|><|assistant_start|> ASSISTANT <|assistant_end|>
BOS, USR_S, USR_E, ASS_S = 32759, 32760, 32761, 32762
user_msg = "What is the boiling point of water in Fahrenheit?"
ids = [BOS, USR_S] + tokenizer.encode(user_msg) + [USR_E, ASS_S]
input_ids = torch.tensor([ids], device=model.device)

with torch.no_grad():
 out = model.generate(input_ids, max_new_tokens=80, do_sample=False)
print(tokenizer.decode(out[0, len(ids):]))
# -> "The boiling point of water is 100°C (212°F). At sea level, ..."

Base-style prompts (just <|bos|> + raw text, no chat wrapper) work better than the chat format for open-ended continuation — see the evaluation table below. For greedy decoding prefer a repetition penalty (repetition_penalty=1.15); without it most prompts loop.

The cascade loader does the following when you call from_pretrained:

1. Reads base_model_name_or_path from config.json and recursively loads the parent model.
2. Instantiates an extended architecture at num_hidden_layers=26 (base 24 + this stage's 2 new layers).
3. Copies the parent's weights into layers 0-23, layers 0-23 KV-embeds, and the base lm_head.
4. Loads delta.safetensors on top - this fills the 2 new layer blocks, the additive lm_head_stage, the warm-initialized wte, and resid_lambdas / x0_lambdas at their new length 26.

Architecture

This stage is trained per paper Section 6.5 (dual-head SFT) and Section 8.8 (elastic depth):

Why dual-head and not a full fine-tune? Freezing the base lm_head and training a zero-init additive specialist lets the stage specialize without touching the base's next-token prediction of general web text. At training step 0 the stage contributes nothing (logits identical to the base); by the final step it has learned chat-specific biases and token-level corrections on top of the frozen base — a controllable, resettable delta. The key empirical claim is that base capability is preserved exactly (Δlogit = 0.0000 across checkpoints) — see the evaluation table below for a live demonstration that base-style continuation remains strong.

Training

All numbers are also in config.json → stage_training for programmatic access.

Validation bpb trajectory

Assistant-only bpb. Warmdown (90 %) drives steady convergence; the acceleration at step 2400–2600 coincides with the learning rate decaying below ~0.15 of peak.

Evaluation (honest)

This is a 1.3 B research preview, not a chat product. Below are the actual outputs we observed on a diverse subjective probe. We report Pass / Partial / Fail verdicts so readers can calibrate expectations.

Interpretation

• Base preservation (4/4) is the primary finding this artifact contributes. Dual-head SFT did not damage the base's knowledge representation, consistent with the paper's §6.5 Δlogit = 0.0000 result measured over 12 training checkpoints.
• Chat-format weakness is a capacity story, not an architecture story. The specialist head (107 M) plus 2 new layers cannot overcome the 1.3 B backbone's limits on arithmetic, multi-step reasoning, or strict formatting. 1.3 B-class chat SFTs in the literature (Phi-1.5, SmolLM-1.7B, TinyLlama-1.1B) exhibit the same profile — emergent reasoning in chat style typically requires 7 B+ (scaling laws).
• Greedy loops are a sampling-strategy issue. Using repetition_penalty=1.15 or nucleus sampling (do_sample=True, top_p=0.9, temperature=0.8) visibly reduces loops in our probe. The README example above uses greedy for minimal surprise; production use should pick different defaults.

Stacking further stages

base_model_name_or_path supports chaining: point a new stage repo's config at this repo and the cascade loader will resolve base → this SFT stage → new stage transparently. Each stage adds its own lm_head_stage and, at load time, the loader folds ancestor stages' lm_head_stage into the effective lm_head_base so all specialists compose additively into the final projection (logits = lm_head_base + Σ lm_head_stage_k). See the paper for the formal account of this construction. Planned sibling stages (math, code) will publish at separate repos and may be stacked on top of this one or attached directly to the base.

Files

Limitations — explicit list

• Research preview at 1.3 B. Chat behavior is bounded by capacity. Do not deploy in production chat surfaces.
• Arithmetic and reasoning are unreliable at this parameter scale regardless of SFT data. GSM8K in training does not close this.
• Greedy decoding loops. Use repetition_penalty >= 1.1 or sampling.
• Strict format constraints are not respected (numbered lists, length limits, yes/no).
• Multi-turn context is fragile beyond 2–3 turns.
• No RLHF / preference tuning / safety tuning. Output may be factually wrong or include fabricated biographical details (we observed this on art-history prompts).
• Assistant-only loss — validation bpb 2.0610 is measured on assistant tokens only; do not compare to the base's full-text train_val_bpb = 0.7209.

Citation

If you use this model, please cite the companion paper and the nanochat toolkit:

@article{jeong2026poe,
 title = {Product of Experts as Scalable Local Learning: Modular Construction at 1.3B Parameters},
 author = {Jeong, Jaepil},
 year = {2026},
 institution = {Cognica, Inc.},
 doi = {10.5281/zenodo.19547653},
 url = {https://doi.org/10.5281/zenodo.19547653}
}

@misc{cognica-poe-stage-chat-2026,
 title = {Cognica-PoE-v1.0-1.3B-stage-chat: Dual-head SFT specialist over a PoE base (research preview)},
 author = {{Cognica, Inc.}},
 year = {2026},
 howpublished = {\url{https://huggingface.co/cognica/Cognica-PoE-v1.0-1.3B-stage-chat}}
}

License

Apache 2.0 — see LICENSE and NOTICE. Same terms as the base model. Training datasets (SmolTalk, MMLU, GSM8K) each carry their own licenses (Apache 2.0, MIT, MIT respectively) and are acknowledged in NOTICE.

Base model

cognica/Cognica-PoE-v1.0-1.3B-base

(6)

Product of Experts (PoE) replaces backprop's global state with local learning, validated at 1.3B across five modularity axes. • 7 items • Updated Apr 22