Configuration Parsing Warning:In UNKNOWN_FILENAME: "auto_map.AutoTokenizer" must be a string

Cognica-PoE-v1.0-1.3B-stage-math

Paper: Product of Experts as Scalable Local Learning: Modular Construction at 1.3B Parameters (Jeong, 2026)

Not a production math model. This release is empirical validation of PoE's post-hoc specialist construction at 4-layer depth — paper §6.5 extended.

A 164 M-parameter math-domain SFT specialist stage trained directly on the frozen Cognica-PoE-v1.0-1.3B-base. The stage is a sibling of cognica/Cognica-PoE-v1.0-1.3B-stage-chat — both branch from the same base, not from each other. Use the cascade loader with this repo (or compose both stage heads additively in a custom inference script; see the paper).

At 1.3 B base + 164 M specialist, arithmetic answers are still often wrong; the specialist's contribution is in format and reasoning chain, not exact numeric computation. Emergent arithmetic in the literature requires 7 B+ — that scaling ceiling cannot be moved by a specialist head.

What this artifact does demonstrate:

1. A deeper (4-layer) specialist trains cleanly. Val bpb converges monotonically from 2.78 → 2.41 across 2 267 steps with no divergence. One transient blip at step 1 200 (2.692 → 2.716) self-corrects by step 1 400.
2. Dataset composition for math. Mixing GSM8K main/train (×20 epochs) with MathInstruct (×4 epochs) gives a balanced reasoning-style corpus at ~297 M tokens total.
3. Additive composition at inference. The new cascade loader (v1.0.1+) folds ancestor stages' lm_head_stage into the effective lm_head_base when chaining, so stage-chat → stage-math composition produces correct summed logits.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "cognica/Cognica-PoE-v1.0-1.3B-stage-math"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
 model_id,
 trust_remote_code=True,
 torch_dtype=torch.bfloat16,
 device_map="auto",
)
model.eval()

# Chat-format prompt (the SFT data was formatted with SmolTalk special tokens):
BOS, USR_S, USR_E, ASS_S = 32759, 32760, 32761, 32762
q = "Sarah has 15 candies. She gives 3 to each of her 4 friends. How many candies does she have left?"
ids = [BOS, USR_S] + tokenizer.encode(q) + [USR_E, ASS_S]
input_ids = torch.tensor([ids], device=model.device)

with torch.no_grad():
 out = model.generate(
 input_ids, max_new_tokens=120, do_sample=False,
 repetition_penalty=1.15, pad_token_id=BOS,
 )
print(tokenizer.decode(out[0, len(ids):]))

repetition_penalty=1.15 or sampling (do_sample=True, temperature=0.7, top_p=0.9) is recommended — greedy decoding loops on most prompts at this model scale.

Architecture

Training

Validation bpb trajectory

Stacking further stages

base_model_name_or_path supports chaining. Point a new stage repo's config at this repo and the cascade loader will resolve base → stage-math → new stage transparently. The loader folds each ancestor's lm_head_stage into the effective lm_head_base at load time, so all specialist heads compose additively into the final projection (logits = lm_head_base + Σ lm_head_stage_k). See the paper for the formal account of this construction.

Files

Limitations — explicit list

• Research preview at 1.3 B. Arithmetic answers are unreliable at this parameter scale. GSM8K in training does not close this.
• The specialist learns format, not exact computation. Expect chain-of-thought style output with wrong final numbers.
• Greedy decoding loops. Use repetition_penalty >= 1.1 or sampling.
• No RLHF / preference tuning / safety tuning.
• Assistant-only loss — validation bpb is measured on assistant tokens only; do not compare to the base's full-text train_val_bpb = 0.7209.

Citation

@article{jeong2026poe,
 title = {Product of Experts as Scalable Local Learning: Modular Construction at 1.3B Parameters},
 author = {Jeong, Jaepil},
 year = {2026},
 institution = {Cognica, Inc.},
 doi = {10.5281/zenodo.19547653},
 url = {https://doi.org/10.5281/zenodo.19547653}
}

@misc{cognica-poe-stage-math-2026,
 title = {Cognica-PoE-v1.0-1.3B-stage-math: Math-domain dual-head specialist (4-layer) over a PoE base (research preview)},
 author = {{Cognica, Inc.}},
 year = {2026},
 howpublished = {\url{https://huggingface.co/cognica/Cognica-PoE-v1.0-1.3B-stage-math}}
}

License

Apache 2.0 — see LICENSE and NOTICE. Same terms as the base model. Training datasets (GSM8K, MathInstruct) each carry their own licenses and are acknowledged in NOTICE.

Base model

cognica/Cognica-PoE-v1.0-1.3B-base

(6)

Product of Experts (PoE) replaces backprop's global state with local learning, validated at 1.3B across five modularity axes. • 7 items • Updated Apr 22