VOOZH about

URL: https://huggingface.co/changminbark/gemma-3-1b-it-ttt-longalpaca-full

⇱ changminbark/gemma-3-1b-it-ttt-longalpaca-full · Hugging Face

Model Card for Model ID

This Gemma3-1B-it model had its base model frozen and then had some of its MLP layers modified (global attention layers) to support In-Place Test Time Training. This was trained on 2 epochs of 12k samples of the LongAlpaca-12k dataset. The code can be found in this repo.

Model Details

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

  • Developed by: Chang Min Bark and Hung Ngo
  • Funded by [optional]:
  • Shared by [optional]:
  • Model type: Dense
  • Language(s) (NLP): English
  • License: Apache-2.0
  • Finetuned from model [optional]: google/gemma-3-1b-it

Model Sources [optional]

Uses

Direct Use

This was developed for the final project at AI with Neural Networks class at Bucknell University.

Downstream Use [optional]

[More Information Needed]

Out-of-Scope Use

This model cannot be used for any use cases outlined for google/gemma-3-1b-it.

Bias, Risks, and Limitations

This model carries all of the same biases, risks, and limitations carried by google/gemma-3-1b-it.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Check the GitHub repo for more information.

Training Details

Training Data

This was trained using LongAlpaca.

Training Procedure

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

  • Training regime:
    Hyperparameter TinyStories LongAlpaca Flag
    max_length 1024 4096 --max-length
    epochs 1.0 2.0 --epochs
    batch_size 8 1 --batch-size
    grad_accum 8 16 --grad-accum
    effective batch 64 16 —
    lr 1e-4 5e-5 --lr
    weight_decay 0.1 0.1 --weight-decay
    warmup_steps 500 100 --warmup-steps
    max_grad_norm 1.0 1.0 --max-grad-norm

Shared optimizer / schedule (not per-dataset): AdamW with (β1, β2) = (0.9, 0.95), cosine LR decay, bf16 (fp16 fallback on non-bf16 GPUs), gradient checkpointing off.

TTT-module knobs (defaults match the In-Place TTT paper):

Hyperparameter Default Flag
ttt_layers [0, 6, 12, 18, 24] --ttt-layers
ttt_chunk 2048 --ttt-chunk
ttt_lr (η) 0.3 --ttt-lr
ttt_proj on --no-ttt-proj
ttt_target hidden_states --ttt-target

Speeds, Sizes, Times [optional]

Evaluation

Testing Data, Factors & Metrics

Testing Data

Tested using synthetic NVIDIA RULER.

Factors

  • Context length

Metrics

  • Accuracy

Results

Summary

The vanilla base google/gemma-3-1b-it performed better than the In-Place TTT augmented model.

Model Examination [optional]

[More Information Needed]

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

  • Hardware Type: [More Information Needed]
  • Hours used: [More Information Needed]
  • Cloud Provider: [More Information Needed]
  • Compute Region: [More Information Needed]
  • Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

Based on google/gemma-3-1b-it.

Compute Infrastructure

Run using Google Colab.

Hardware

Run using NVIDIA A100 GPU in Google Colab.

Software

Run using Google Colab.

Citation [optional]

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]

Downloads last month
11
Safetensors
Model size
1B params
Tensor type
BF16
·

Model tree for changminbark/gemma-3-1b-it-ttt-longalpaca-full

Finetuned
(558)
this model

Dataset used to train changminbark/gemma-3-1b-it-ttt-longalpaca-full

Paper for changminbark/gemma-3-1b-it-ttt-longalpaca-full