Paper • 2504.21798 • Published • 15
Qwen2.5-Coder-7B-SWEsmith-400
A full fine-tune of Qwen/Qwen2.5-Coder-7B-Instruct on 400 SWE-agent trajectories from the SWE-smith project, intended for use as a software-engineering agent (SWE-agent / SWE-bench style tasks).
⚠️ Experimental / small-scale. This was trained on only 400 trajectories as a workflow validation run, not a production model. Treat it as a reproducibility / sanity baseline.
Training data
- Source:
SWE-bench/SWE-smith-trajectories,xmlsplit (the format SWE-agent-LM was trained on). - Subset: first 400 of the 26,076 trajectories (multi-turn agent rollouts, avg ~67 messages/traj).
- Format: OpenAI-style
messages, supervised on assistant turns only (train_on_input=False).
Method
Full-parameter SFT with torchtune
(full_finetune_distributed), using SWE-smith's default 7B config.
| Hyperparameter | Value |
|---|---|
| Base model | Qwen2.5-Coder-7B-Instruct |
| Optimizer | AdamW (fused), wd=0.01 |
| LR / schedule | 1e-4, cosine, 5 warmup steps |
| Epochs | 3 |
| Max seq len | 32768 |
| Precision | bf16 |
| Per-device batch | 1, grad-accum 4 |
| Hardware | 7× A100 80GB (FSDP) → effective batch 28 |
| Total optimizer steps | 42 |
| Activation checkpointing | on |
Training loss: 0.41 → 0.19 (ep0) → 0.11 (ep1) → 0.085 (ep2). This repo is the epoch 2 checkpoint.
Intended use & limitations
Designed to be served (vLLM / SGLang) behind SWE-agent for automated bug-fixing on repository-level tasks. Given the tiny training set, expect limited generalization; it has not been evaluated on SWE-bench. Inherits the base model's capabilities and biases.
Citation
Built with SWE-smith (Yang et al., NeurIPS 2025). Base model license: Apache-2.0.
- Downloads last month
- 38
Safetensors
Model size
8B params
Tensor type
BF16
·