This model was presented in the paper SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion.

🚀 Features:

• 🏷️ DocTags for Efficient Tokenization – Introduces DocTags an efficient and minimal representation for documents that is fully compatible with DoclingDocuments.
• 🔍 OCR (Optical Character Recognition) – Extracts text accurately from images.
• 📐 Layout and Localization – Preserves document structure and document element bounding boxes.
• 💻 Code Recognition – Detects and formats code blocks including identation.
• 🔢 Formula Recognition – Identifies and processes mathematical expressions.
• 📊 Chart Recognition – Extracts and interprets chart data.
• 📑 Table Recognition – Supports column and row headers for structured table extraction.
• 🖼️ Figure Classification – Differentiates figures and graphical elements.
• 📝 Caption Correspondence – Links captions to relevant images and figures.
• 📜 List Grouping – Organizes and structures list elements correctly.
• 📄 Full-Page Conversion – Processes entire pages for comprehensive document conversion including all page elements (code, equations, tables, charts etc.)
• 🔲 OCR with Bounding Boxes – OCR regions using a bounding box.
• 📂 General Document Processing – Trained for both scientific and non-scientific documents.
• 🔄 Seamless Docling Integration – Import into Docling and export in multiple formats.
• 💨 Fast inference using VLLM – Avg of 0.35 secs per page on A100 GPU.

🚧 Coming soon!

• 📊 Better chart recognition 🛠️
• 📚 One shot multi-page inference ⏱️
• 🧪 Chemical Recognition
• 📙 Datasets

⌨️ Get started (code examples)

You can use transformers, vllm, or onnx to perform inference, and Docling to convert results to variety of output formats (md, html, etc.):

💻 Local inference on Apple Silicon with MLX: see here

DocTags

👁 Image description
DocTags create a clear and structured system of tags and rules that separate text from the document's structure. This makes things easier for Image-to-Sequence models by reducing confusion. On the other hand, converting directly to formats like HTML or Markdown can be messy—it often loses details, doesn’t clearly show the document’s layout, and increases the number of tokens, making processing less efficient. DocTags are integrated with Docling, which allows export to HTML, Markdown, and JSON. These exports can be offloaded to the CPU, reducing token generation overhead and improving efficiency.

Supported Instructions

Description	Instruction	Comment
Full conversion	Convert this page to docling.	DocTags represetation
Chart	Convert chart to table.	(e.g., <chart>)
Formula	Convert formula to LaTeX.	(e.g., <formula>)
Code	Convert code to text.	(e.g., <code>)
Table	Convert table to OTSL.	(e.g., <otsl>) OTSL: Lysak et al., 2023
Actions and Pipelines	OCR the text in a specific location: <loc_155><loc_233><loc_206><loc_237>
	Identify element at: <loc_247><loc_482><10c_252><loc_486>
	Find all 'text' elements on the page, retrieve all section headers.
	Detect footer elements on the page.

📊 Datasets

• SynthCodeNet
• SynthFormulaNet
• SynthChartNet
• DoclingMatix

Model Summary

• Developed by: Docling Team, IBM Research
• Model type: Multi-modal model (image+text)
• Language(s) (NLP): English
• License: Apache 2.0
• Architecture: Based on Idefics3 (see technical summary)
• Finetuned from model: Based on SmolVLM-256M-Instruct

Repository: Docling

Paper: arXiv

Project Page: Hugging Face

Citation:

@misc{nassar2025smoldoclingultracompactvisionlanguagemodel,
 title={SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion}, 
 author={Ahmed Nassar and Andres Marafioti and Matteo Omenetti and Maksym Lysak and Nikolaos Livathinos and Christoph Auer and Lucas Morin and Rafael Teixeira de Lima and Yusik Kim and A. Said Gurbuz and Michele Dolfi and Miquel Farré and Peter W. J. Staar},
 year={2025},
 eprint={2503.11576},
 archivePrefix={arXiv},
 primaryClass={cs.CV},
 url={https://arxiv.org/abs/2503.11576}, 
}

Demo: HF Space

Base model

HuggingFaceTB/SmolLM2-135M

HuggingFaceTB/SmolLM2-135M-Instruct

HuggingFaceTB/SmolVLM-256M-Instruct

(23)

3 items • Updated Apr 7, 2025 • 9