Computer Science > Computer Vision and Pattern Recognition
arXiv:2303.15446 (cs)
[Submitted on 27 Mar 2023 (v1), last revised 25 Jul 2023 (this version, v2)]
Title:SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications
Authors:Abdelrahman Shaker, Muhammad Maaz, Hanoona Rasheed, Salman Khan, Ming-Hsuan Yang, Fahad Shahbaz Khan
View a PDF of the paper titled SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications, by Abdelrahman Shaker and 5 other authors
View PDF
Abstract:Self-attention has become a defacto choice for capturing global context in various vision applications. However, its quadratic computational complexity with respect to image resolution limits its use in real-time applications, especially for deployment on resource-constrained mobile devices. Although hybrid approaches have been proposed to combine the advantages of convolutions and self-attention for a better speed-accuracy trade-off, the expensive matrix multiplication operations in self-attention remain a bottleneck. In this work, we introduce a novel efficient additive attention mechanism that effectively replaces the quadratic matrix multiplication operations with linear element-wise multiplications. Our design shows that the key-value interaction can be replaced with a linear layer without sacrificing any accuracy. Unlike previous state-of-the-art methods, our efficient formulation of self-attention enables its usage at all stages of the network. Using our proposed efficient additive attention, we build a series of models called "SwiftFormer" which achieves state-of-the-art performance in terms of both accuracy and mobile inference speed. Our small variant achieves 78.5% top-1 ImageNet-1K accuracy with only 0.8 ms latency on iPhone 14, which is more accurate and 2x faster compared to MobileViT-v2. Code: this https URL
| Comments: | Accepted at ICCV 2023 |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2303.15446 [cs.CV] |
| (or arXiv:2303.15446v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2303.15446
arXiv-issued DOI via DataCite
|
Submission history
From: Abdelrahman Shaker [view email][v1] Mon, 27 Mar 2023 17:59:58 UTC (4,216 KB)
[v2] Tue, 25 Jul 2023 19:56:00 UTC (4,216 KB)
Full-text links:
Access Paper:
- View PDF
- TeX Source
View a PDF of the paper titled SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications, by Abdelrahman Shaker and 5 other authors
References & Citations
Loading...
BibTeX formatted citation
Data provided by:
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.