Meta

• License: Apache Software License (Apache 2.0)
• Author: Juncong Moo
• Tags gptq

Classifiers

• Development Status
  • 5 - Production/Stable
• Intended Audience
  • Developers
  • Education
  • Information Technology
  • Science/Research
• License
  • OSI Approved :: Apache Software License
• Operating System
  • MacOS
  • POSIX :: Linux
• Programming Language
  • Python :: 3.6
  • Python :: 3.7
  • Python :: 3.8
  • Python :: 3.9
  • Python :: 3.10
  • Python :: 3.11
• Topic
  • Scientific/Engineering :: Mathematics
  • Software Development :: Libraries

🔮 GPTQ - Accurate Post-Training Compression for Generative Pretrained Transformers

This repo is a extended and polished version of the original code for the paper GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers.

🔥 SOTA on LLM PTQ

• An efficient implementation of the GPTQ algorithm
• 2/3/4/8-bit quantized matrix full-precision vector product CUDA kernel
• Bug fix for old consumer-grade GPU

👁 Image

📥 Installation

pipinstallgptq

🛟 Install PyTorch

gptq requires PyTorch and GPU, and installing PyTorch with CUDA is tricky. To install PyTorch correctly, the following steps are recommended:

• run nvcc --version to get the version. For example, the following result means we have cuda compiler version 116

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0

• run pip install light-the-torch to install ltt
• run ltt install --pytorch-computation-backend=cu116 torch torchvision torchaudio to install the torch suite. Please replace the 116 according to your environment!

TODO

• GPTQ with CNN

---

Algorithm credits go to IST Austria Distributed Algorithms and Systems Lab

Verified details

These details have been verified by PyPI

Maintainers

👁 Avatar for juncongmoo from gravatar.com
juncongmoo

Unverified details

These details have not been verified by PyPI

• Download URL: gptq-0.0.3.tar.gz
• Upload date: Mar 23, 2023
• Size: 21.4 kB
• Tags: Source
• Uploaded using Trusted Publishing? No
• Uploaded via: twine/4.0.2 CPython/3.8.16

Hashes for gptq-0.0.3.tar.gz

Algorithm	Hash digest
SHA256	05121652e59fd5cc9c6cf9530bb999bb4d843fdbbe81ee532e06c6f8023b812f
MD5	e36064eeaae8f9c0edb7864648f58317
BLAKE2b-256	25062e8e087ec1572fca200c591442ed92c4df7ac8a854ecdf32a7b8065ce14d

See more details on using hashes here.