knowledge-distillation

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

sparsity pruning quantization knowledge-distillation auto-tuning int8 low-precision quantization-aware-training post-training-quantization awq int4 large-language-models gptq smoothquant sparsegpt fp4 mxformat

Updated
Python

👁 EasyNLP

alibaba / EasyNLP

Star

EasyNLP: A Comprehensive and Easy-to-use NLP Toolkit

nlp machine-learning deep-learning text-classification transformers pytorch transfer-learning pretrained-models knowledge-distillation bert text-to-image-synthesis fewshot-learning text-image-retrieval knowledge-pretraining

Updated
Python

haitongli / knowledge-distillation-pytorch

Star

A PyTorch implementation for exploring deep and shallow knowledge distillation (KD) experiments with flexibility

deep-neural-networks computer-vision pytorch knowledge-distillation cifar10 dark-knowledge model-compression

Updated
Python

horseee / Awesome-Efficient-LLM

Star

A curated list for Efficient Large Language Models

compression language-model knowledge-distillation model-quantization pruning-algorithms llm llm-compression efficient-llm

Updated
Python

robert-mcdermott / ai-knowledge-graph

Star

AI Powered Knowledge Graph Generator

visualization artificial-intelligence networkx knowledge-graph knowledge-distillation pyvis llm

Updated
Python

microsoft / Cream

Star

This is a collection of our NAS and Vision Transformer work.

efficiency nas knowledge-distillation rpe automl vision-transformer vit-compression

Updated
Python

AberHu / Knowledge-Distillation-Zoo

Star

Pytorch implementation of various Knowledge Distillation (KD) methods.

knowledge-distillation teacher-student knowledge-transfer model-compression distillation kd kd-methods

Updated
Python

open-mmlab / mmrazor

Star

OpenMMLab Model Compression Toolbox and Benchmark.

detection pytorch classification segmentation pruning darts quantization nas knowledge-distillation spos autoslim

Updated
Python

👁 torchdistill

yoshitomo-matsubara / torchdistill

Sponsor

Star

A coding-free framework built on PyTorch for reproducible deep learning studies. PyTorch Ecosystem. 🏆26 knowledge distillation methods presented at TPAMI, CVPR, ICLR, ECCV, NeurIPS, ICCV, AAAI, etc are implemented so far. 🎁 Trained models, training logs and configurations are available for ensuring the reproducibiliy and benchmark.

nlp natural-language-processing text-classification glue pytorch transformer imagenet image-classification coco object-detection semantic-segmentation knowledge-distillation cifar10 pascal-voc cifar100 google-colab colab-notebook amazon-sagemaker-lab pytorch-ecosystem

Updated
Python

szagoruyko / attention-transfer

Star

Improving Convolutional Networks via Attention Transfer (ICLR 2017)

deep-learning pytorch attention knowledge-distillation

Updated
Jupyter Notebook

lxztju / pytorch_classification

Star

利用pytorch实现图像分类的一个完整的代码，训练，预测，TTA，模型融合，模型部署，cnn提取特征，svm或者随机森林等进行分类，模型蒸馏，一个完整的代码

flask deployment random-forest svm cnn pytorch image-classification densenet resnet knn knowledge-distillation resnext label-smoothing

Updated
Jupyter Notebook

microsoft / NeuronBlocks

Star

NLP DNN Toolkit - Building Your NLP DNN Models Like Playing Lego

natural-language-processing deep-learning text-classification dnn pytorch artificial-intelligence question-answering knowledge-distillation sequence-labeling text-matching qna model-compression

Updated
Python

huawei-noah / Efficient-Computing

Star

Efficient computing methods developed by Huawei Noah's Ark Lab

pruning quantization knowledge-distillation model-compression self-supervised binary-neural-networks

Updated
Jupyter Notebook

Tebmer / Awesome-Knowledge-Distillation-of-LLMs

Star

This repository collects papers for "A Survey on Knowledge Distillation of Large Language Models". We break down KD into Knowledge Elicitation and Distillation Algorithms, and explore the Skill & Vertical Distillation of LLMs.

compression feedback survey alignment self-training multi-modal knowledge-distillation data-augmentation kd data-synthesis self-distillation instruction-following llm large-language-model supervised-finetuning

Updated

MingSun-Tse / Efficient-Deep-Learning

Star

Collection of recent methods on (deep) neural network compression and acceleration.

deep-neural-networks deep-learning knowledge-distillation model-compression network-pruning efficient-deep-learning

Updated

Improve this page

Add a description, image, and links to the knowledge-distillation topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the knowledge-distillation topic, visit your repo's landing page and select "manage topics."

Learn more

URL: https://github.com/topics/knowledge-distillation