👁 Image

aibarito-ua

Jul 26, 2023

Hello!
I am trying to run this model on one A100, but the speed is quite slow - 2 tokens/sec. Does anybody know how to make it faster?
I have tried 8-bit-mode and it is allocating twice less gpu memory, but the speed is not increasing.

👁 Image

Limerobot

upstage org Aug 2, 2023

https://www.google.com/search?q=llama+inference+speed+optimization&source=lnt&tbs=qdr:m&sa=X&ved=2ahUKEwjTw-3X5LyAAxULG4gKHa23BrcQpwV6BAgDEA8&biw=2168&bih=1065&dpr=2.36
I hope this helps you.