Benchmarking Falcon-40B

We benchmark the performance of Falcon-40B-Instruct in this article from latency, cost, and requests per second perspective. This will help us evaluate if it can be a good choice based on the business requirements. Please note that we don't cover the qualitative performance in this article - there are different methods to compare LLMs which can be found here.

Model: Falcon-40B-Instruct

In this blog, we have benchmarked the Falcon-40B-Instruct model from tiiuae. Falcon-40B-Instruct is a 40B parameters causal decoder-only model built by TII based on Falcon-40B and finetuned on a mixture of Baize. It is made available under the Apache 2.0 license.

‍

Metrics to Benchmark

1. Requests per second. (RPS): Requests per second that the model is handling. With higher RPS, the latency usually goes up.
2. Latency: How much time is taken to complete an inference request?
3. Economics: What are the costs associated with deploying an LLM?

Use cases & Deployment Modes Benchmarked

The key factors across which we benchmarked are:

GPU Type:

1. 4 x A100 40GB GPU

Prompt Length:

1. 1500 Input tokens, 100 output tokens (Similar to Retrieval Augmented Generation use cases)
2. 50 Input tokens, 500 output tokens (Generation Heavy use cases)

Benchmarking Setup

For benchmarking, we have used locust, an open-source load-testing tool. Locust works by creating users/workers to send requests in parallel. At the beginning of each test, we can set the Number of Users and Spawn Rate. Here the Number of Users signify the Maximum number of users that can spawn/run concurrently, whereas the Spawn Rate signifies how many users will be spawned per second.

In each benchmarking test for a deployment config, we started from 1 user and kept increasing the Number of Users gradually till we saw a steady increase in the RPS. During the test, we also plotted the response times (in ms) and total requests per second.

In each of the 2 deployment configurations, we have used the huggingface text-generation-inference model server having version=0.9.4. The following are the parameters passed to the text-generation-inference image for different model configurations:

‍

‍

Benchmarking Results Summary

Latency, RPS, and Cost

We calculate the best latency based on sending only one request at a time. To increase throughput, we send requests parallelly to the LLM. The max throughput is the case when the model is able to process the input requests without significant deterioration in latency.

Tokens Per Second

LLMs process input tokens and generation differently - hence we have calculated the input tokens and output tokens processing rate differently.

Detailed Results

4 x A100 40GB GPU (1500 input + 100 output tokens)

We can observe in the above graphs that the Best Response Time (at 1 user) is 4.6 seconds. We can increase the number of users to throw more traffic at the model - we can see the throughput increasing till 2.0 RPS without a significant drop in latency. Beyond 2.0 RPS, the latency increases drastically which means requests are being queued up.

4 x A100 40GB GPU (50 input + 500 output tokens)

We can observe in the above graphs that the Best Response Time (at 1 user) is 20 seconds. We can increase the number of users to throw more traffic at the model - we can see the throughput increasing till 2.5 RPS without a significant drop in latency. Beyond 2.5 RPS, the latency increases drastically which means requests are being queued up.

Hopefully, this will be useful for you to decide if Falcon-40B-Instruct will suit your use case and the costs you can expect to incur while hosting Falcon-40B-Instruct.