Cache vs. Database: Has Performance Converged?

Although caches and databases are different animals, databases have always cached data and caches are starting to use disks. Is their performance converging over time?

In-memory caches have long been regarded as one of the fastest infrastructure components around. Yet, it’s been a few years since caching solutions started to look into the realm of flash disks. These initiatives obviously pose an interesting question: If an in-memory cache can rely on flash storage, then why can’t a persistent database also work as a cache?

To quantify the strengths and tradeoffs of each, ScyllaDB joined forces with memcached’s maintainer to compare both across different scenarios.

We’ll explain our motivation for these tests, provide a summary of the tested scenarios and results, then present recommendations for anyone who might be deciding between ScyllaDB and memcached. Along the way, we analyze the architectural differences behind these two solutions and discuss the trade-offs involved in each.

Spoiler: the results reveal that:

• Both memcached and ScyllaDB maximized disks and network bandwidth while being stressed under similar conditions, sustaining similar performance overall.
• While ScyllaDB required data modeling changes to fully saturate the network throughput, memcached required additional IO threads to saturate disk I/O.
• Although ScyllaDB showed better latencies when compared to memcached pipelined requests to disk, memcached latencies were better for individual requests.

There’s also a detailed Gitbook for this project, with a more extensive look at the tests and results and links to the specific configurations you can use to perform the tests yourself.

ScyllaDB would like to acknowledge memcached maintainer Alan Kasindorf (a.k.a. dormando) and Danny Kopping for their contributions to this project, as well as thank them for their support and patience. Bonus: dormando recently joined me for a P99 CONF talk: “Why Databases Cache, but Caches Go to Disk.” You can watch that talk, plus hundreds of others (including a great one by Kopping) at https://www.p99conf.io/

RAM Caching Efficiency

The more items you can fit into RAM, the better your chance of getting cache hits. More cache hits result in significantly faster access than going to disk. Ultimately, that improves latency.

This project began by measuring how many items we could store to each data store. Throughout our tests, the key was between 4 to 12 bytes (key0 .. keyN) for memcached, and 12 bytes for ScyllaDB. The value was fixed to 1,000 bytes.

Memcached

Memcached stored roughly 101 million items until eviction started. It’s memory efficient. Out of memcached’s 114 gigabytes assigned memory, this is approximately 101G worth of values, without considering the key size and other flags:

ScyllaDB

ScyllaDB stored between 60 to 61 million items before evictions started. This is no surprise, given that its protocol requires more data to be stored as part of a write (such as the write timestamp since epoch, row liveness, etc). ScyllaDB also persists data to disk as you go, which means that Bloom filters (and optionally indexes) need to be stored in memory for subsequent disk lookups.

Takeaways

• Memcached stored approximately 65% more in-memory items than ScyllaDB.
• ScyllaDB rows have higher per-item overhead to support a wide-column orientation.
• In ScyllaDB, Bloom filters, index caching and other components are also stored in-memory to support efficient disk lookups, contributing to yet another layer of overhead.

Read-Only In-Memory Workload

The ideal (though unrealistic) workload for a cache is one where all the data fits in RAM so that reads don’t require disk accesses and no evictions or misses occur. Both ScyllaDB and memcached employ LRU (least recently used) logic for freeing up memory: When the system runs under pressure, items get evicted from the LRU’s tail; these are typically the least active items.

Taking evictions and cache misses out of the picture helps measure and set a performance baseline for both data stores. It places the focus on what matters most for these kinds of workloads: read throughput and request latency.

In this test, we first warmed up both stores with the same payload sizes used during the previous test. Then, we initiated reads against their respective ranges for 30 minutes.

Memcached

Memcached achieved an impressive 3 million Gets per second, fully maximizing AWS network interface card (NIC) bandwidth (25 Gbps)!

The parsed results show that p99.999 responses completed below 1millisecond:

stat: cmd_get :
Total Ops: 5503513496
Rate: 3060908/s
=== timer mg ===
1-10us 0 0.000%
10-99us 343504394 6.238%
100-999us 5163057634 93.762%
1-2ms 11500 0.00021%

ScyllaDB

To read more rows in ScyllaDB, we needed to devise a better data model for client requests due to protocol characteristics (in particular, no pipelining). With a clustering key, we could fully maximize ScyllaDB’s cache, resulting in a significant improvement in the number of cached rows. We ingested 5 million partitions, each with 16 clustering keys, for a total of 80 million cached rows.

As a result, the number of records within the cache significantly improved compared to the key-value numbers shown previously.

As dormando correctly pointed out (thanks!), this configuration is significantly different from the previous memcached setup. While the memcached workload always hits an individual key-value pair, a single request in ScyllaDB results in several rows being returned. Notably, the same results could be achieved using memcached by feeding the entire payload as the value under a single key, with the results scaling accordingly.

We explained the reasons for these changes in the detailed write-up. There, we covered characteristics of the Cassandra Query Language (CQL) protocol — for example, the per-item overhead (compared to Memcached) and no support for pipelining — which make wide-partitions more efficient on ScyllaDB than single-key fetches.

With these adjustments, our loaders ran a total of 187,000 read ops/second over 30 minutes. Each operation resulted in 16 rows getting retrieved.

Similarly to memcached, ScyllaDB also maximized the NIC throughput. It served roughly 3 million rows/second solely from in-memory data:

ScyllaDB exposes server-side latency information, which is useful for analyzing latency without the network. During the test, ScyllaDB’s server-side p99 latency remained within 1ms bounds:

The client-side percentiles are, unsurprisingly, higher than the server-side latency with a read P99 of 0.9ms.

Takeaways

• Both memcached and ScyllaDB fully saturated the network; to prevent saturating the maximum network packets per second, memcached relied on request pipelining whereas ScyllaDB was switched to a wide-column orientation.
• ScyllaDB’s cache showed considerable gains following a wide-column schema, able to store more items compared to the previous simple key-value orientation.
• On the protocol level, memcached’s protocol is simpler and lightweight, whereas ScyllaDB’s CQL provides richer features but can be heavier.

Adding Disks to the Picture

Measuring flash storage performance introduces its own set of challenges, which makes it almost impossible to fully characterize a given workload realistically.

For disk-related tests, we decided to measure the most pessimistic situation: Compare both solutions serving data (mostly) from block storage, knowing that:

• The likelihood of realistic workloads doing this is somewhere close to zero.
• Users should expect numbers in between the previous optimistic cache workload and the pessimistic disk-bound workload in practice.

Memcached Extstore

The Extstore wiki page provides extensive detail into the solution’s inner workings. At a high level, it allows memcached to keep its hash table and keys in memory, but store values in external storage.

During our tests, we populated memcached with 1.25 billion items with a value size of 1KB and a keysize of up to 14 bytes:

With Extstore, we stored around 11 times the number of items compared to the previous in-memory workload until evictions started to kick in (as shown in the right-hand panel in the image above). Even though 11X is an already impressive number, the total data stored on flash was only 1.25TB out of the total 3.5TB provided by the AWS instance.

Read-Only Performance

For the actual performance tests, we stressed Extstore against item sizes of 1KB and 8KB. The table below summarizes the results:

ScyllaDB

We populated ScyllaDB with the same number of items as used for memcached. Although ScyllaDB showed higher GET rates than memcached, it did so under slightly higher tail latencies compared to memcached’s non-pipelining workloads. This is summarized below:

Takeaways

• Extstore required considerable tuning to its settings in order to fully saturate flash storage I/O.
• Due to memcached architecture, smaller payloads are unable to fully use the available disk space, providing smaller gains compared to ScyllaDB.
• ScyllaDB rates were overall higher than memcached in a key-value orientation, especially under higher payload sizes. Latencies were better than pipelined requests, but slightly higher than individual GETs in memcached.

Overwrite Workload

Following our previous disk results, we then compared both solutions in a read-mostly workload targeting the same throughput (250K ops/sec). The workload in question is a slight modification of memcached’s “basic” test for Extstore, with 10% random overwrites. It is considered a “semi-worst-case scenario.”

Memcached

Memcached achieved a rate of slightly under 249,000 during the test. Although the write rates remained steady during the duration of the test, we observed that reads fluctuated slightly throughout the run:

We also observed slightly high `extstore_io_queue` metrics despite the lowered read ratios, but latencies still remained low. These results are summarized below:

ScyllaDB

The ScyllaDB test was run using two loaders, each with half of the target rate. Even though ScyllaDB achieved a slightly higher throughput (259.5K), the write latencies were kept low throughout the run and the read latencies were higher (similarly as with memcached):

The table below summarizes the client-side run results across the two loaders:

Takeaways

• Both memcached and ScyllaDB write rates were steady, with reads slightly fluctuating throughout the run.
• ScyllaDB writes still account for the commitlog overhead, which sits in the hot write path.
• ScyllaDB server-side latencies were similar to those observed in memcached results, although client-side latencies were slightly higher.

Read a more detailed analysis in the Gitbook for this project.

Wrapping Up

Both memcached and ScyllaDB managed to maximize the underlying hardware utilization across all tests and keep latencies predictably low. So which one should you pick? The real answer: It depends.

If your existing workload can accommodate a simple key-value model and it benefits from pipelining, then memcached should be more suitable to your needs. On the other hand, if the workload requires support for complex data models, then ScyllaDB is likely a better fit.

Another reason for sticking with memcached: It easily delivers traffic far beyond what a NIC can sustain. In fact, in this Hacker News thread, dormando mentioned that he could scale it up past 55 million read ops/sec for a considerably larger server. Given that, you could make use of smaller and/or cheaper instance types to sustain a similar workload, provided the available memory and disk footprint suffice your workload needs.

A different angle to consider is the data set size. Even though Extstore provides great cost savings by allowing you to store items beyond RAM, there’s a limit to how many keys can fit per GB of memory. Workloads with very small items should observe smaller gains compared to those with larger items. That’s not the case with ScyllaDB, which allows you to store billions of items irrespective of their sizes.

It’s also important to consider whether data persistence is required. If it is, then running ScyllaDB as a replicated distributed cache provides you greater resilience and non-stop operations, with the trade-off being (and as memcached correctly states) that replication halves your effective cache size. Unfortunately, Extstore doesn’t support warm restarts and thus the failure or maintenance of a single node is prone to elevating your cache miss ratios. Whether this is acceptable depends on your application semantics: If a cache miss corresponds to a round trip to the database, then the end-to-end latency will be momentarily higher.

With regard to consistent hashing, memcached clients are responsible for distributing keys across your distributed servers. This may introduce some hiccups, as different client configurations will cause keys to be assigned differently, and some implementations may not be compatible with each other. These details are outlined in Mmemcached’s ConfiguringClient wiki. ScyllaDB takes a different approach: Consistent hashing is done at the server level and propagated to clients when the connection is first established. This ensures that all connected clients always observe the same topology as you scale.

So who won — or who lost? Well, this does not have to be a competition, nor an exhaustive list outlining every single consideration for each solution. Both ScyllaDB and memcached use different approaches to efficiently use the underlying infrastructure. When configured correctly, both have the potential to provide great cost savings.

We were pleased to see ScyllaDB matching the numbers of the industry-recognized memcached. Of course, we had no expectations of our database being “faster.” In fact, as we approach microsecond latencies at scale, the definition of faster becomes quite subjective.