In-memory cache design generally faces the choice of introducing internal fragmentation (allocating more than used) or external fragmentation (taking more memory than allocated). The former reduces effective capacity, while the latter could trigger OOM. #CachingWisdomOfTheDay

Does cache have to keep data in DRAM to perform? The answer turns out to be highly workload/design dependent. The performance gap between DRAM and SSD (or PMEM) continues to be much bigger for random writes than reads. For many reads, SSD is fast enough. #CachingWisdomOfTheDay

It’s easier to design data structures for cache than database, meaning very loose consistency and relatively forgiving availability requirements. But even so, perfect generic data designs don’t exist. It’s often more valuable to specialize by workloads. #CachingWisdomOfTheDay

What can you do when you can consistently log every command except the values to (a slice of) you cache backend? A lot. Hot keys, heat map, working set size and miss rate estimates… In fact, most of the cache config parameters can be correctly inferred. #CachingWisdomOfTheDay

Threading model is especially important for secure cache access. While modern hardware has made encryption much faster, TLS frames add memory & CPU overhead. Worse still, TLS handshake is 10x more expensive than TCP– connect is a serious threat to SLO. #CachingWisdomOfTheDay

Cache writes are not always more expensive than reads. Simple read/write can achieve similar RPS because RPC costs dominate. There are some “tipping points” wrt RPC- e.g. you can expect sudden dip on RPS at MTU, socket buffer, and/or request buffer sizes. #CachingWisdomOfTheDay

Always have a TTL for anything put in cache. To list a few reasons: - it provides a well defined bound on data inconsistency; - it gives cache backend a strong hint on how to retain useful data for the right duration; - it helps you stay GDPR-compliant. #CachingWisdomOfTheDay

In cache with data structures, writes tend to be incremental. It’s important to keep memory op proportional to update instead of object size. A DIMM carries ~40GB/s. If updates against 1MB objects involve memcpy whole objects, throughput will be <40K RPS. #CachingWisdomOfTheDay

Accessing a large in memory dataset is unlikely to be CPU cache friendly, even if it has a fairly skewed popularity distribution. What could / should be kept in CPU caches? IO buffers. How? CPU pinning, flow steering, and keeping storage threads away. #CachingWisdomOfTheDay

You're probably familiar with read vs. write. But also consider: - idempotent vs. non-idempotent write: the former can be safely retried, not the latter; - regular vs. privileged command: e.g. you probably don't want people to run FLUSH_ALL willy-nilly. #CachingWisdomOfTheDay

Friends don't let friends enable dynamic scripting in cache. All it takes is one bug or a poorly implemented algorithm to completely kill server performance. Scripting is useful but should be treated as other server code- tested, benchmarked, deployed. #CachingWisdomOfTheDay

Almost no where else in the infrastructure software would the following statement be more true than it is for caching: "Service delayed is service denied." Make it fast. Make it predictably fast. Make it consistently fast. #CachingWisdomOfTheDay

We know our documentation has a long way to go, so we are spending September, heads-down, to write about the design of Pelikan. If you want to known about anything in particular, please tell us in the reply. In October, we'll return with a daily tweet series on Rust + Pelikan.

🎙️E49 of the Open Source Startup Podcast is LIVE🎙️ Ft. Daniela Miao, Cofounder of caching platform @MomentoHQ & Yao Yue 岳峣 from the Twitter team for Pelikan Cache ✨Check it out✨ anchor.fm/ossstartuppodc…

Yep, we agree. betterprogramming.pub/software-compo… Corollary: If you do not know what *exactly* Pelikan Cache is, don't worry. It's by design.

I am still amazed at how easy it was to run Pelikan Cache on Ampere Altra-based Tau T2A VMs on GoogleCloud. Ali Zhairati was able to make it work instantly and make it fly in the matter of hours! 🙇‍♂️

If you are attending QCon SF, check out Juncheng (Juncheng Yang)'s talk on Ubiquitous Caching for the "Building Modern Backend" track at 1:40pm, where he discusses hardware trend, emerging workloads, and running large in-process caching using Segcache. qconsf.com/presentation/o…

I talked about S3-FIFO at #SOSP today. S3-FIFO is a FIFO-based eviction algorithm that is simple, scalable, and efficient. For example, it can reduce the miss ratio of your LRU caches by up to 72% and improve throughput by 6x at 16 threads. S3-FIFO is being tried out at a few