<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" version="2.0">
  <!-- Source: https://gomomento.com/feed -->
  <channel>
    <title>Momento</title>
    <atom:link href="https://siftrss.com/f/0BgwYvPmQz" rel="self" type="application/rss+xml"/>
    <link>https://siftrss.com/f/0BgwYvPmQz</link>
    <description>An enterprise-ready serverless platform for caching and pub/sub</description>
    <lastBuildDate>Wed, 05 Aug 2026 09:00:00 GMT</lastBuildDate>
    <language>en</language>
    <sy:updatePeriod>hourly</sy:updatePeriod>
    <sy:updateFrequency>1</sy:updateFrequency>
    <image>
      <url>https://www.gomomento.com/wp-content/uploads/2024/06/cropped-favicon-green-32x32.png</url>
      <title>Momento</title>
      <link>https://www.gomomento.com/</link>
      <width>32</width>
      <height>32</height>
    </image>
    <item>
      <title>Consistency compounds: Valkey's journey to 200 Gbps</title>
      <link>https://www.gomomento.com/blog/consistency-compounds-valkeys-journey-to-200-gbps/</link>
      <dc:creator><![CDATA[Khawaja Shams]]></dc:creator>
      <pubDate>Thu, 23 Jul 2026 16:00:00 GMT</pubDate>
      <category><![CDATA[Caching]]></category>
      <category><![CDATA[Valkey]]></category>
      <guid isPermaLink="true">https://www.gomomento.com/blog/consistency-compounds-valkeys-journey-to-200-gbps/</guid>
      <description><![CDATA[<p>Across four releases, I/O-threading changes removed a serial copy bottleneck, cut p99 latency, and brought large GETs to line rate on our 200 Gbps test rig.</p>]]></description>
      <content:encoded><![CDATA[<p><img src="https://www.gomomento.com/assets/content/blog/consistency-compounds-valkeys-journey-to-200-gbps/read-bandwidth.svg" alt="Valkey GET bandwidth by value size across versions 7.2, 8.1, 9.0, and 9.1, with the 200 Gbps NIC limit marked"></p>
<p>If you came here looking for a post from me on consistency models, I have to disappoint you. Today, I want to talk about the value of consistency in life. In life, effort is additive, but consistency is multiplicative.</p>
<p>Over the last few years, I have had the honor of watching the Valkey project blossom from an idea into an inspirational, community-driven effort with consistent improvements in each release. These improvements range from memory efficiency to availability at scale to substantial performance gains. These small improvements compound.</p>
<p>This compounding effect and the power of a driven community is perhaps best illustrated by the journey of the I/O-threading architecture and its impact on large objects from 1 MB to 64 MB in Valkey. Larger items are becoming increasingly important for inference KV caches, 4K video streaming, and other enticing use cases. At the very least, the impact of large objects should not be overlooked, as they can <a href="https://www.gomomento.com/blog/large-objects-in-valkey-9-0/">impact everyone else’s latency</a>; this post measures how fast the large objects themselves go.</p>
<p>Buckle up, because it’s about to get spicy.</p>
<h2 id="a-brief-history-of-io-threads">A brief history of I/O threads</h2>
<p>Valkey 7.2 inherited an I/O-thread model from a software stack doing its best to stay single-threaded. The main thread and I/O threads worked in coordinated phases, separated by synchronization barriers, and the <a href="https://github.com/valkey-io/valkey/blob/7.2/valkey.conf">7.2 config file</a> is candid about the result: “Usually threading reads doesn’t help much.”</p>
<p>Valkey 8.0 delivered a fundamental rearchitecture of I/O threads, <a href="https://valkey.io/blog/unlock-one-million-rps/">tripling throughput to over a million requests per second</a>. This idea was not new. In August 2023, seven months before the fork, Dan Touitou filed <a href="https://github.com/redis/redis/issues/12489">redis#12489</a>, laying out exactly this design in detail, benchmarks included.</p>
<blockquote>
<p>Redis let #12489 sit. The issue is still open in the tracker today, unassigned and without a milestone.</p>
</blockquote>
<p>Within days of the fork, the Valkey community copied the proposal verbatim into <a href="https://github.com/valkey-io/valkey/issues/22">issue #22</a>, greeted it as “a true gem,” and shipped it. The new architecture enabled continuously running I/O threads connected by queues, so reads, parses, and writes proceed on separate cores while the main thread executes commands. Valkey 8.1 delivered TLS handshake offload to I/O threads in <a href="https://github.com/valkey-io/valkey/pull/1338">#1338</a>.</p>
<p>Redis shipped a <a href="https://redis.io/blog/redis-8-0-m03-is-out-even-more-performance-new-features/">strikingly similar asynchronous I/O-threading model</a> in Redis 8 in May 2025, a year after the fork and 21 months after the design landed in its own tracker. Around the same time, we put <a href="https://www.gomomento.com/blog/valkey-turns-one-how-the-community-fork-left-redis-in-the-dust/">Valkey 8.1 and Redis 8.0 head to head</a> on small objects. Valkey 8.1 outran Redis 8.0 by 37% on writes and 16% on reads.</p>
<p>Valkey 9.0 brought <a href="https://github.com/valkey-io/valkey/pull/2078">reply copy avoidance</a>, changing the performance of large items entirely. Before this change, the main thread copied the entire object into a connection reply buffer before moving to the next command. While a large item is being copied, the entire pipeline stalls. Small objects are not serviced until the copy finishes.</p>
<p>Valkey 9.0 instead passes a reference to the I/O threads and keeps the object alive with a reference count. The I/O worker for that connection hands the object’s memory directly to <code>writev()</code>. This shortens the handoff to the I/O threads and gets the main thread back to handling requests.</p>
<p>Valkey 9.1 then redesigned communication between the main thread and I/O threads around <a href="https://github.com/valkey-io/valkey/pull/3324">lock-free queues</a>, credited in the release notes with an 8-17% throughput gain.</p>
<p>Based on these changes, we expected 8.0 to lift reads and writes for smaller items but still struggle to fill the network link for larger items. We expected 9.0 to bring GETs to line rate and 9.1 to improve writes.</p>
<p>This is what open-source competition buys everyone, including teams that never leave Redis. A performance design that sat for seven months as an unassigned issue became table stakes for both projects within two years of being filed.</p>
<h2 id="show-me-the-numbers">Show me the numbers</h2>
<p>We swept values from 1 MB to 64 MB across four Valkey releases on two nodes with 200 Gbps of bandwidth between them. The full setup is below.</p>
<p><img src="https://www.gomomento.com/assets/content/blog/consistency-compounds-valkeys-journey-to-200-gbps/8mb-release-journey.svg" alt="GET and SET bandwidth for 8 MB values across Valkey 7.2, 8.1, 9.0, and 9.1"></p>
<h3 id="valkey-72-reads-capped-at-35-gbps-writes-at-55">Valkey 7.2: reads capped at 35 Gbps, writes at 55</h3>
<p><img src="https://www.gomomento.com/assets/content/blog/consistency-compounds-valkeys-journey-to-200-gbps/release-7-2-bandwidth.svg" alt="GET and SET bandwidth by value size for Valkey 7.2.13, with GET shown as a solid line and SET shown as a dashed line"></p>
<p>An 8 MB value delivers 21 Gbps on reads and 23 Gbps on writes, with p99 latencies of 174 and 164 ms. Turning on <code>io-threads-do-reads</code> moved only the 1 MB read cell.</p>
<h3 id="valkey-80-and-81-writes-take-off-large-reads-stay-put">Valkey 8.0 and 8.1: writes take off, large reads stay put</h3>
<p><img src="https://www.gomomento.com/assets/content/blog/consistency-compounds-valkeys-journey-to-200-gbps/release-8-1-bandwidth.svg" alt="GET and SET bandwidth by value size, comparing Valkey 7.2.13 in red with Valkey 8.1.8 in orange"></p>
<p>The threading rebuild lifts our 8 MB write from 23 to 137 Gbps, six times faster, and 1 MB reads reach 183 Gbps. Larger reads settle at 30-33 Gbps whether the value is 8 MB or 64 MB. That flat floor points to a serial, per-byte bottleneck. Valkey 8.0.9 and 8.1.8 measured the same at every size, so one line carries both.</p>
<h3 id="valkey-90-large-gets-jump-to-line-rate">Valkey 9.0: large GETs jump to line rate</h3>
<p><img src="https://www.gomomento.com/assets/content/blog/consistency-compounds-valkeys-journey-to-200-gbps/release-9-0-get-bandwidth.svg" alt="GET bandwidth by value size, comparing Valkey 8.1.8 in orange with Valkey 9.0.4 in green"></p>
<p>Every size from 1 MB to 64 MB reads at 190-201 Gbps. The 8 MB p99 falls from 112 to 24 ms, and a 64 MB read drops from just under a second to 230 ms. Writes do not move because the ingest path was never copy-bound. That ceiling waits for 9.1.</p>
<h3 id="valkey-91-more-headroom-for-writes">Valkey 9.1: more headroom for writes</h3>
<p><img src="https://www.gomomento.com/assets/content/blog/consistency-compounds-valkeys-journey-to-200-gbps/release-9-1-set-bandwidth.svg" alt="SET bandwidth by value size, comparing Valkey 9.0.4 in dark green with Valkey 9.1.0 in light green"></p>
<p>With reads pinned at the network limit, the gain surfaces on writes. The 8 MB write rises from 134 to 166 Gbps, a 24% gain, while values from 12 MB to 64 MB gain 11-19%.</p>
<h2 id="the-short-run-held">The short run held</h2>
<p>To make sure the 15-second runs were not catching a lucky window, I reran every 8 MB GET and SET cell for 15 minutes. Valkey 9.1 is a good example: GET held 200.8 Gbps and SET held 165.5 Gbps, right on top of the original 201 and 166 Gbps results. Nothing sagged as the runs went on.</p>
<picture>
  <source media="(max-width: 640px)" srcset="https://www.gomomento.com/assets/content/blog/consistency-compounds-valkeys-journey-to-200-gbps/long-run-stability-mobile.svg">
  <img src="https://www.gomomento.com/assets/content/blog/consistency-compounds-valkeys-journey-to-200-gbps/long-run-stability.svg" alt="Valkey 9.1 GET and SET throughput holding steady over 15 minutes" loading="lazy">
</picture>
<h2 id="detailed-results">Detailed results</h2>
<h3 id="reads-get">Reads (GET)</h3>
<p>GET-only, 32 connections, 100% hit rate.</p>
<h4 id="get-bandwidth-gbps">GET bandwidth (Gbps)</h4>



























































<table><thead><tr><th>Value size</th><th align="right">7.2.13</th><th align="right">8.1.8</th><th align="right">9.0.4</th></tr></thead><tbody><tr><td>1 MB</td><td align="right">32</td><td align="right">183</td><td align="right">200</td></tr><tr><td>2 MB</td><td align="right">35</td><td align="right">55</td><td align="right">193</td></tr><tr><td>4 MB</td><td align="right">26</td><td align="right">45</td><td align="right">197</td></tr><tr><td>8 MB</td><td align="right">21</td><td align="right">33</td><td align="right">200</td></tr><tr><td>12 MB</td><td align="right">21</td><td align="right">32</td><td align="right">191</td></tr><tr><td>16 MB</td><td align="right">21</td><td align="right">32</td><td align="right">201</td></tr><tr><td>32 MB</td><td align="right">21</td><td align="right">32</td><td align="right">196</td></tr><tr><td>64 MB</td><td align="right">22</td><td align="right">31</td><td align="right">191</td></tr></tbody></table>
<p>Note: Valkey 9.1 reads also hold line rate, so the table stops at 9.0.</p>
<h4 id="get-p99-latency-ms">GET p99 latency (ms)</h4>



























































<table><thead><tr><th>Value size</th><th align="right">7.2.13</th><th align="right">8.1.8</th><th align="right">9.0.4</th></tr></thead><tbody><tr><td>1 MB</td><td align="right">13</td><td align="right">2.2</td><td align="right">2.4</td></tr><tr><td>2 MB</td><td align="right">20</td><td align="right">19</td><td align="right">5.1</td></tr><tr><td>4 MB</td><td align="right">58</td><td align="right">41</td><td align="right">9.4</td></tr><tr><td>8 MB</td><td align="right">174</td><td align="right">112</td><td align="right">24</td></tr><tr><td>12 MB</td><td align="right">244</td><td align="right">166</td><td align="right">56</td></tr><tr><td>16 MB</td><td align="right">329</td><td align="right">238</td><td align="right">56</td></tr><tr><td>32 MB</td><td align="right">531</td><td align="right">489</td><td align="right">116</td></tr><tr><td>64 MB</td><td align="right">948</td><td align="right">1,020</td><td align="right">230</td></tr></tbody></table>
<h3 id="writes-set">Writes (SET)</h3>
<p>SET-only, 32 connections.</p>
<h4 id="set-bandwidth-gbps">SET bandwidth (Gbps)</h4>




































































<table><thead><tr><th>Value size</th><th align="right">7.2.13</th><th align="right">8.1.8</th><th align="right">9.0.4</th><th align="right">9.1.0</th></tr></thead><tbody><tr><td>1 MB</td><td align="right">51</td><td align="right">201</td><td align="right">201</td><td align="right">201</td></tr><tr><td>2 MB</td><td align="right">53</td><td align="right">200</td><td align="right">200</td><td align="right">201</td></tr><tr><td>4 MB</td><td align="right">55</td><td align="right">200</td><td align="right">200</td><td align="right">201</td></tr><tr><td>8 MB</td><td align="right">23</td><td align="right">137</td><td align="right">134</td><td align="right">166</td></tr><tr><td>12 MB</td><td align="right">23</td><td align="right">135</td><td align="right">139</td><td align="right">166</td></tr><tr><td>16 MB</td><td align="right">24</td><td align="right">137</td><td align="right">139</td><td align="right">165</td></tr><tr><td>32 MB</td><td align="right">24</td><td align="right">139</td><td align="right">134</td><td align="right">159</td></tr><tr><td>64 MB</td><td align="right">24</td><td align="right">130</td><td align="right">134</td><td align="right">149</td></tr></tbody></table>
<h4 id="set-p99-latency-ms">SET p99 latency (ms)</h4>




































































<table><thead><tr><th>Value size</th><th align="right">7.2.13</th><th align="right">8.1.8</th><th align="right">9.0.4</th><th align="right">9.1.0</th></tr></thead><tbody><tr><td>1 MB</td><td align="right">8.7</td><td align="right">3.3</td><td align="right">3.2</td><td align="right">3.2</td></tr><tr><td>2 MB</td><td align="right">17</td><td align="right">6.5</td><td align="right">6.7</td><td align="right">5.3</td></tr><tr><td>4 MB</td><td align="right">35</td><td align="right">12</td><td align="right">13</td><td align="right">14</td></tr><tr><td>8 MB</td><td align="right">164</td><td align="right">41</td><td align="right">49</td><td align="right">34</td></tr><tr><td>12 MB</td><td align="right">289</td><td align="right">62</td><td align="right">57</td><td align="right">52</td></tr><tr><td>16 MB</td><td align="right">357</td><td align="right">77</td><td align="right">70</td><td align="right">60</td></tr><tr><td>32 MB</td><td align="right">1,580</td><td align="right">118</td><td align="right">143</td><td align="right">107</td></tr><tr><td>64 MB</td><td align="right">2,580</td><td align="right">237</td><td align="right">213</td><td align="right">206</td></tr></tbody></table>
<h2 id="the-setup">The setup</h2>
<p><strong>Machines.</strong> Two <code>c8gn.16xlarge</code> instances with Graviton4 and 200 Gbps networking in the same availability zone and cluster placement group, running Amazon Linux 2023.</p>
<p><strong>Server.</strong> We tested the official <code>valkey/valkey</code> Docker image at versions <code>7.2.13</code>, <code>8.1.8</code>, <code>9.0.4</code>, and <code>9.1.0</code>. We also measured <code>8.0.9</code>. It matched <code>8.1.8</code> within run-to-run noise at every size, so the tables show 8.1 as the 8.x column. Valkey 7.2 ran with the same flags as the newer versions. A control with <code>io-threads-do-reads yes</code> changed only the 1 MB GET cell, from 32 to 41 Gbps. Each version ran in a fresh container with host networking:</p>
<pre><code class="language-sh">docker run --network host --cpuset-cpus 8-23 \
  --ulimit nofile=32768:65536 valkey/valkey:&#x3C;version> \
  --save '' --appendonly no --io-threads 16 \
  --protected-mode no --maxmemory 60gb
</code></pre>
<p>Persistence was off. The 16 threads in Valkey’s <code>io-threads</code> count were one main thread plus 15 I/O workers. The process was pinned to cores 8-23 so it never fought the kernel for the cores doing network interrupt work.</p>
<p><strong>Interrupts.</strong> <code>irqbalance</code> was off on both machines. The ENA NIC was configured with four combined queues and their IRQs pinned to cores 0-3. Without this step, results wander from run to run as the kernel shuffles interrupts onto whatever cores the server or client threads happen to be using. If you benchmark at these speeds, pin your IRQs first and thank yourself later.</p>
<pre><code class="language-sh"># Run on both machines. ens50 is the ENA interface name on these instances.
sudo systemctl stop irqbalance
sudo ethtool -L ens50 combined 4
i=0
for irq in $(grep ens50 /proc/interrupts | awk '{print $1}' | tr -d ':'); do
  echo $((i%4)) | sudo tee /proc/irq/$irq/smp_affinity_list > /dev/null
  i=$((i+1))
done
</code></pre>
<p><strong>Client.</strong> We used <a href="https://github.com/cachecannon/cachecannon">valkey-lab</a>, built on cachecannon, with 16 worker threads pinned to cores 4-19, 32 connections, and pipeline depth 1. Reads and writes were measured in separate passes. Read passes prefilled the keyspace and ran at a 100% hit rate. Each short-run cell was a 15-second measurement after a five-second warmup, over a 500-key keyspace with 16-byte keys.</p>
<p>We kept concurrency at 32 connections because AWS caps a single TCP flow at roughly 9.5 Gbps. We verified 9.53 Gbps with iperf3. Saturating a 200 Gbps NIC requires spreading the load across flows.</p>
<p>One invocation per cell, with <code>-s</code> swept across the value sizes:</p>
<pre><code class="language-sh"># GET pass: prefill the 500-key keyspace, then measure at a 100% hit rate.
valkey-lab -h &#x3C;server> -c 32 -P 1 -t 16 --cpu-list 4-19 \
  -n 500 -r 100:0 --prefill --warmup 5s -d 15s \
  -s &#x3C;value_bytes> --key-size 16

# SET pass.
valkey-lab -h &#x3C;server> -c 32 -P 1 -t 16 --cpu-list 4-19 \
  -n 500 -r 0:100 --warmup 5s -d 15s \
  -s &#x3C;value_bytes> --key-size 16
</code></pre>
<p><strong>Scope of the result.</strong> This is a single-node, no-TLS, persistence-disabled test with 32 closed-loop connections, pipeline depth 1, and a 100% hit rate. Most table cells are one 15-second measurement after warmup. The results describe this rig and workload. Broader production claims need repeated runs and additional configurations.</p>
<p><em>We benchmarked whole releases rather than individual changes, so we treat the pull requests named in this post as leading explanations rather than proof of causality.</em></p>
<h2 id="what-this-means-if-you-run-valkey">What this means if you run Valkey</h2>
<p>Valkey 9.x delivers roughly 6x the 64 MB read bandwidth of 8.x while cutting p99 from just under a second to about 230 ms on this single-node, no-TLS test. Our earlier mixed-workload test also found far less collateral latency for small requests when a large read arrived.</p>
<p>The practical takeaway is narrower than “Valkey is always faster.” Valkey 8.1 is essentially flat on this workload, Valkey 9.0 changes the large-GET path, and Valkey 9.1 improves large SETs on this rig. If large values matter to your workload, test 9.x with your object-size distribution, concurrency, TLS, and persistence settings rather than extrapolating from a small-object benchmark, or from this one.</p>
<p>Valkey has consistently improved performance, memory efficiency, and availability at scale. Each version brings about a new set of improvements driven by issues faced by real users in production. A vibrant community where nobody is incentivized to withhold features for the sake of revenue and everyone is incentivized to chase continuous improvement is what makes Valkey truly special. Effort is additive. Consistency is multiplicative.</p>
<p>Serving megabyte-sized objects at wire speed is what I’ve been spending a lot of time on lately. If you are wrangling larger objects, let’s talk. <a href="https://valkey.io/slack/">Join me on Valkey Slack</a>.</p>]]></content:encoded>
    </item>
    <item>
      <title>Introducing valkey-lab: Stop Guessing When Your Cache Hits Its Limit</title>
      <link>https://www.gomomento.com/blog/introducing-valkey-lab-stop-guessing-when-your-cache-hits-its-limit/</link>
      <dc:creator><![CDATA[Khawaja Shams]]></dc:creator>
      <pubDate>Tue, 26 May 2026 20:14:51 GMT</pubDate>
      <category><![CDATA[Valkey]]></category>
      <guid isPermaLink="true">https://www.gomomento.com/blog/introducing-valkey-lab-stop-guessing-when-your-cache-hits-its-limit/</guid>
      <description><![CDATA[<p>Benchmarking a cache should be about finding how much load your system can sustain before latency SLOs break. Introducing valkey-lab.</p>]]></description>
      <content:encoded><![CDATA[<p><img src="https://www.gomomento.com/wp-content/uploads/2026/05/valkey-lab-1024x566.jpg" alt=""></p>
<p>Pop quiz: how many requests per second can your cache take before it stops meeting your latency SLO?</p>
<p>Chances are good you don’t know the answer, and that’s not a knock on you. It’s a genuinely hard number to come by. The standard tool for this is <a href="https://valkey.io/topics/benchmark/">valkey-benchmark</a>, and it’s great at exactly one thing: you point it at a server, you hammer it with some commands at full speed, and it prints a throughput number at the end. That number tells you the box is alive and roughly how fast it goes flat out.</p>
<p>From a production standpoint, that’s not as useful as it sounds.</p>
<p>How does the p999 hold up under an 80:20 read/write mix when half the requests land on hot keys? What does the tail look like at the rate you actually plan to run? How much headroom do you have before the SLO breaks? Was that latency spike at 10:32 a fluke or the ceiling? A single summary number printed after a sixty-second run can’t answer any of those, because it threw away the useful bits on the way to computing the average.</p>
<p><a href="https://github.com/cachecannon/cachecannon/blob/main/VALKEY-LAB.md">valkey-lab</a> was built to answer these questions.</p>
<p>It’s a high-performance Valkey and Redis benchmark that uses <code>io_uring</code> for kernel-bypassed I/O, per-connection pipelining, and multi-threaded workers. The defaults are deliberately familiar. Run it without any arguments:</p>
<p>📄</p>
<pre><code>valkey-lab

</code></pre>
<p>and you get a sixty-second run against <code>localhost:6379</code> with an 80:20 GET/SET ratio and a million keys. Same shape as the tool you already know, same short flags (<code>-h</code>, <code>-p</code>, <code>-c</code>, <code>-P</code>, <code>-r</code>). The interesting part starts once you begin asking harder questions.</p>
<h2 id="saturation-search-the-headroom-number-found-for-you"><strong>Saturation search: the headroom number, found for you</strong></h2>
<p>Here’s how you find your headroom number today, by hand. You run a benchmark at some rate, read the p999, decide it looks healthy, bump the rate, run it again, read it again. You do this five or ten times, squinting at each result, trying to find the rate where the tail skyrockets. Somewhere in that loop you lose track of which run had which config. Eventually you settle on a number you’re “pretty sure” is right and plan capacity around it. valkey-lab does that whole search for you with one command.</p>
<p>📄</p>
<pre><code>valkey-lab saturate --slo-p999 1ms -c 16 -P 32

</code></pre>
<p>A synthetic benchmark might say a cache can handle 2M requests per second. But when you add a realistic read/write mix, hot keys, and a warm cache, your p999 suddenly crosses your SLO at 1.2M. Technically speaking, the server is still processing 2M requests per second, but the usable ceiling is significantly lower.</p>
<p><code>saturate</code> starts issuing requests at whatever is provided in <code>--start-rate</code> (or 1000 if not provided) and multiplies the request rate by the <code>--step</code> factor on every step. The default step is <code>1.05</code>, so the load compounds over time. Each step holds its rate for a sample window, measures the full percentile spread, and checks it against your SLO. The moment a percentile crosses the line, the ramp stops and reports the last rate that held.</p>
<p>When a step fails, valkey-lab tells you <em>how</em> it failed, either throughput-limited or latency-exceeded, which helps you tune your clusters more accurately.</p>
<p>Throughput-limited means the server couldn’t generate the requested rate at all. It topped out below the target. That’s a capacity problem: you need more CPU, more shards, or a different topology.</p>
<p>Latency-exceeded means the server kept up with the rate, but the tail blew past the SLO. The server can sustain the requested rate, but something in the path is introducing tail spikes under load. Could be a hot key, a GC pause, a scheduler stall, network jitter. You fix that by chasing the spike, and adding hardware won’t help.</p>
<p>So based on your failure, your mitigation strategy varies wildly. And it would be impossible to know which one to pursue if all you had was the throughput number.</p>
<h2 id="averages-hide-the-interesting-failures"><strong>Averages hide the interesting failures</strong></h2>
<p>The next problem surfaces when the benchmark completes. Summary statistics hide the behavior you’re usually trying to find.  If your p999 was 312µs for fifty-nine seconds and 4.2 ms for one second, the run-level p999 still looks fine. The spike is the important part that you need to focus on.</p>
<p>valkey-lab streams one row per second with the full latency spread:</p>
<p><img src="https://www.gomomento.com/wp-content/uploads/2026/05/valkey-lab-1-1.jpg" alt=""></p>
<p>Every major percentile from p50 to p99.99 plus the max, the error count, and the cache hit rate, all per second. A spike that lasts one second appears as one row with a tall tail, a vast improvement over the executive summary at the end of a run. When you need it machine-readable instead, –output json gives you newline-delimited JSON you can pipe straight into something else, and –output quiet collapses the whole run to a single summary line. </p>
<h2 id="make-the-benchmark-look-like-your-workload"><strong>Make the benchmark look like your workload</strong></h2>
<p>There’s an important gotcha with the saturation number, or any benchmark number. A ceiling is only as good as the load that produced it, and the default load most tools run is unrealistic.</p>
<p>Think about what a stock benchmark actually does. It sends all reads, or close to it, because a 100% GET run posts the biggest number (or it’s the easiest to simulate). It picks keys uniformly at random, so every key is equally cold and nothing is ever hot. It runs flat out, measuring throughput at saturation. And it normally starts against an empty cache. Now think about your production traffic. It’s a read-write mix. It has hot keys, with a small fraction of the keyspace taking most of the requests. And the cache is warm. Every one of those differences takes away from the realism of the benchmark run.</p>
<p>valkey-lab addresses each one of these gaps. Set the real read-write split with -r so you’re measuring the write path your cache actually carries. Turn on –distribution zipf so a small fraction of keys receives most of the traffic, like production systems often do. Uniform access patterns avoid contention and hide the behavior of your actual hot paths. </p>
<p>Pin the load with –rate-limit to track latency at the rate you plan to run. And warm the cache with –prefill, or model a read-through cache that fills on miss with –backfill, so a GET benchmark measures hits the way production would.</p>
<p>📄</p>
<pre><code>valkey-lab --prefill -r 100:0 --distribution zipf -c 16 -P 32

</code></pre>
<p>Stack those and the ceiling you measure is a ceiling that meaningfully tracks production. There’s more depth when you need it, warmup tuning, RESP3, pinning workers to cores with –cpu-list, TLS, full TOML configs, but the move that matters is making the four big assumptions match your reality before you trust the number.</p>
<h2 id="getting-the-important-data-from-a-run"><strong>Getting the important data from a run</strong></h2>
<p>Now that we have realistic benchmark data, we have to make sure it’s useful after the run ends.</p>
<p><code>--parquet results.parquet</code> saves the full dataset to disk. It stores the full metric set per snapshot: the counters, the gauges, and the latency distributions as actual nanosecond histograms. Combine this with the visualization functionality in valkey-lab, and you have a rich experience that lets you dig into every tiny detail.</p>
<p>📄</p>
<pre><code>valkey-lab --parquet results.parquet
valkey-lab view results.parquet

</code></pre>
<p><code>view</code> opens an interactive dashboard against the file, with a synchronized time axis you can zoom and pan through dimensions like throughput, hit rate, error rate, and latency split out by GET, SET, and combined, all on a log scale. Scrub to the exact second p999 jumped and read every other metric in that same window. </p>
<p><img src="https://www.gomomento.com/wp-content/uploads/2026/05/valkey-lab-2-1024x508.jpg" alt=""></p>
<p>One use case for this is regression testing. Because every run is a Parquet file with the same schema, runs are directly comparable to each other. Benchmark before a Valkey upgrade and after, and the question “<em>did this move my tail latency</em>” is easily answered with a diff. The viewer is one way to read these files, but using your own queries is another easy way to act on changes in performance. <a href="https://duckdb.org/">DuckDB</a>, <a href="https://pandas.pydata.org/">pandas</a>, and <a href="https://pola.rs/">Polars</a> all read Parquet directly, so a few lines of SQL across a directory of runs is a regression suite for cache performance. Point DuckDB at a folder of recorded runs and let it compute peak throughput per file:</p>
<p>📄</p>
<pre><code>SELECT
  filename,
  max(responses_received) AS total_responses,
  max(request_errors)     AS errors
FROM read_parquet('runs/*.parquet', filename = true)
GROUP BY filename
ORDER BY filename;
</code></pre>
<p>That is a before-and-after table for every benchmark you have ever saved, built from data you already recorded. </p>
<p>Another use case for the Parquet output is root cause analysis. A spike on the latency chart tells you when something went wrong, not why. Point <code>view</code> at a <a href="https://github.com/iopsystems/rezolus">Rezolus</a> capture from the server or the client and it overlays system telemetry, CPU utilization, network, scheduler behavior, aligned to the same benchmark timeline. When a p999 spike lines up exactly with a scheduler stall or a network hiccup on the axis above it, you have your answer as simple as that.</p>
<h2 id="stop-guessing">Stop guessing</h2>
<p>Back to the pop quiz. The reason it’s so hard to answer is that traditionally the tool you use to measure max RPS reports a summary and throws the important bits away. valkey-lab changes the approach. It remembers the mix, the hot keys, the per-second tail, and records your runs so you can come back to them. The headroom number that used to take an afternoon of manual ramping is now a single command, and it comes with the failure mode attached so you know what to do about it.</p>
<p>valkey-lab is built on top of <a href="https://github.com/cachecannon/cachecannon">cachecannon</a>, inheriting its workload generation, saturation search, telemetry collection, and analysis capabilities. It needs Linux for io_uring (kernel 6.0+) and builds with Rust, under your choice of Apache-2.0 or MIT. Here is the whole getting-started path:</p>
<p>📄</p>
<pre><code>cargo install --path . --bin valkey-lab
valkey-lab saturate --slo-p999 1ms
</code></pre>
<p>Run that against a Valkey server and see what number comes back. Stop asking “<em>how fast can my cache go</em>” and start asking “<em>how fast can it go before my production workload breaks?</em>” That’s the number you capacity-plan around if you want predictable systems at 3 AM.</p>]]></content:encoded>
    </item>
    <item>
      <title>Why Snap Was Willing to Fork, and Why They Still Came Back</title>
      <link>https://www.gomomento.com/blog/why-snap-was-willing-to-fork-and-why-they-still-came-back/</link>
      <dc:creator><![CDATA[Allen Helton]]></dc:creator>
      <pubDate>Thu, 21 May 2026 19:36:05 GMT</pubDate>
      <category><![CDATA[Caching]]></category>
      <category><![CDATA[Valkey]]></category>
      <guid isPermaLink="true">https://www.gomomento.com/blog/why-snap-was-willing-to-fork-and-why-they-still-came-back/</guid>
      <description><![CDATA[<p>Snap ran 100% of their caching infrastructure on KeyDB, a Redis fork they acquired in 2022. Two years later, they moved to Valkey. Here&amp;#039;s why.</p>]]></description>
      <content:encoded><![CDATA[<p><img src="https://www.gomomento.com/wp-content/uploads/2026/05/fork-hero-1024x683.jpg" alt=""></p>
<p>I have no intention of ever forking a database. The amount of bravery and engineering mastery that goes into it scares me to no end. But Snap did. They committed to it so hard that they acquired the company building it, open sourced the entire commercial codebase, and ran 100% of their caching infrastructure on it for years. <a href="https://docs.keydb.dev/">KeyDB</a> powered Snapchat at a scale most companies can only dream of.</p>
<p>And then they migrated to Valkey anyway.</p>
<p>At <a href="https://www.unlockedconf.io/san-jose-replays">Unlocked San Jose</a>, Ovais Khan, Principal Software Engineer at Snap, walked through that migration. As interesting as it was to hear <em>how</em> they did it, it was all the more interesting to hear <em>why.</em> Why it happened, why it wasn’t worth staying on the fork, and why when they came back, they came back to Valkey. </p>
<h2 id="the-case-for-forking-in-2019"><strong>The case for forking in 2019</strong></h2>
<p>KeyDB started in 2019 as a project by John Sully and Ben Schermel at EQ Alpha Technology. The premise was simple. Redis ran a single-threaded event loop. Modern servers had 32, 64, 96 cores. To get peak throughput out of a single machine, you had to run a cluster of Redis nodes on it. That was wasteful, and Salvatore Sanfilippo, the creator of Redis, was on record <a href="https://antirez.com/news/126">arguing against changing it</a>: “<em>I/O threading is not going to happen in Redis AFAIK, because after much consideration I think it’s a lot of complexity without a good reason.</em>” Simplicity of the codebase was a value he was actively protecting. </p>
<p>KeyDB took the other side of that bet. It added real multithreading, with per-thread event loops and lock-based synchronization on shared state. It also added active-active replication and FLASH storage for cost-efficient large datasets. On the same hardware, it could move several times the operations per second that Redis could.</p>
<p>This is the textbook case for forking. The upstream project had made a deliberate architectural choice. That choice was the right one for them and the wrong one for a certain kind of user (Snap) who needed to push a single node harder. A fork was the only way forward.</p>
<p>By 2021, Snap was running KeyDB across enough of their caching infrastructure to want a permanent stake in it. They <a href="https://docs.keydb.dev/news/2022/05/12/keydb-joins-snap/">acquired the team in May 2022</a> and brought the formerly commercial KeyDB Pro features into the open source codebase under BSD-3. For about two years after that, all of Snap was running on KeyDB.</p>
<h2 id="what-forking-buys-you"><strong>What forking buys you</strong></h2>
<p>The benefits of forking are easy to articulate when you ship. Snap got features that were important for their specific operating model:</p>
<ul>
<li>Multithreaded command execution, which let them get more out of every node</li>
<li>Zone-aware read routing, which kept cross-AZ traffic down and cut data transfer costs considerably</li>
<li>Forkless background saves, which made snapshots predictable at high memory</li>
<li>Same-zone replica behavior that reduced timeout blast radius during upgrades</li>
</ul>
<p>These features weren’t going to make it into Redis on Snap’s timeline. The fork gave them room to build it as soon as they were ready.</p>
<p>As far as forking goes, that’s usually the part written in blog posts and talked about on the conference loop. You wanted a feature, the upstream said no, you built it yourself, and now it works. Forking feels like freedom.</p>
<h2 id="what-forking-costs-you"><strong>What forking costs you</strong></h2>
<p>Every change to upstream Redis after the fork point became a decision. Does it get ported over? Rewritten? Skipped? There’s a long tail at the end of whatever decision was made. Porting means you carry merge conflicts forever. Rewriting means you have two implementations of the same idea drifting apart. Skipping means your fork stops being a superset of upstream and starts being something else.</p>
<p>Ovais addressed this specifically in his talk. Snap could not easily move from KeyDB’s Redis 6.2 base to Redis 7.2. The cost of staying current with upstream had become high enough that they were stuck on a flavor of 6.2 while everyone else moved on. That meant they were also stuck without features the broader community had built on top of 7.2.</p>
<p>The same goes for the ecosystem. Every client library, operator, monitoring tool, and benchmark gets tested against upstream first. Your fork either matches upstream behavior closely enough that those tools just work, or it doesn’t, and you start maintaining your own.</p>
<p>While forking might have started off feeling like an accelerator, it quickly became a drag.</p>
<h2 id="the-redis-license-change"><strong>The Redis license change</strong></h2>
<p>In March 2024, Redis Ltd. changed the Redis license <a href="https://redis.io/blog/redis-adopts-dual-source-available-licensing/">from BSD-3 to a dual SSPL and RSALv2 model</a>. Neither license is OSI-approved. For any company offering Redis as a managed service, this was an immediate problem. AWS, Google Cloud, Oracle, and Ericsson responded by forking the last BSD release, Redis 7.2.4, and donating it to the Linux Foundation. Eight days after the license change, Valkey existed.</p>
<p>Up until then, the case for staying on KeyDB was obvious. The KeyDB team was inside Snap. The codebase was theirs. The performance was what they needed. </p>
<p>But Valkey made them pause. The project had open governance under the Linux Foundation, with a Technical Steering Committee across multiple companies and no single controlling vendor. It was BSD-licensed and would stay that way. Its roadmap included the things Snap had previously forked to get: <a href="https://valkey.io/blog/unlock-one-million-rps/">I/O threading</a>, <a href="https://github.com/valkey-io/valkey/issues/2083">dual-channel replication</a>, and a path toward features Snap wanted. And every major cloud provider was committing serious engineering effort to it.</p>
<p>The KeyDB story also got more complicated from the inside. In January 2025, John Sully, KeyDB’s original creator, <a href="https://github.com/Snapchat/KeyDB/issues/895">left Snap</a>. His parting note on the KeyDB repository said it plainly:</p>
<p>“When we made KeyDB we wanted to prove that caches should have great performance and I think we succeeded. Now there are many options, including Valkey which is fully open source and based on my testing has matched KeyDB’s performance. I’m not sure what Snap will do with the project, but I think that development effort should move to Valkey moving forward as they have clear momentum and are the most up to date.”</p>
<p>When the person who started the fork tells you the fork is done, the fork is done.</p>
<h2 id="the-secret-migration-back"><strong>The secret migration back</strong></h2>
<p>Snap runs caching at a scale where you can’t just swap a binary. The migration had to be invisible to application teams, comparable in cost, and safe across radically different workload types. Ovais walked through the major decisions that made their migration as easy as possible.</p>
<h3 id="abstraction-layers-are-key-to-managing-workloads-at-scale"><strong>Abstraction layers are key to managing workloads at scale</strong></h3>
<p>Snap had built a storage abstraction with a RESP proxy in front of every cluster. Applications never talked to KeyDB directly. They talked to the proxy, which spoke Redis wire protocol back to whatever was running behind it. That layer of indirection made this migration possible. Without it, every application team at Snap would have needed to know about the change. With it, nobody had to.</p>
<p>These layers let them migrate around 30 caches per week. By the time Ovais gave this talk, 70 to 80 percent of workloads were on Valkey.</p>
<h3 id="do-a-gap-analysis-before-changing-any-code"><strong>Do a gap analysis before changing any code</strong></h3>
<p>Snap did a feature-by-feature comparison between KeyDB and Valkey before touching anything in production. KeyDB’s multithreading and Valkey’s I/O threading work differently, so they benchmarked carefully to confirm comparable throughput. </p>
<p>Some KeyDB features were blockers and had to be ported to Valkey. Zone awareness was the first one Snap contributed. Replica MOVED behavior during upgrades was another. CPU throttling at high utilization was a third. </p>
<p>A hidden gap that wasn’t found until much later was with <code>MGET</code>. KeyDB supported it across slots, but Valkey does not. So after moving to Valkey, Snap had issues with command parsing pressure in large batching workloads. They quickly ported cross-slot <code>MGET</code> to their internal build, and are working with the core maintainers to get it added upstream.</p>
<h3 id="pick-a-stable-version-for-a-base-not-a-new-one"><strong>Pick a stable version for a base, not a new one</strong></h3>
<p>Snap started on Valkey 8.2 RC, ported the features they needed, and immediately ran into crashes at 9 to 10k QPS. The root cause was new TLS offloading work. They rolled back to 8.0.2, ported the necessary fixes onto that, and benchmarked from there. New releases need a baking period, and a migration is the wrong time to find out.</p>
<h3 id="categorize-and-prioritize-your-workloads"><strong>Categorize and prioritize your workloads</strong></h3>
<p>Snap divided their caches into three categories: CPU-bound, high-memory, and high-write-rate. Each category needed different validation. CPU-bound workloads were primarily a throughput question. High-memory workloads were really about replication buffer behavior during full syncs, because if the buffer fills before a snapshot completes, you enter a sync loop that never finishes. High-write workloads required tuning replica buffer sizes and primary write throttling, because Valkey’s dual-channel replication puts buffers on replicas rather than primaries. Inside each category, they went lowest-criticality first, highest-criticality last.</p>
<h2 id="lessons-from-going-full-circle"><strong>Lessons from going full circle</strong></h2>
<p>The fork was the right call in 2019. Redis was not going to go multithreaded, and the workloads Snap was running needed it. KeyDB was a solid piece of engineering that pushed the ceiling on what a single Redis-compatible node could do.</p>
<p>The migration back was the right call in 2025 because the conditions that justified the fork had changed. The upstream that resisted features they needed was no longer the upstream they cared about. Valkey’s governance was open. Its roadmap included the work Snap had previously done alone. And every additional year on a Redis 6.2 build was another year of compounding distance from where the ecosystem was going.</p>
<p>Forks are leverage. They are also debt. Be honest with yourself about which one you are accumulating at any given moment. Snap was. They forked when forking gave them speed, and they came back when the fork started to cost more than it earned.</p>
<p>I don’t want you to take away from this that forking is bad. Sometimes it’s the right thing to do. The decision to fork is not permanent, and treating it like it is permanent is how you end up running a five-year-old codebase while your competitors are shipping on a roadmap you helped fund.</p>
<p>When the world moves, move with it.</p>
<p>Happy coding!</p>]]></content:encoded>
    </item>
  </channel>
</rss>
