A beacon carries device telemetry with selected signals for downstream analysis. The event data it carries might support quality-of-experience monitoring, anti-piracy detection or personalization. Selecting those signals matters to ingestion as well as analysis. Every added field increases the payload, and those extra bytes multiply across the beacon traffic that the system must receive and hold.

Beacon ingestion therefore needs a protective boundary between device traffic and internal processing. In this architecture, Momento Valkey Router receives beacons through a gateway tier, manages device connections and batches data for forwarding. Shared, open-source Valkey collects related events and buffers them until downstream services can process them. Together, the gateway and Valkey turn an unpredictable beacon firehose into organized data that internal services can consume at a manageable rate.

Valkey Router and shared Valkey collection form a protective ingestion boundary between device traffic and downstream services. Kafka, ClickHouse and S3 illustrate downstream choices.

Protect internal services from device traffic

The gateway separates the audience’s connections from the connections that internal infrastructure must support. Without that boundary, a growing device population can increase database connection pressure as well as beacon volume. These are distinct demands. An internal service might have enough processing capacity for the events while struggling to manage the connections delivering them.

Valkey Router handles device connections and reuses a pool of connections to Valkey. Pooling prevents each device connection from becoming a separate Valkey connection. The gateway can receive traffic from many devices while controlling the number of database connections. Momento supplies this gateway layer alongside Valkey’s open-source storage foundation, placing device-facing capabilities ahead of shared event collection.

The gateway also controls access to that collection. Authentication and access checks determine whether incoming traffic is allowed to reach internal infrastructure. HTTPS and TLS termination protect device-to-gateway communication, and custom certificates let the endpoint use the certificates required by the deployment. These capabilities put connection and access handling at the boundary where device traffic enters the system.

Throttling limits the forwarding rate when incoming traffic would otherwise overwhelm the next stage. The gateway controls ingestion before excess traffic reaches the database and consumers. Connection pooling and throttling address different parts of the same protection. One controls database connections, while the other controls the rate of data reaching internal services.

Share forwarding overhead across small beacons

Managing connections still leaves a large number of small payloads to forward. Sending every beacon separately repeats communication and request-handling overhead. When that repeated overhead consumes a substantial part of ingestion capacity, the system spends effort moving small pieces of telemetry instead of collecting their event data efficiently.

Valkey Router batches compatible beacons locally before forwarding them. Several beacons share the overhead of a batch, reducing the repeated forwarding cost per beacon. Their event data remains available for collection. Batching changes how data is transferred without requiring the application to remove the observations it needs for analysis.

Batching introduces a short wait at the gateway because data must accumulate before it can be sent together. The gateway bounds that wait and the amount of data it gathers. A count trigger sends a batch when enough beacons have accumulated. A timer sends the accumulated data when the waiting interval ends. During busy periods, the count trigger can fill batches quickly. During quieter periods, the timer lets small batches progress instead of leaving beacons waiting for traffic that may not arrive soon.

A size bound can also limit how much data a batch contains. This matters because beacon sizes can vary with the selected signals. A batch with an acceptable beacon count could still contain too many bytes if those beacons have larger payloads. Count, time and size bounds serve complementary purposes. They limit local accumulation while balancing forwarding efficiency against the delay introduced by waiting.

Each gateway assembles its own batches from the traffic it receives. This keeps batching close to device ingestion, where small payloads first accumulate. The local batch holds data briefly to improve transfer efficiency. Shared Valkey collection serves the longer-lived need to hold events for downstream processing. Efficient batches from separate gateways must also reach the right collection point, because each gateway may receive only part of a partition’s activity.

Bring related events to the same collection point

A gateway tier distributes ingestion across multiple instances, but related beacons can reach different gateways. A consumer processing an application-defined partition needs its related events together, even when they come from different devices. Leaving those events fragmented across gateway-local collections would push the task of gathering them onto every downstream consumer.

The architecture collates related events by deriving a grouping key deterministically from beacon properties. Every gateway applies the same grouping rule. Beacons with the same grouping property produce the same grouping key, giving their related activity a consistent logical address. The gateways use that address to route event data to the same Valkey shard. The shared collection brings related events together in a coherent data stream for downstream consumption.

Consider two devices sending beacons assigned to partition S42. One beacon reaches one Valkey Router instance and the other reaches another. Both carry the partition identifier, and both gateways derive the grouping key partition:S42 from it. Each gateway uses that logical address to route the related event data to the selected Valkey shard. The partition’s events meet in shared collection even though different devices and gateways handled the beacons.

The selected shard is a portion of shared Valkey collection that holds multiple keyed groups. The grouping key keeps the partition’s events associated alongside data for other partitions. A consumer can process the collected partition activity without gathering fragments from individual gateways. This separates the gateway that receives a beacon from the collection where its related events belong, allowing the ingestion tier to distribute device traffic while preserving the relationships downstream analysis needs.

Each gateway buffers events locally for efficient forwarding in batches. Matching partition keys bring related events from different gateways to the same Valkey collection point. The arrows follow the three marked S42 events; other partitions and shard placements are illustrative.

Give events room to wait for consumers

Organized events can still arrive faster than downstream services process them. A traffic burst increases ingestion immediately, while a consumer may continue at its previous processing rate. Consumer slowdowns can create the same imbalance even when incoming traffic remains steady. Shared buffering gives the architecture somewhere to hold that difference.

Valkey holds collected event data while consumers process it. During a burst, the buffer grows as ingestion outpaces consumption. When traffic subsides or consumer capacity increases, consumers can catch up and the backlog can shrink. The shared buffer lets the gateway continue receiving traffic through a temporary mismatch in rates, within the capacity available to hold the data.

That capacity is finite. The amount of data waiting depends on both incoming volume and how long it waits for downstream processing. Larger beacon payloads or heavier traffic increase the data entering the buffer. Slower consumers extend the waiting duration. Either change can increase the space required, and both can occur during the same live event.

Headroom accommodates those departures from normal operation. It provides space for traffic spikes, periods of slower consumption and the interval before autoscaling adds capacity. Autoscaling takes time to respond to increased demand. Existing capacity must hold the accumulating data during that response interval, so scaling complements headroom rather than replacing it.

This buffer serves a different purpose from gateway batching. The gateway waits briefly to forward small beacons efficiently. Valkey holds shared, collected events while downstream processing catches up. Keeping those responsibilities distinct explains why efficient forwarding alone cannot absorb a sustained difference between ingestion and consumption.

Hand collected events to downstream services

At the downstream boundary, internal services receive event data that has been collected, grouped and buffered. The ingestion architecture has already handled device connections and brought related events into shared collection. Consumers can use that data for the processing their applications require.

Destinations can include Kafka for event distribution, ClickHouse for analytics and S3 for object storage. These services have different downstream roles, but each can begin from the organized event data available at the handoff. The ingestion boundary’s contribution is to prepare that data while accommodating the device traffic that produces it.

Make live-event traffic manageable

The gateway and shared Valkey buffer protect internal processing through cooperating responsibilities. Valkey Router handles device connections and controls ingestion. Batching shares forwarding overhead across small beacons. Deterministic grouping keys bring related events from different gateways into coherent data streams. Shared buffering gives those events room to wait through temporary differences between ingestion and consumption.

Together, these mechanisms let internal services process organized event data while the boundary handles unpredictable device traffic. Their combined effect depends on sufficient capacity for the traffic and its waiting duration. The architecture makes that relationship explicit, connecting efficient ingestion with the headroom needed to support real-time processing during live events.

If you are designing this boundary, bring your expected beacon traffic, payload sizes, grouping requirements and downstream processing needs to Momento. We can help you explore how Valkey Router and shared Valkey collection fit your workload and topology.

A shared Valkey tier gives those results a place that every compatible worker can reach. Momento’s RDMA and AWS EFA modules provide a direct path between inference GPUs and KV blocks in Valkey RAM. Its NVMe module adds local disk capacity, and its object storage connector asynchronously flushes data to inexpensive long term storage. These capabilities let you retain reusable state beyond the lifetime and memory budget of an individual worker.

Consider an assistant answering questions about an operations manual. Each prompt begins with the same system instructions and manual, followed by a different question. The objective is to reuse that shared prefix wherever the next question runs.

Keep the useful state

KV cache contains the attention keys and values computed for prior tokens. Prefill processes the input and establishes this state. Decode uses it to generate the response and extends it as new tokens arrive. Keeping those tensors avoids recreating them during generation.

Prefix caching extends reuse across requests. If a later prompt has the same token prefix and compatible model state, the engine can reuse completed cached blocks. It still processes the new question and generates an answer. Matching subject matter is insufficient. Changing the instructions or moving the manual later in the prompt can change the reusable prefix.

Remote reuse also differs from prefill and decode disaggregation. Disaggregation runs the two phases on separate worker pools and requires a state handoff between them. A shared cache can support that design, but it also benefits workers that run both phases together. The manual example uses such workers.

Put Valkey between inference and storage

Deploy Valkey on storage nodes separate from the inference workers. GPU memory holds state needed for active inference. Valkey RAM holds reusable blocks shared across the fleet. Local NVMe on the Valkey nodes and an object bucket extend the retained working set.

Separating storage from compute lets you add cache capacity without buying another inference GPU. It also gives new or replaced workers access to previously computed prefixes. Valkey supplies an open source storage foundation with a module extension mechanism. Momento’s modules add the transfer and storage capabilities needed for this architecture.

The following reference design stages lower tier blocks through Valkey RAM. Placement, promotion, and coordination are architectural choices for this design.

Direct GPU transfers use Valkey RAM. Lower tier blocks return through RAM in this reference design. Object writes run asynchronously, separately from immediate reuse.

The runtime integration identifies reusable blocks, coordinates availability, and arranges for tensors to become usable GPU cache state. Lookup and bulk transfer have separate roles. Finding a block does not mean that the GPU is ready to consume it.

Suppose worker A has processed the manual and exported eligible completed blocks. Worker B receives another question and checks its local cache first. For a local miss, the integration looks for compatible prefix blocks in Valkey. A RAM hit uses Momento’s RDMA or EFA module to move the blocks into worker B’s GPU memory. The runtime then processes the unmatched prompt suffix and continues generation.

If no useful blocks are available, worker B computes the prefix. Its new blocks can be admitted to shared storage for later requests. Keep retrieval bounded so a slow lookup does not turn a recoverable miss into a long wait. Once RAM reuse works, the next step is retaining more useful prefixes than RAM alone can hold.

Extend the working set with NVMe and object storage

Give each tier a distinct role. Valkey RAM serves shared prefixes used frequently. Momento’s NVMe module provides more capacity on the storage node for blocks worth keeping outside RAM. Object storage provides a larger retention tier for contexts that may return after hours, sessions, or worker replacements.

In this reference design, requested NVMe blocks are staged into Valkey RAM and then transferred to the GPU through the same fast path. The disk is local to Valkey, so inference workers access it through the storage service. They do not each need a disk copy of every manual.

Momento’s object connector flushes data asynchronously. A block already available in RAM can be reused while its object copy is being written. That decouples immediate reuse from the upload, but it does not make every cache write immediately durable. If the only usable copy disappears before the flush completes, the prefix may need recomputation.

For an object hit, the proposed return path reads retained blocks into Valkey RAM before loading the GPU. Recoverable lookup metadata must accompany retained state so the system can identify compatible objects after a restart. Object storage extends the reuse window, while retrieval still has to fit the request’s latency budget. Choose a storage class that supports the access times you need.

Admission and retention policies keep this hierarchy useful. Prefer blocks likely to be reused over a stream of one-time prompts. Keep copies available until transfers using them finish. Coordinate expiry across lookup metadata and lower tiers so deleted or incompatible state cannot appear as a usable hit. Evaluate the tradeoffs between capacity and coordination when defining your architecture.

Workers still perform attention using state loaded into their GPU cache. The remote tiers preserve reusable blocks between uses. If reading and installing an older prefix costs more than recomputing it, a bounded fallback is the better serving decision. That comparison makes the network part of the cache design.

Match the network to the workload

Measure effective block transfer bandwidth under serving load, including lookup, lower tier reads, and GPU installation. Network line rate alone cannot tell you whether a hit will beat prefill. Concurrent inference traffic and cache fills also compete for bandwidth.

RDMA reduces overhead by moving data between registered memory regions without the usual socket data path. With compatible hardware and software, GPU direct transfer avoids staging the payload through inference host memory. Momento supplies the Valkey side of that direct path.

On AWS, use Momento’s EFA module with compatible GPU and storage endpoints. Confirm the instance pairing and GPU direct support before provisioning. EFA device traffic stays within an Availability Zone and VPC. Place the object bucket in the same Region and access it through normal object APIs.

Connect the serving stack

Momento Cache is the leading provider of Valkey, an open-source fork of Redis built for real-time workloads. With Momento Cache and Valkey, your team can enjoy the same high-performance infrastructure that powers global leaders like Snap, FOX, Coinbase, and Capcom.

Momento provides direct integrations with SGLang, vLLM, and LMCache. Choose the path that fits your serving stack. For the manual example, start with the direct vLLM integration if vLLM already runs your workers. If LMCache already manages offloading, use Momento’s LMCache integration. The direct SGLang integration offers the corresponding entry point for SGLang deployments.

Both workers need the same model revision and adapters, matching tokenized prefixes, and compatible tensor representation and parallel layout. Give a new model or cache format its own namespace. Scope sharing to authorized tenants or trust groups, and protect storage access independently of prefix matching. A content hash alone is not an authorization boundary. These requirements belong in the integration and deployment configuration so application code can stay focused on requests.

How to measure the impact of a shared cache

A useful first deployment needs two compatible inference workers, one separate Valkey node with RAM and NVMe, and an object bucket. Obtain the Momento modules and runtime integration for the chosen topology. Start by proving shared RAM reuse before introducing disk and object retrieval.

Send a long manual prefix to worker A, then send a different question with that same prefix to worker B. Keep local prefix caching enabled in your baseline. Confirm that B lacks a local copy and obtains compatible blocks from shared storage. Compare its time to first token with recomputing the prefix under the same serving conditions.

Next, grow the working set beyond the RAM tier to exercise NVMe. Test object retrieval using blocks whose flush has completed and whose faster copies have been removed through supported controls. Restart an inference worker to check that reuse survives worker turnover. Test an unavailable remote tier and verify bounded retrieval followed by recomputation.

Measure tail latency and transfer time as concurrency rises. Track hits and retrieved bytes by tier, GPU memory pressure, and the object flush backlog. Include cache memory, disk, object requests, and network costs when judging the result. A hit rate without its retrieval cost can make an expensive cache look successful.

Accelerate your KV caching with Valkey and Momento

Valkey is a practical solution when repeated prefixes outlive one worker’s cache and loading their state beats recomputation. Momento’s modules connect that shared RAM tier to inference GPUs and extend it onto disk and object storage. Looking for advice or help to trying it out? Get in touch with Momento and one of our solution engineers will be happy to take a look!

I wanted a reproducible benchmark to assess the progress through the builds. Here is the same benchmark, same tool (sorted-set-benchmark, pipelined ZADD NX, members m:{i}, integer scores), same Graviton4 box (r8g.4xlarge), same protocol: ten repetitions per version with a flush in between, official docker images with --save '' --appendonly no, one server at a time, single benchmark client on the same host. Six Redis versions, five Valkey versions.

The numbers

Memory figures are the server’s used_memory (decimal GB, as the tool reports it). Every version’s ten repetitions landed on the same value at the tool’s reporting precision, roughly 10 MB granularity, with mem_fragmentation_ratio at 1.01x throughout, so RSS tracks these numbers closely. Throughput spreads across the ten runs were tight; the two headline configurations don’t overlap (Valkey 9.1: 643 to 652k/s; Redis 8.10: 628 to 639k/s).

Three things stand out. First, every version of both engines that predates this optimization work sits at the same 4.83 GB for this workload, which is what you’d expect given the shared codebase history of the ZSET implementation. Second, Redis ships a comparable reduction: memory drops from 4.83 GB to 3.73 GB, 23%, exactly at the 8.6 boundary (8.4.7 has the old layout, 8.6.7 the new one), with median inserts rising from 536k/s to 613k/s. Third, Valkey 9.1.2 remains the smallest and fastest configuration I tested: about 10% less memory than Redis 8.10.2 and a small but consistent throughput edge.

The same idea, traveling

I found it really interesting that Redis’s newest number lands within one percent of Valkey 8.1’s: 3.73 GB versus 3.77, about 75 bytes per member either way. Redis in late 2026 is, byte for byte, roughly Valkey of April 2025. That’s not a coincidence, and you can trace exactly why through public pull requests and release notes.

Valkey PR #1427 (merged January 2025, shipped in Valkey 8.1 that April) restructured the sorted set so the hash lookup structure points directly at the skiplist node, instead of keeping its own copy of the mapping. In this benchmark that’s the 4.83 to 3.77 GB step. Valkey PR #2508 (merged November 2025, shipped in 9.1) then embedded the member string inside the skiplist node itself, removing a pointer and a separate allocation per element. That’s the 3.77 to 3.34 GB step.

Redis PR #14701 (merged January 2026, a year and ten days after Valkey’s) copied this work to Redis, and the PR description says where it came from, verbatim: “This is based on: valkey-io/valkey/pull/1427.” Credit to the engineer for saying so plainly; not everyone does. The adaptation kept Redis’s dict (in no_value mode) instead of adopting Valkey’s new hashtable, grafted on the node embedding, and shipped in Redis 8.6. My measurements agree with their release notes: the drop lands exactly at 8.6.

So why is Redis still 10% behind? Because copying is a trailing indicator. Redis took the design but not the hashtable it was built on, and by the time the port shipped, Valkey had already taken the next step. That’s the whole gap: the piece of the design they didn’t adopt, plus the release they haven’t caught up to yet. I mean this without a drop of snark toward the engineer who did the work: Valkey is BSD licensed precisely so anyone, including Redis, can build on it, and Redis users are genuinely better off for it. If the ideas flowed the other way, I’d report that too. But I benchmark what shipped, and what shipped says one project is setting the pace on this data structure and the other is matching it, about a year behind.

The operator’s takeaway

Same caveats as always: one large skiplist-encoded ZSET, short members, integer scores, allocator-level accounting, a single pipelined client. Your data shape will land somewhere else, so run the tool against your own workload.

If you run Valkey, this is simple: 9.1 posted the lowest memory and the highest insert rate of everything on this table, and the upgrade is the whole optimization. I’ll keep running this benchmark as both projects ship.

If you run Redis and are worried that Valkey is more efficient, you have two options. You can wait for Redis to eventually copy the latest Valkey PR, or you can switch to Valkey. What do you plan to do?

Momento Cache: the fastest Valkey in the cloud

You always get the latest Valkey through Momento’s managed services, without managing the infrastructure yourself. Momento smoothly handles upgrades, failover, optimization, and more with the same technology that powers some of the world’s largest caches. Join companies like Snap, Coinbase, Paramount, and Capcom: try Momento Cache now!

We say it jokingly, but it’s true. Inference isn’t well-known for being fast. Powerful, sure. But not fast.

We’ve done a lot of experimentation to make it faster. Parallel fetch, zero-copy reads, batched metadata lookups, the whole retrieval path. But there was one particularly intriguing experiment that has a lot of upside, but some huge trade-offs. What if we stored the KV cache remotely instead of locally? There’s just one problem. The KV cache is huge. Transporting it over a network is a non-starter. Right?

We know how it works. GPU memory is fastest, then host DRAM, then the NVMe bolted to the same box, then, if you really have to, go across the network. You go remote when you need to share, but you pay for it in latency.

I’m not onto a novel idea though. Our industry is all over it right now. NVIDIA has an offloading tier in Dynamo. Mooncake is building a store. Xinnor published a piece in June arguing that Lustre with a client-side cache can hold its own against local NVMe. Khawaja Shams made the case that once you disaggregate prefill and decode, the cache has to travel between machines whether you like it or not.

Clearly there’s merit to it. But I didn’t see much about the cost of the tradeoff. How much of a latency hit do you really take storing your KV cache in something like Valkey, and is skipping prefill still worth it after you’ve paid it?

So I took it upon myself to run the numbers.

The setup

The benchmark uses three machines: two identical GPU nodes running vLLM, and a Valkey box sitting one 10 Gbps hop away from both of them.

gpu-a does all the computing for prefill. The DRAM and NVMe tiers live inside it, and Valkey sits across the network. gpu-b is used to determine what happens when a cached request lands on a machine that did not perform the prefill.

The workload is thirty realistically generated legal and medical documents, all trimmed to exactly 10,000 tokens. Every document goes to vLLM twice. The first pass does the prefill and stores the KV cache. The second pass sends the same prompt, and I time how long the first token takes to come back (TTFT). The only thing that changes between runs is where the KV cache lives.

Everything runs vLLM 0.28.0 with LMCache 0.5.4, valkey-glide-sync 2.5.1, and Valkey 9.1.1, serving Qwen2.5-7B-Instruct-AWQ on NVIDIA L4s.

To get accurate benchmark results I had to turn off vLLM prefix caching so the GPU wouldn’t serve every repeat request. I also had to clear the disk tier and drop the page cache before every single run, because Linux keeps recently written disk data in memory. If I didn’t do that, the “NVMe” reads would come out of RAM and I wouldn’t have actually tested the disk speed.

The goal was to beat cold TTFT, which falls between 3.1 and 3.3 seconds. I ran the entire benchmark twice on separate instances a day apart, and got reproducible numbers within 2%. So I’m feeling good about what I found. Let’s take a peek and see how the network did.

Keeping everything on one machine

For this first test, gpu-a does all the work. It computes the KV, stores it in whichever tier is being tested, and reads it back on the second pass. Even in the Valkey run it’s gpu-a writing and reading its own cache, just over a wire instead of a local bus.

You’re reading that right. The network was the fastest. And by quite a large margin, at that. It beat the local NVMe by 2.7x on the same machine. The disk is physically inside the box with no switch in the way, and the cache was still quicker to reach from across the network.

(Ignore the DRAM row for now. Something interesting happened there, and we’ll talk about it later.)

At first I thought I broke something. Maybe I accidentally put the “disk” tier on the EBS root volume instead of the instance store? But lsblk showed the tier on nvme0n1 (the 250 GB instance store), and df showed 16 GiB written to it during the run, which is about right for 300,000 tokens of KV.

After doing some math, it started making sense.

Each document’s cached KV is 39 chunks of 256 tokens at 57,344 bytes per token, roughly 572 MB (39 x 256 x 57,344). While I was sanity-checking the harness, I ran the DRAM tier with two documents (so everything fit in memory), and hits came back in about 82 ms. A DRAM hit barely transfers anything, which means 82 ms is basically the overhead cost. So any time beyond 82 ms is what I would consider to be transfer time.

That means Valkey’s 557 ms is really about 473 ms of transfer. 572 MB in 473 ms is 1.2 GB/s, which is a 10 Gbps NIC running flat out. Compare that to the disk, which moved 572 MB in ~1,438 ms. That’s 400 MB/s, which is odd, because NVMe should be able to read north of 2 GB/s.

So the remote tier is bound by the network, and the local tier is bound by its own software. Clearly a lot of work went into LMCache’s Valkey connector this year. It pulls with eight parallel workers straight into caller-owned buffers. The local disk path hasn’t gotten the same attention yet, and 400 MB/s from an NVMe says there’s plenty of opportunity. I imagine it won’t stay this way for long.

Anyway, from these results we see that putting the KV cache on the network didn’t have the latency penalty I thought it would. It ran as fast as physics allows 🔥. Even if the disk path gets some love next month and starts doing 1.5 GB/s, using the network is still a solid choice. Plus, Valkey has room to go even faster with a bigger NIC.

There is one thing I can’t explain. That 2.18x speedup for the disk is a median, and the per-document numbers show something weird. The first document requested in the cached pass comes back around 11x faster. The second comes back around 3.5x. The other 28 are all right at 2.1x. Oddly enough, both runs were basically the same:

Something is warm for the first request or two, then stops. I don’t know what it is, and I’m not going to make up a reason that sounds good. Whatever it’s doing, it makes the disk look better than the median I’m quoting, so it isn’t driving my conclusion.

Let’s talk about DRAM

In my results table above, the DRAM row says it had a 0.95x “speedup”. I know what you’re thinking: I misconfigured it. I didn’t. Remember that two-document sanity check where hits came back in 82 ms? That was this tier with the same config. The TTFT was 3122.8 ms cold and 82.5 ms cached, which is a 37.93x speedup 🤯.

DRAM isn’t slow. DRAM is the fastest thing in the entire stack by a mile. It just… runs out.

Thirty documents is 300,000 tokens of KV. At 57,344 bytes per token, that’s 16 GiB. I gave the tier 8 GB, and the entire host only has 16. The box simply didn’t have the memory, and it evicted everything before I could even run the second pass.

Not only that, but every tier that served nothing came back 3% to 5% slower than cold, on at least 29 of the 30 documents. A lookup that misses is still a lookup, and a store that’s evicting still costs a store. In this instance, the undersized cache was basically a tax.

And thirty documents is nothing when compared to a real production corpus. If your working set is two documents, use DRAM and stop reading this. If it’s thirty or more, skip the DRAM for something with more capacity.

Expanding to two machines

Everything we’ve covered so far has been a single machine writing and then reading its own cache. But the big question is what happens when a request is routed somewhere else, because it will in production. So I ran the cold pass on gpu-a and the cached pass on gpu-b (the one that didn’t do the prefill).

Unsurprisingly, NVMe went down to zero reuse and 3.2 second prefill on gpu-b because the KV didn’t exist there and there was nothing to access. It makes NVMe seem a lot less exciting once you start using it in a distributed environment.

That said, Valkey went from 5.98x to 5.97x. Reading from a machine that did not compute the KV cost nothing measurable. The worst document slipped from 5.55x to 3.92x, and the medians are effectively the same. The cache doesn’t belong to either node. And that was the whole point.

My takeaways

If your working set fits in host DRAM, use that. A cache hit came back 37x faster than recomputing the prefill. Just be honest with yourself about capacity. Measure your corpus in tokens and multiply by your model’s per-token KV. Don’t eyeball it.

Once it stops fitting, local disk gets you 2.18x on the machine that wrote it, but nothing anywhere else. On a single node that’s a considerable boost. But across a fleet, every entry is isolated to the box that produced it, and your hit rate becomes the odds of getting routed back to the same machine.

The network tier isn’t free on the way in. Cold TTFT ran about 110 to 150 ms slower with Valkey than with DRAM, which is a 3% to 4% cost for storing over the wire instead of in memory. So the tradeoff is that you pay 110-150ms per cache fill in exchange for about 2.7 seconds back on every hit after it.

The most surprising result was that the shared tier was already the faster one here. On this hardware, with this model, at this corpus size, putting the cache on another machine made it quicker. I went into this assuming I’d be arguing that sharing is worth a latency cost. I am happy to have been wrong on that one. Skipping prefill is worth what it costs (and then some).

Your KV cache belongs on the network. Not because it has to be shared (though it does). Because that’s where it was fastest.

Try it yourself

The harness, the 30-document corpus, the tier configs and the infrastructure scripts are all in valkey-kvcache-bench. One command provisions all three hosts, runs the matrix, pulls the results down and terminates everything:

cd infra
.\Run-TierBench.ps1 -AwsProfile <profile> -Teardown

It costs about $2.14 an hour and the full benchmark run takes under an hour.

One warning if you try this yourself - the first time I ran the two-machine test, every tier came back 0/30 (including Valkey). It turns out LMCache’s default cache key hash is Python’s hash(), which gets a different random salt in every process. So gpu-a and gpu-b were generating different keys for identical tokens, which made cross-node reuse impossible. LMCache even warns you at startup with a Using builtin hash without PYTHONHASHSEED set message, and I had completely missed it. But you can fix it by setting this config value:

​yaml pre_caching_hash_algorithm: sha256_cbor_64bit ​

If your shared KV cache ever shows a zero hit rate across nodes, it’s probably that.

Anyway, try it out and if you see something different, please feel free to reach out so we can compare.

Happy coding!

Momento Cache is built for fast-moving teams who want optimal Valkey performance without the hassle of babysitting infrastructure. Cluster capacity and Flex capacity bring single-tenant, fully-managed resources to this self-service infrastructure platform.

Cluster provides direct control over instance type, shards, replicas, and availability zones. Flex automatically optimizes resources within specified bounds. Both were designed for workloads that need stronger isolation, more control, and a higher performance ceiling than Momento Cache’s Serverless configuration.

Avoiding the trap of operational creep

In the AI era, it’s trivial to stand up basic infrastructure at near-zero cost. We’ve all been there: it’s fast, it’s easy, it mostly works. It’s the right solution when you need to ship.

Then, growth hits. Traffic changes shape. Memory fills unevenly. A shard needs to move. A primary fails. Clients stampede. A zero-day patch lands at midnight. Suddenly, operational creep has consumed the time and the token budget that you wanted to spend on building.

And the problems only multiply as you add more features, products, and services. Soon your entire team is stuck fighting against infrastructure as latency, cost, and complexity steadily creep up and to the right.

Fast, reliable, efficient - pick all three

The Momento platform powers critical features for millions of users around the world at companies like Capcom, Coinbase, Paramount, and Snap. Now, Momento Cache puts the full power and flexibility of Valkey at your fingertips, packaging up hard-won production lessons into a streamlined service.

The new Cluster and Flex configurations help you to tailor a Valkey deployment to fit your specific needs. Momento seamlessly operates the lifecycle behind that system: provisioning, health, failover, rolling topology changes, version upgrades, and security patches.

Whether you’re pushing one thousand or one million requests per second, Momento delivers unrivaled performance and resource utilization. As you grow, your infrastructure grows alongside you. For companies with a mature platform org, Momento Cache can even be deployed in a BYOC configuration, as part of your internal developer platform.

Transparent pricing

Momento Cache pricing is designed to be simple, predictable, and cost-effective. Pricing in us-east-1:

• Flex starts at $13 per GB-month of physical Valkey storage
• Cluster applies a 25% surcharge to the list price for every deployed instance
• Data transfer includes 200 GB of ingress + egress each month, then $0.05/GB

Momento Cache supports rapid autoscaling within a specified capacity range, making it easy to reduce the cost of idle resources.

Designing for scale

Momento Cache employs a two-tiered architecture that significantly improves reliability and efficiency at scale. Each valkey cluster sits behind a gateway that handles the hard traffic problems like hot keys and connection storms before they hit the data layer.

The gateway exposes a RESP endpoint, so Momento Cache is compatible with all standard Valkey and Redis clients. It masks the underlying cluster, presenting a single stable endpoint across any topology changes.

block
  app("Redis or Valkey\nclient"):3
  space
  gateway("gateway"):3
  space
  pool("Valkey cluster"):3

  app --> gateway
  gateway --> pool

The gateway is optimized to quickly process TLS, auth, rate limits, and other traffic management concerns at high concurrency and high throughput. It multiplexes client traffic across a pool of warm connections to the Valkey cluster. This reduces connection latency, and enables advanced capabilities like request coalescing.

The result is an efficient system with deliberate separation of responsibilities. The gateway absorbs traffic concerns, while Valkey nodes focus their resources on handling data.

Get started with Momento Cache

Try out Momento Cache in a few short steps with the Momento CLI. While the service is still in preview, you’ll also need to request access via the web console.

First, create an API key in the Momento console, copy the endpoint for your region, and configure the default CLI profile:

# paste the api key and endpoint when prompted
momento configure

Then, create a Capacity Pool. Be sure to provide valid zone IDs for your region:

momento preview pool create \
  --name example-pool \
  --capacity-gib 32..128 \
  --replicas-per-shard 1..2 \
  --zones use1-az1,use1-az2

momento preview pool describe --name example-pool

Once the Capacity Pool status is active, you can create a Database:

momento preview database create \
  --name example-db \
  --pool-name example-pool

Back in the console, open the pool’s Databases tab and copy the regional RESP endpoint. Connect any Valkey or Redis client in standalone mode to this endpoint over TLS on port 6379. The Database name is the username, and your Momento API key or token is the password. Here, we’ll demonstrate with the official valkey cli:

valkey-cli -h <resp-endpoint> -p 6379 --tls \
  --user example-db --pass <momento-api-key>

> SET example-key "ready"
OK
> GET example-key
"ready"

Congratulations! You now have a high-performance Valkey cluster ready to go.

Next, check out the docs to learn more about Momento Cache’s capabilities. Or, if you want to push your cache to the limit, load up cachecannon on a c7g.xlarge instance in the same region and zone as your Database!

Up next

This launch begins the next chapter for Momento Cache. Stay tuned as we port more features from our enterprise services into Momento Cache, including VPC peering and S3 integration!

We’re looking for feedback from teams operating demanding Valkey workloads as we refine the product. If you’re building a fast-growing product, operating a large Valkey or Redis cluster, designing an internal caching platform, or helping teams adopt Valkey, we would love to hear what you need next and where we can help out.

We’re grateful to the engineers and partners who turned years of demanding operating experience into a service anyone can start using today. Special thanks to Dylan Abraham and Jason LaPier for leading the development effort!

But doubling the number of indexes I was using made me nervous. I didn’t know what impact that would have on performance. I’m using Valkey because I needed ultra-low latency, I didn’t want to shoot myself in the foot trying to be clever.

And yes, going from one to two indexes is probably not a big deal. But indexes have a way of piling up. You create one to filter products by category, then add price to it a month later, then the search team wants a vector field, then the payment team indexes their keys, then fulfillment indexes theirs. Before you know it, there are a dozen FT.CREATE statements in the repo with no single owner.

My instinct tells me that performance scales inversely with the count. Every index in valkey-search subscribes to a key prefix. When you write a matching key, that index gets an entry in a mutation queue, and your client stays blocked until every entry is processed. That’s what gives you read-after-write consistency. If you have a dozen indexes, you also have a dozen entries, and a dozen times the wait. Right?

So I built a benchmark to see how much additional indexes cost in valkey-search. And I discovered I was very wrong.

The setup

The server was an r7i.4xlarge with 8 physical cores, running Valkey 9.1.1 and valkey-search 1.2.1. valkey-search sized its own writer pool to 8 threads. I turned persistence off, because I didn’t need a background save forking mid-run throwing a wrench into my latency numbers. Load came from a separate c7i.8xlarge in the same placement group, over 384 connections.

Every write is the same 4.6KB product hash that includes a category, a price, a SKU, a title, a description, and a 1024-dimension embedding. The load generator sends on a schedule and times each request from when it was supposed to go out, not when it managed to (more on this later). Every point runs for 25 seconds, three times, and I capture the median.

Do idle indexes cost anything?

As I mentioned earlier, indexes subscribe to a key prefix. If a key is upserted that doesn’t match on an index, does that slow things down? In other words, my test here determines if the existence of indexes slows down the performance of others.

I ran the same write workload against product: keys in two setups: one index on product:, and another one with the same index plus fifty more indexes on unrelated prefixes.

Nice! I couldn’t measure a difference between them at any rate I tried.

This is because the prefix subscriptions live in a trie. When you write product:88213, valkey-search walks the trie and only notifies the indexes whose prefix matches. An index subscribed to orders: doesn’t hear about it. FT.INFO idx:other0 reports a field called mutation_queue_size, and across the entire run it never left zero, which validates the promise of the trie architecture.

So that hodgepodge of indexes in your repo isn’t what’s slowing your writes down. Only indexes whose prefix matches the keys you write are in the path at all.

The impact of matching indexes

So what’s the performance impact of having multiple matching indexes? I ran the same workload again, this time adding multiple identical indexes directly on the product: prefix.

Every matching index gets an indexing job, and the write isn’t done until all of them are. To figure this out, I took Valkey’s search_ingest_hash_keys counter and divided by the writes that completed in the same window. I took this number and multiplied it by the write rate to calculate the indexing jobs/sec column.

The expensive part, naturally, is switching indexing on in the first place. Going from zero indexes to one cost me 58% of my write rate. The second index cost another 41%. Doubling from two to four only cost 16%. Each additional index cost less than the previous one.

Adding indexes pulls more total indexing work per second out of the same server, 33,636 jobs a second at one index and 67,272 at four. So one or two indexes clearly weren’t saturating the writer pool, because it eventually went on to do twice the work.

Indexing jobs fan out across a pool sized to your physical core count, and the blocked-client handles collapse into a single block on your connection. The write waits for the slowest single job, which is why four indexes don’t cost four times what one does.

Unfortunately I don’t know what the limiting factor is. My hunch is that it’s the per-index bookkeeping that happens on the main thread before a job reaches the pool, but I didn’t measure that, so 🤷.

NOTE - Adding matching indexes won’t appear to add latency until it’s too late. At 2,000 writes/sec, one index and four indexes were within a millisecond of each other. You won’t catch it watching p99 on a healthy system, because what you’re spending is headroom. You find out it’s gone when you need it.

Vector fields hit different

The benchmarks above use TAG and NUMERIC fields, just the normal filter field types. So I went back to the first benchmark, added a single 1024-dimension HNSW vector field to the one-index setup, and re-ran it to find some staggering results.

32,000 writes per second became 2,828. 🤯

That’s an 11x difference in throughput because of a single field. My napkin math says that’s 2-3 milliseconds of CPU per vector insert spread across eight writer threads. To make matters worse, the performance cliffs. I increased the rate on my benchmark by ~40%, and the wheels fell off.

The performance hit goes from five milliseconds to two seconds. The queue went from basically empty to 376 entries deep, and it stayed there for every rate I tried above that. You’re either under the line and fine, or over it and everything is late. If you’re capacity planning, be sure to check whether an index on your hot prefix has a vector field.

Do vector fields affect other matching indexes?

Back to the search feature I was building that required two indexes. A single index can’t do both jobs because of stop words. The text pipeline is configured for the entire index, and it splits words on punctuation before dropping anything in the stop word list. So IT-500 becomes it and 500, it is a stop word, so just 500 is added to the index. Turning on NOSTOPWORDS means your description field will index every the, is, and, and it (plus a lot more) in the catalog. So we need two indexes to have it turned on for one and off for the other.

But that made me wonder what the second index costs when the first one has a vector field. We saw how much of a hit it made to throughput in our earlier benchmarks.

About a three percent difference. The exact match index processed 141,408 text fields during the benchmark run. Both mutations hit the pool together, the write waits for the slower one (the vector). So if you’re already paying the HNSW tax, the lexical index has essentially no additional latency.

My takeaways

So it turns out I was asking the wrong question when I started this experiment. I thought the number of indexes I had was going to slow performance down to a crawl. But it doesn’t. The real question is what fields are inside the indexes that match your keys? Which is a relief, when I think about it.

I also learned a couple of things about benchmarking while I was busy answering the wrong question. 😅

Every sustained writes/sec number in this post could have been bigger. At the top rung of my ladder, the no-index setup completed 128,000 writes a second (but my tables show 80,000). At that run rate, it was making every request wait 1.9 seconds in a queue first. A server running at full utilization drains as fast as it fills, so throughput looks perfect, but at a cost to latency. So the number I used in every table is the highest rate where p99 stayed within reason.

It took me three runs to believe what I saw in that four-row table early in this post. The first run stepped the rate by 1.4x per rung, which put two indexes and four indexes on the same 16,000 rung. Which at first made me think indexes three and four had no performance implications. But in reality, they had both fallen apart somewhere between rungs and the ladder couldn’t show me where. So I re-ran it with 1.19x steps and got a cleaner separation. Then I noticed that ladder started at 14,000, which is already well up the curve, so it never measured what latency looks like when the server is idle. My rule for picking a sustainable rate is relative to that idle number, which meant the second run was grading itself on a curve. The third run started at 2,000 and is the one in the table. A rate ladder can’t resolve a difference smaller than its own step, and it can’t tell you where the knee is if it never saw the flat part before it.

Try it yourself

I have the benchmark, scripts, and results available in GitHub. If you want to check my numbers (or disagree with them!), please do and let me know what you find.

If you want to run the same tests on your own Valkey cluster, you can run these three commands:

FT.INFO <index> # shows mutation_queue_size (write backlog) for the specified index
INFO search # shows search_writer_queue_size for the whole pool
CONFIG SET search.info-developer-visible yes # unlocks per-field-type counters like search_ingest_field_vector

It’s cheap to experiment with your existing clusters because these mutations are reversible. FT.DROPINDEX is instant and your data was never in the index to begin with. Add a shape, measure it, then discard it.

Happy coding!

Unfortunately it’s not that simple. Memories need to be similar, yes, but they also need to be recent. You don’t want memories from a year ago influencing your agent. And outcome is important too. If one approach worked and another failed, you probably want the successful one steering behavior.

I ran into this while building a small demo that puts Valkey Search inside an agent’s inference loop. Every task the agent finishes gets written as a memory. It persists the task text, the approach that worked, whether it succeeded, when it happened, and a vector of the task. Next time a similar task shows up, the agent recalls what worked instead of rediscovering it.

I ended up learning how valkey-search handles this the hard way, because my first FT.SEARCH call did not work the way I expected:

FT.SEARCH idx:memory "(@outcome:{success} @created_at:[1782900000 +inf])=>[KNN 3 @vector $vec]" PARAMS 2 vec <query-vector> DIALECT 2

Left of the => is an ordinary boolean filter of a tag and a numeric range over two fields I indexed alongside the embedding. Right of it is the vector search.

I thought Valkey was going to find the nearest vectors and apply the tag and timestamp filters afterward. But it doesn’t work like that (in a good way). The filter criteria are actually inputs to the query planner. Depending on how many memories they match, Valkey chooses a different algorithm to search the vector index. In other words, adding a filter changes how the search runs, which meant I needed to rethink my initial memory schema. Turns out my retrieval policy was also an index-design decision.

Nobody post-filters anymore

If you’ve used Pinecone, Qdrant, Weaviate, or Milvus, then this should be familiar. All of them decide at query time whether to walk the graph with your filter applied or abandon the graph and brute-force the matching subset instead. Pinecone calls it single-stage filtering. Qdrant calls it query planning. Weaviate calls it a flat search cutoff. Milvus doesn’t really call it anything, it just does it 😂. valkey-search is the same idea, and on the FT.SEARCH vector path it doesn’t implement post-filtering at all.

If you’re a pgvector user, however, filtering happens after the index scan. With the default hnsw.ef_search of 40, a filter that keeps roughly 10% of those candidates might leave you with only 4 results. Iterative index scans were added in 0.8.0 to make that a little better, and they’re off by default.

valkey-search calls it a query planner. It estimates how many keys your filter matches and picks one of two algorithms to perform the search.

If the estimate is small compared to the index, it pre-filters results by walking the qualified key set, computing each distance directly, and keeping a top-k heap. The HNSW graph is never traversed. So valkey-search is essentially doing a brute-force scan over a tiny set.

If the estimate is large, it performs an inline filter. Your predicate is handed to hnswlib as an isIdAllowed functor and evaluated during traversal of the base layer. Non-matching nodes are still visited and expanded, they just don’t get added to the result set.

The cutoff point for one algorithm vs the other is 0.001. So if the number of estimated matching keys is at most .1% of the number of vectors in the index, it will go the brute-force route. Otherwise it uses the inline filter.

For reference, the cutoff point for Milvus is around 7%, which makes valkey-search about 70x stricter. You end up on the inline path more often than you’d think.

Make the filter disappear

On my demo index with a few hundred memories, the threshold works out to well under 1 key, so every recall that matches anything takes the inline path. You’d never notice either way at that scale.

But what would happen with the same schema in production with 1,000,000 memories? The threshold is 1,000 keys. The planner considers the selectiveness of the filter. If @outcome:{success} matches 70% of the index and your @created_at window is also broad, you’re squarely in the inline path. And the planner is right to put you there. If you scope recall to a single tenant with 400 memories, you drop under the threshold, and the query becomes an exact scan. Perfect, fast recall because 400 distance computations is nothing.

Let’s make it more difficult. A filter matching 1% of a large index sits 10x above the cutoff point, so it takes the inline path, where HNSW traverses a significant number of candidates for every one it’s allowed to keep. That results in a lot of extra latency you didn’t account for. And you can’t change that by tuning it, because search.prefiltering-threshold-ratio is immutable unless search.debug-mode is on, and it isn’t in the public configurables table at all.

You can check which path you’re actually on, by the way. valkey-search counts both, named search_prefiltering_requests_count and search_inline_filtering_requests_count. They’re module fields, so you need INFO SEARCH and not plain INFO. You can run your recall query a hundred times to see which one moves.

So your best bet is the index itself. Make the filter act like a namespace. Put a hash tag in the index name, prefix the keys to match, and each tenant’s recall hits one shard against a small index where the filter is mostly irrelevant. Valkey enforces it in both directions, too. A tagged index name requires every prefix to carry the same tag, and an untagged one requires that none of them do. Be careful with this though, because it’s not easy to undo if you change your mind since there’s no FT.ALTER here.

Forgetting is expensive

Removing a vector from an HNSW index calls markDelete and returns. The node isn’t removed from the graph, it’s still visited, it still routes other searches through itself, it just fails the deleted check. search.hnsw-allow-replace-deleted would let the next insert reuse that slot, but it’s false by default and more of a non-production flag. In production the space is stranded until you drop the index.

Luckily, updates are a different story. Modifying an indexed vector routes to an in-place update under the existing label, so the node keeps its slot and just gets its links rewired. Writing a hash field that isn’t the vector doesn’t mess with the graph, and rewriting the vector with identical bytes short-circuits before it gets there.

So updates are cheap, but deletion is where it gets expensive. That includes DEL, eviction, and expiry.

That’s a little scary, because expiry is the recency policy. valkey-search subscribes to generic, expired, and evicted keyspace notifications, so a TTL’d memory really does leave the vector index when the key goes away. Which is what you want, but it’s also what makes graph nodes stranded.

Which means you have to change how you key your memories. Creating a new key every run feels like a safe and reasonable default, and it’s what my demo does with its memory:<id> per completed task. But add a TTL to keep things fresh, and every expiring key leaves a node behind. That’s an expensive default at scale. Instead, use a stable key derived from a task fingerprint, and update it in place as the agent learns more about that kind of task. This means the task costs only one node instead of one per attempt.

There are two recency controls here to consider. The @created_at range decides what you’re willing to believe on any given query, and it doesn’t delete anything. The TTL decides what you’re willing to pay to store. Range should be your first lever and the TTL should be the slower fallback.

Decide before you have data

This demo surprised me twice: the filter I wrote as a query detail decides which algorithm runs, and the TTL I (almost) added to keep memories fresh led to stranding the index.

Everything here is discoverable, at least. The planner code is a short function to read and understand. The filtering comes from hnswlib. Even the deleted node behavior is a comment in the source, and the counters are available in INFO SEARCH. You don’t have to take my word for any of it (but you should 😜).

What you can’t read your way out of is when you have to make decisions. How the index is scoped and how a memory is keyed are day-one calls, made before you have a single memory to check them against. Outside of that dev-only flag, a rebuild is the only way to reclaim space that deletion stranded. Get those wrong and you’re reindexing.

The demo is on GitHub if you want somewhere to start. docker compose up -d gets you Valkey with the search module and an agent that writes its own memories. Run a few tasks through it, then look at INFO SEARCH and find out which path your queries are actually taking.

Happy coding!

If you came here looking for a post from me on consistency models, I have to disappoint you. Today, I want to talk about the value of consistency in life. In life, effort is additive, but consistency is multiplicative.

Over the last few years, I have had the honor of watching the Valkey project blossom from an idea into an inspirational, community-driven effort with consistent improvements in each release. These improvements range from memory efficiency to availability at scale to substantial performance gains. These small improvements compound.

This compounding effect and the power of a driven community is perhaps best illustrated by the journey of the I/O-threading architecture and its impact on large objects from 1 MB to 64 MB in Valkey. Larger items are becoming increasingly important for inference KV caches, 4K video streaming, and other enticing use cases. At the very least, the impact of large objects should not be overlooked, as they can impact everyone else’s latency; this post measures how fast the large objects themselves go.

Buckle up, because it’s about to get spicy.

A brief history of I/O threads

Valkey 7.2 inherited an I/O-thread model from a software stack doing its best to stay single-threaded. The main thread and I/O threads worked in coordinated phases, separated by synchronization barriers, and the 7.2 config file is candid about the result: “Usually threading reads doesn’t help much.”

Valkey 8.0 delivered a fundamental rearchitecture of I/O threads, tripling throughput to over a million requests per second. This idea was not new. In August 2023, seven months before the fork, Dan Touitou filed redis#12489, laying out exactly this design in detail, benchmarks included.

Redis let #12489 sit. The issue is still open in the tracker today, unassigned and without a milestone.

Within days of the fork, the Valkey community copied the proposal verbatim into issue #22, greeted it as “a true gem,” and shipped it. The new architecture enabled continuously running I/O threads connected by queues, so reads, parses, and writes proceed on separate cores while the main thread executes commands. Valkey 8.1 delivered TLS handshake offload to I/O threads in #1338.

Redis shipped a strikingly similar asynchronous I/O-threading model in Redis 8 in May 2025, a year after the fork and 21 months after the design landed in its own tracker. Around the same time, we put Valkey 8.1 and Redis 8.0 head to head on small objects. Valkey 8.1 outran Redis 8.0 by 37% on writes and 16% on reads.

Valkey 9.0 brought reply copy avoidance, changing the performance of large items entirely. Before this change, the main thread copied the entire object into a connection reply buffer before moving to the next command. While a large item is being copied, the entire pipeline stalls. Small objects are not serviced until the copy finishes.

Valkey 9.0 instead passes a reference to the I/O threads and keeps the object alive with a reference count. The I/O worker for that connection hands the object’s memory directly to writev(). This shortens the handoff to the I/O threads and gets the main thread back to handling requests.

Valkey 9.1 then redesigned communication between the main thread and I/O threads around lock-free queues, credited in the release notes with an 8-17% throughput gain.

Based on these changes, we expected 8.0 to lift reads and writes for smaller items but still struggle to fill the network link for larger items. We expected 9.0 to bring GETs to line rate and 9.1 to improve writes.

This is what open-source competition buys everyone, including teams that never leave Redis. A performance design that sat for seven months as an unassigned issue became table stakes for both projects within two years of being filed.

Show me the numbers

We swept values from 1 MB to 64 MB across four Valkey releases on two nodes with 200 Gbps of bandwidth between them. The full setup is below.

Valkey 7.2: reads capped at 35 Gbps, writes at 55

An 8 MB value delivers 21 Gbps on reads and 23 Gbps on writes, with p99 latencies of 174 and 164 ms. Turning on io-threads-do-reads moved only the 1 MB read cell.

Valkey 8.0 and 8.1: writes take off, large reads stay put

The threading rebuild lifts our 8 MB write from 23 to 137 Gbps, six times faster, and 1 MB reads reach 183 Gbps. Larger reads settle at 30-33 Gbps whether the value is 8 MB or 64 MB. That flat floor points to a serial, per-byte bottleneck. Valkey 8.0.9 and 8.1.8 measured the same at every size, so one line carries both.

Valkey 9.0: large GETs jump to line rate

Every size from 1 MB to 64 MB reads at 190-201 Gbps. The 8 MB p99 falls from 112 to 24 ms, and a 64 MB read drops from just under a second to 230 ms. Writes do not move because the ingest path was never copy-bound. That ceiling waits for 9.1.

Valkey 9.1: more headroom for writes

With reads pinned at the network limit, the gain surfaces on writes. The 8 MB write rises from 134 to 166 Gbps, a 24% gain, while values from 12 MB to 64 MB gain 11-19%.

The short run held

To make sure the 15-second runs were not catching a lucky window, I reran every 8 MB GET and SET cell for 15 minutes. Valkey 9.1 is a good example: GET held 200.8 Gbps and SET held 165.5 Gbps, right on top of the original 201 and 166 Gbps results. Nothing sagged as the runs went on.

Detailed results

Reads (GET)

GET-only, 32 connections, 100% hit rate.

Writes (SET)

SET-only, 32 connections.

The setup

Machines. Two c8gn.16xlarge instances with Graviton4 and 200 Gbps networking in the same availability zone and cluster placement group, running Amazon Linux 2023.

Server. We tested the official valkey/valkey Docker image at versions 7.2.13, 8.1.8, 9.0.4, and 9.1.0. We also measured 8.0.9. It matched 8.1.8 within run-to-run noise at every size, so the tables show 8.1 as the 8.x column. Valkey 7.2 ran with the same flags as the newer versions. A control with io-threads-do-reads yes changed only the 1 MB GET cell, from 32 to 41 Gbps. Each version ran in a fresh container with host networking:

docker run --network host --cpuset-cpus 8-23 \
  --ulimit nofile=32768:65536 valkey/valkey:<version> \
  --save '' --appendonly no --io-threads 16 \
  --protected-mode no --maxmemory 60gb

Persistence was off. The 16 threads in Valkey’s io-threads count were one main thread plus 15 I/O workers. The process was pinned to cores 8-23 so it never fought the kernel for the cores doing network interrupt work.

Interrupts. irqbalance was off on both machines. The ENA NIC was configured with four combined queues and their IRQs pinned to cores 0-3. Without this step, results wander from run to run as the kernel shuffles interrupts onto whatever cores the server or client threads happen to be using. If you benchmark at these speeds, pin your IRQs first and thank yourself later.

# Run on both machines. ens50 is the ENA interface name on these instances.
sudo systemctl stop irqbalance
sudo ethtool -L ens50 combined 4
i=0
for irq in $(grep ens50 /proc/interrupts | awk '{print $1}' | tr -d ':'); do
  echo $((i%4)) | sudo tee /proc/irq/$irq/smp_affinity_list > /dev/null
  i=$((i+1))
done

Client. We used valkey-lab, built on cachecannon, with 16 worker threads pinned to cores 4-19, 32 connections, and pipeline depth 1. Reads and writes were measured in separate passes. Read passes prefilled the keyspace and ran at a 100% hit rate. Each short-run cell was a 15-second measurement after a five-second warmup, over a 500-key keyspace with 16-byte keys.

We kept concurrency at 32 connections because AWS caps a single TCP flow at roughly 9.5 Gbps. We verified 9.53 Gbps with iperf3. Saturating a 200 Gbps NIC requires spreading the load across flows.

One invocation per cell, with -s swept across the value sizes:

# GET pass: prefill the 500-key keyspace, then measure at a 100% hit rate.
valkey-lab -h <server> -c 32 -P 1 -t 16 --cpu-list 4-19 \
  -n 500 -r 100:0 --prefill --warmup 5s -d 15s \
  -s <value_bytes> --key-size 16

# SET pass.
valkey-lab -h <server> -c 32 -P 1 -t 16 --cpu-list 4-19 \
  -n 500 -r 0:100 --warmup 5s -d 15s \
  -s <value_bytes> --key-size 16

Scope of the result. This is a single-node, no-TLS, persistence-disabled test with 32 closed-loop connections, pipeline depth 1, and a 100% hit rate. Most table cells are one 15-second measurement after warmup. The results describe this rig and workload. Broader production claims need repeated runs and additional configurations.

We benchmarked whole releases rather than individual changes, so we treat the pull requests named in this post as leading explanations rather than proof of causality.

What this means if you run Valkey

Valkey 9.x delivers roughly 6x the 64 MB read bandwidth of 8.x while cutting p99 from just under a second to about 230 ms on this single-node, no-TLS test. Our earlier mixed-workload test also found far less collateral latency for small requests when a large read arrived.

The practical takeaway is narrower than “Valkey is always faster.” Valkey 8.1 is essentially flat on this workload, Valkey 9.0 changes the large-GET path, and Valkey 9.1 improves large SETs on this rig. If large values matter to your workload, test 9.x with your object-size distribution, concurrency, TLS, and persistence settings rather than extrapolating from a small-object benchmark, or from this one.

Valkey has consistently improved performance, memory efficiency, and availability at scale. Each version brings about a new set of improvements driven by issues faced by real users in production. A vibrant community where nobody is incentivized to withhold features for the sake of revenue and everyone is incentivized to chase continuous improvement is what makes Valkey truly special. Effort is additive. Consistency is multiplicative.

Serving megabyte-sized objects at wire speed is what I’ve been spending a lot of time on lately. If you are wrangling larger objects, let’s talk. Join me on Valkey Slack.