Distributed Caches in Infinispan
August 23, 2025Infinispan is an open-source in-memory data grid and forms the core of Red Hat Data Grid. It can also serve as a cache server for KeyCloak, which provides single sign-on and identity brokering.
The official documentation describes Infinispan as a drop-in replacement for Redis or Memcached. It also implements JSR107, which allows Java clients to interact with an Infinispan cluster via the JCache API. For example, a client can use a CacheManager to obtain a Cache—a map-like data structure—and perform updates or lookups.
Although it is possible to run an Infinispan node in the same JVM as the client, the KeyCloak documentation recommends running clusters and clients on separate servers. Keeping them separate makes it easier to scale both independently and improves availability.
Infinispan supports four cache modes: local, replicated, distributed, and invalidation. When clusters and clients run on different servers, only replicated or distributed caches are suitable. Invalidation caches are not recommended in this case.
A replicated cache stores every entry on all nodes, while a distributed cache stores each entry on a subset of nodes. Because the number of replication requests grows linearly with the number of nodes, the documentation recommends distributed caches when you have ten nodes or more. In short, for both high availability and performance, clusters should run on separate servers and use distributed caches.
Node assignment for cache entries is determined using a consistent hashing algorithm. More precisely, an implementation of ConsistentHashFactory decides how keys are mapped to nodes. The original consistent hashing algorithm maps keys and nodes into a unit interval, assigning each key to its successor node. This balances load and minimizes redistribution. See this post for details on the original algorithm.
Infinispan’s ConsistentHashFactory extends that idea to support replication and transactions, which the original algorithm does not address.
The group of nodes that stores a given entry is called a segment.
The first node in the segment is the primary owner, while the others serve as backups.
Writes are always directed to the primary owner, which stores the entry and then replicates it to the backups.
Thus, ConsistentHashFactory is responsible not only for distributing entries but also for selecting primary owners and backup nodes.
By default, Infinispan uses SyncConsistentHashFactory.
The implementation in version 15.2.5-final exposes a create method with the following parameters:
numOwners: the number of nodes that should hold each entrynumSegments: the total number of segmentsmembers: the list of cluster nodescapacityFactors: a weight per node that influences how many segments it receives
A segment is defined as a list of numOwners nodes, with the primary owner listed first.
SyncConsistentHashFactory creates numSegments such segments.
Each key is hashed and mapped to the segment whose hash is closest, ensuring balanced distribution.
By varying the node composition and order per segment, the algorithm achieves load balancing, primary assignment, and replication.
Segment placement also considers the distance between node hashes, while capacityFactors can be used to skew assignments based on node performance.
Notably, numOwners can be adjusted at runtime, but numSegments is fixed.
The hash space spans all non-negative long values, from \(0\) to \(2^{63}-1\).
Segments are placed at evenly spaced intervals (Long.MAX_VALUE / numSegments), with the first segment starting at Long.MAX_VALUE / (2 * numSegments).
This allocation is implemented in computeSegmentHashes:
private long[] computeSegmentHashes(int numSegments) {
assert segmentSize != 0;
long[] segmentHashes = new long[numSegments];
long currentSegmentHash = segmentSize >> 1;
for (int s = 0; s < numSegments; s++) {
segmentHashes[s] = currentSegmentHash;
currentSegmentHash += segmentSize;
}
return segmentHashes;
}
Keys are hashed using MurmurHash3. The getSegmentForHash method maps the MurmurHash3 result to a segment by dividing the lower 64 bits by the segment size:
public int getSegmentForHash(int hash) {
// Ensure the result is always positive
return (hash & Integer.MAX_VALUE) / segmentSize;
}
Nodes are also hashed with MurmurHash3, but through the use of virtual nodes. Each node is assigned \(\lceil \log_2(\text{numSegments}-1) \rceil\) virtual nodes. The \(i\)-th virtual node’s hash is computed from the node’s UUID with \(i\) as the seed.
Primary owners are assigned first, followed by backups. The distance between a segment and a node is defined as the scaled distance between the segment’s hash and the nearest virtual node hash, adjusted by the node’s capacity relative to the minimum capacity. Hashes are treated as circular, so the maximum and minimum values are neighbors. Backups are then selected by inserting all nodes into a priority queue, ordered by scaled distance, and dequeuing \(\texttt{numOwners} - 1\) of them.