Matching-engine failover: hot-hot replicas on one tape
A streaming platform that survives node loss is table stakes. The harder question is what happens to the application running on top of it. If you run an exchange on AlgoX2 and the matching engine itself crashes, the platform staying up is not much comfort.
The answer is not obvious, so it helps to start one layer down: with how the matching engine attaches to the platform in the first place.
The matching engine lives inside the cluster
On AlgoX2, one append-only stream carries the entire input to the matching engine: every incoming order, interleaved in one sequence. The matcher’s outputs go onto the same tape.
The matching engine does not sit outside the platform as an ordinary publisher and subscriber. If it did, failover would be your problem again, and the point of AlgoX2 is to take that problem away rather than hand you a new one. Instead, the matching engine runs as a user process inside the cluster, spawned on one of its nodes.
It talks to the host node over a shared-memory transport that speaks the same protocol AlgoX2 uses on the outside. Message latency between the process and the host node is on the order of 100 nanoseconds. None of this shows in the application code: a library attaches to the process and handles the I/O.
So the matching engine is receiving orders, performing matches and writing out trades. Then it crashes.
Checkpoints and automatic restart
The first line of defence is one AlgoX2 offers every user process: automatic restart from a checkpoint.
A process writes a checkpoint as another set of streaming data, then adds a soft link to it in the AlgoX2 file system. When the process dies, the platform restarts it automatically, with a limit on the number of restarts, and on another node if its own node is gone. The restarted process gets the same subscriptions to the same streams, starting from the last checkpoint. None of that needs code in the application.
Why 30 seconds is fatal
For most applications, that is enough. For an exchange, it is not. Loading a checkpoint and restarting takes time, say 30 seconds, and an exchange cannot afford any gap at all.
The reason is the money resting inside the matching engine: orders that have been placed and not yet filled. While the exchange stalls, the market moves elsewhere. The resting orders go stale, high-frequency traders pick them off, and people lose money. An exchange that is slow, or whose latency spikes unpredictably, is a sitting duck. For an exchange, low and predictable latency is a matter of survival.
Determinism makes a second copy possible
The way out starts with one property of the matching engine: it is deterministic. Given the same input twice, it produces the same output twice.
That single property is what lets two copies run side by side. How to build an exchange explains why a matching engine written as a deterministic function of one ordered log is the right design to begin with.
Hot-hot: two replicas, one tape
With a deterministic matching engine, you can run two of them on AlgoX2. You spawn them with a replica flag, replica A and replica B, which is a command-line switch. Both get the same starting point and the same output location. They compute the same transform on the same input, so they emit the same output.
That output is not interleaved. It is deduplicated by sequence number. One replica’s answer arrives first and takes the sequence number; the other’s arrives second and is dropped as a duplicate. The faster answer lands on the tape.
If one replica dies, the other carries on as if nothing happened. It does not even sense the loss, and neither do the exchange’s users. There is no failover step, because there is nothing to fail over to: the survivor was already producing the output.
A side benefit: fewer latency spikes
Running two replicas hot-hot also shaves off latency spikes caused by the operating system’s scheduler. Those events tend to be decorrelated between the two replicas. When one replica stalls for a couple of hundred microseconds because the OS had something else to think about, the other one probably did not, and its answer wins.
You can eliminate those spikes another way, but it is exponentially hard. You pin processes to cores, pre-allocate all memory, isolate CPUs, keep anything from interrupting the process and steer every interrupt to a different core. It is a long laundry list. Hot-hot removes most of the spikes for you, and you ship the product.
Failover without failover code
Checkpoints handle the ordinary crash. Hot-hot handles the crash an exchange cannot afford. In both cases, the application contains no failover logic of its own: it is a deterministic process reading one stream and writing another, and the platform does the rest.
Watch the video
Want more context on the platform? Read about the product, or get in touch with us on the contact page.