Exactly-Once Without the Latency Tax: Positional Idempotency in AlgoX2
Press an elevator button twice and the elevator still comes once. That is idempotency: doing a thing twice has the same effect as doing it once. If the button were not idempotent, it would queue the second press, and the elevator might leave and come back for nobody.
A payment terminal has the same problem. Tap a card twice and it must not charge twice, so somewhere a unique payment ID tells the second tap that the first already went through.
A streaming platform faces this on every publish. Exactly-once is the most expensive checkbox in Kafka. In AlgoX2 it is the default, and this post is about why it costs so little there.
The missing ACK
A producer hands the platform a message, and before anyone knows what happened, the connection drops. The producer reconnects. It never saw an ACK. Was the message published? Was it lost? Was it published, and only the ACK lost?
If the message did go through and the producer resends it, consumers now see it twice. That is how duplicates are born, and how a platform prevents them sets its throughput ceiling.
Idempotency and exactly-once semantics are two names for this one problem. People first solved part of it and called that idempotency, found it was not enough, solved the rest, and named the whole thing exactly-once. For anyone moving financial transactions, the requirement fits in one sentence: at the end of the day, the money adds up.
What Kafka makes you pay
By default, a Kafka message can be duplicated when a producer disconnects. Enabling idempotency attaches a producer ID to each message, and the broker keeps a hash to drop the duplicates it can recognise. That does not remove every duplicate, so the next step is to enable transactions.
Transactions are a two-phase commit protocol on top of everything else Kafka does, and the producer has to wait for it. A single packet might go through in 20 milliseconds. Push any real volume and latency climbs to hundreds of milliseconds, then to seconds. The protocol does not pipeline, and that is the whole problem.
Positional idempotency
In AlgoX2, every publish is idempotent by design. The mechanism is positional idempotency.
Each producer is given a private stream of its own and writes its messages there. That private stream is then interleaved into the topic the producer is writing to. Inside the topic, the producer’s messages keep their order and stay attributable to it.
So a producer always knows which messages it sent and which ones made it. After a disconnect, it finds the boundary on its own stream and sends only what lies past it. Nothing has to wait for a commit protocol, because the record of what happened is the stream itself.
A few bytes instead of seconds
This does keep extra state on disk, because the index has more to record. The overhead is a few bytes per message.
What it buys is throughput on the publish side. The producer sends data and it pipelines all the way through, with nothing to wait for. Benchmarked against Kafka transactions, AlgoX2’s default mode is three to four orders of magnitude faster.
In fact, every message in AlgoX2 carries three identities and is indexed under all three. That scheme is also what keeps a message from being lost when a node fails over after it was submitted. When a Node Crashes follows a message through all three.
Exactly-once is not a mode to turn on and pay for. When a producer’s history is a stream, knowing what was published is a matter of reading it.
Watch the video
Want more context on the platform? Read about the product, or get in touch with us on the contact page.