NATS on AlgoX2: JetStream guarantees without a Raft group per stream
NATS is one of the easiest messaging systems to love. The client is simple, the ecosystem is broad, and the community is good. Implementing compatibility with it was a pleasure.
It is not known for speed, though. A NATS deployment typically tops out at a couple of hundred thousand messages a second, maybe 300,000, and pushing harder does not get you much more. JetStream adds persistence, but each stream costs the cluster a Raft group, so a few thousand streams is already real load.
AlgoX2 speaks NATS on every gateway. You keep your client, and what changes is everything underneath it.
Closer to an OS than a broker
AlgoX2 is, at its core, a file system. Streams live at path names, and access control lists attach to those paths, so one user can be scoped to one part of the tree and another user to a different part. Users can also run their own processes inside the platform.
Files, permissions and processes: put together, that is an operating system. It is the frame for everything else here, because in an operating system a protocol is a module, not the foundation.
NATS out of the box
Every AlgoX2 gateway already has a NATS port listening. You can turn it off, but you don’t have to turn it on. Connect a NATS client and it works.
Both core NATS and NATS JetStream are supported, and both run at the higher level of service. Data is never lost. There is no at-most-once mode, because AlgoX2 applies the same delivery mechanism to every message, whichever protocol it arrived on.
Millions of streams, not thousands
AlgoX2 does not run a Raft group per stream. Its design point is millions of streams.
With a consensus group behind every stream, a couple of thousand streams is enough to put real load on a cluster. AlgoX2 has no per-stream consensus machinery to pay for. When a Node Crashes explains how AlgoX2 replicates data without Raft.
Replays never slow down real-time readers
Every streaming system eventually meets the crybaby consumer: someone connects and asks to read from the beginning of history. On a shared data path, that one request competes with every consumer that is keeping up, and the consumers who did nothing wrong get delayed. A single replay can bring a cluster down. Teams respond by rationing access, asking people not to connect at certain times of day.
AlgoX2 gives real-time and recovery traffic separate data paths. Consumers reading the live edge of a stream are never held up by someone reading from the start.
The split costs the catching-up reader almost nothing. A consumer reading two-second-old data is already behind, and it makes no difference to it whether the data is two seconds old or two seconds and fifty milliseconds. Dedicated paths for real-time clients are a core feature of AlgoX2, not a tuning option.
One security model for every protocol
NATS on AlgoX2 is not a separate system bolted onto the side. It is an access protocol into the same platform, governed by the same security and tenancy model as Kafka, MQTT, Redis and AlgoX2’s native protocol.
Think of how an operating system supports different network cards or different file system types: any number of modules plug in, and the system underneath stays the same. That is how streaming protocols work on AlgoX2. Publish with NATS and read natively. Write with MQTT or NATS and read with Kafka. That is protocol bridging, and for some use cases it helps a lot.
When to run NATS on AlgoX2
The reasons to move a NATS workload onto AlgoX2 come down to a few:
- Throughput. You need millions of messages a second, not hundreds of thousands.
- Stream count. You want many streams without paying for a consensus group behind each one.
- Gateways. You want gateways sized dynamically, as the load demands.
- Audit. You need security features such as an audit trail and an append-only log of failed commands.
The client stays the same. What is underneath it becomes a platform built for millions of streams, where a replay never costs a live reader anything.
Watch the video
Want more context on the platform? Read about the product, or get in touch with us on the contact page.