The State of Streaming: why it has to get boring
Relational databases have been around for 35 or 40 years. In that time they went from early systems to a new generation that is very different: better, more usable, and easy to reach for. Streaming, by comparison, is about ten years old.
It has done a lot in those ten years and solved real problems. But it is still niche next to the other ways we keep data. Many organizations rely on it without knowing they need it.
We do not think streaming will look the same in another ten years. The change we expect is that it gets more ubiquitous, more approachable, and more boring.
Three shapes of data
Broadly, the data world runs on a few models. There are keys and values, tables and databases. There are blobs and chunks of data: disks once, and now S3 with its practically infinite storage. Both are huge industries, and every engineer uses them every day.
Then there are sequences: logs of data in order. Streaming is just as powerful, but most engineers do not yet think in those terms. They use logs indirectly, not directly.
Closing that gap is the opportunity. Streaming has huge potential and plenty of room to grow, and that growth comes from making it much cheaper and much easier, more like the services we already take for granted.
A commodity, like the database
Today you can spin up a database without worrying about capacity, and it just works. You can use object storage and never think about infrastructure. Both have become commodities.
Streaming has to become one too. That is what the next wave of growth depends on.
This is the job of a platform. A platform takes complex technology and makes it simple enough that other teams move faster. Those teams should be focused on business value and close to the customer’s problem. Every piece of infrastructure they have to think about distracts from that, and streaming is a field that badly needs the simplification.
Processing belongs in the stream
Streaming data rarely moves as plain bytes. Something has to decode it, transform it and write it out again.
Today even a basic transformation is hard and expensive. You fetch the data out of the streaming system, deserialize it, parse the records, serialize them again and write them to a new stream. That new stream is one more thing to manage. Event-driven tools such as AWS Lambda help, but they still duct-tape separate pieces together. The result is a Rube Goldberg machine for some quite simple tasks.
The database world points the other way: do more inside the system itself. Simple transformations, done fast and cheaply inside the stream, make streaming easier to use. Simplifying the inputs and outputs of streams is only part of the work. Bringing processing into the stream is the other part.
From one app to the whole organization
Relational databases can do a great deal, and they can scale massively. Often what makes scaling them hard is not the load but the number of uses. It is never just you, your team and your app. A whole ecosystem of tools needs the same data.
The shift is from “one app, one database” to “my data has to exist across the organization and power five other applications”. Streams are the ideal fit for that. Yet moving from the well-trodden path of an application and its database into streams is still hard. It feels like a separate discipline.
Our goal is to make it so simple that it does not. An application developer should be able to take data from elsewhere, distribute it across the organization, and build higher-level use cases for a broad range of teams and apps.
Boring is good. We want it to be just so simple.
Streaming as a fabric
Part of streaming’s problem has always been discovery and education. It was a whole other way of thinking and a whole other tool set.
Kafka asks for a lot of that knowledge. Scaling, consumer parallelism, and the infrastructure underneath are all things a team has to understand before it gets value. That is a great deal of cognitive overhead.
Lowering it helps people, and it helps language models too. Every dependency has a cost, above all in what you have to think about. When streams carry none of those worries, an LLM building an app can treat a stream as an easy addition that brings real capabilities. Humans still need to understand where their data goes, but onboarding becomes fast and simple.
In that future, streaming is a fabric underneath the application. Developers get its benefits without having to think of it as a separate discipline.
Boring on purpose
Streaming already works. What it lacks is the ordinariness of the database and of object storage, the quality that lets an engineer reach for it without a second thought. Making it cheaper, simpler and closer to the data is how it gets there.
For how AlgoX2 builds that underneath, read Everything is a Stream and Inside the Architecture.
Watch the video
Want more context on the platform? Read about the product, or get in touch with us on the contact page.