Exploring the Inevitable Future of Data-Dependent Applications

Data is the lifeblood of modern applications. Check the weather, play an online game, or plan a travel route — these popular apps depend on data.

In its simplest form, a weather app is just a data app designed to source, process, and store data for fast, high-throughput retrieval by various users in disparate locations.

Online games use local data stores as much as possible but still rely on the real-time sharing of in-game location, actions, and statistics with other players. This makes game data much more dynamic than the weather data as it can require 20+ updates per second to be distributed in a rapid read/write fashion, but only to a relatively small number of players. At the same time, route planning requires data and significant amounts of computation on the server side.

Factors such as where data is needed, how fast it is required, and how much computation is needed influence data application design and infrastructure decisions.

Is a Content Delivery Network (CDN) sufficient, or will this app require the compute power of a hyperscaler?

If latency matters — as it often does — geographically distributed deployments near the edge become necessary, and replicating data using message bus technologies becomes essential.

Since full data replication is impractical, databases are often split between the original database and cached versions at the edge. All this data is then delivered through HTTP API queries from web servers — choose your vendors.

System Fragmentation Folly

The technology stacks underpinning modern applications require careful orchestration of various components to achieve optimal performance and cost efficiency. With so many parts in many places, maintainability is often traded for performance.

Complex technology stacks decay as the needs of an application change, leaving the underlying architecture unfit for purpose — unable to satisfy the shifting demands of the application.

Developers often focus on optimizing just one part of the stack to overcome the performance challenges of fragmented systems. This could involve replacing a database or adding a Redis layer. However, this familiar practice ignores a massive problem and contributes to it. Connecting multiple technologies to create a complex, unwieldy system is inefficient.

Complexity Breeds Inefficiencies

The primary challenge facing enterprise tech stacks today is complexity.

To deliver user experiences, app providers deploy API, database, caching, and messaging systems across servers and regions, using more complexity in the tech stack and, with it, more problems:

1. It’s a lot to manage — driving up latency and costs while limiting throughput.
2. It requires intermediary processes like serialization and network connections that consume many resources.
3. It creates an ecosystem where cloud providers are incentivized to keep systems complex.

Unified Approach Ushers in a New Era of Web Performance 

Instead of combining technologies like MongoDB, Redis, Kafka, and application servers, why not skip system fragmentation and use a single technology platform? Only code makes these systems run, so why not code for just one system?

Imagine combining API, caching, and database functions into a single process on a server. This would remove layers of resource-consuming logic, serialization, and network processes between each technology, resulting in extremely low response times. Such high-performing apps would see greater engagement, user satisfaction, and revenue.

But what about scale? How would a unified system like the platform above manage the heavy load of PlayStation or AccuWeather? An efficient message bus would also need to synchronize data between geographically distributed nodes.

The status quo — separate caches, compute, databases, and synchronization — no longer makes sense for global-scale application delivery. It’s too complex, inefficient, and costly.

A single system with nearly limitless horizontal scale, deep redundancy, and low network latency is the future. It will facilitate smoother development, fewer bugs and caches, and breeze maintenance — even as application requirements change.

Sounds great, right?