Monoliths: A Space Odyssey to Better Developer Experience

This article is the first in a series. Read more:

• Part 2: “Falling into the Star Gate of Hidden Microservices Costs”
• Part 3: “Monorepos: HAL 9000-Approved Code Management and Collaboration”

Aside from depicting a charming but homicidal AI, the movie “2001: A Space Odyssey” is perhaps best remembered for its monoliths. Seemingly out of nowhere, these perfectly smooth and space-black slabs compel prehistoric apes — and later, an advanced team of astronauts — forward to touch and understand them, to see into whatever secrets they might contain.

The film stays purposely ambiguous about the meaning of these monoliths. Instead of offering a specific truth, they reflect humanity’s ambition and growth to whoever might observe them.

Developers have a different conception of monoliths. Equally mysterious and compelling but usually lacking in the perfect proportions and peerless finish, software monoliths tell the story of how a team has designed and grown an application through refactorings, key hires and departures, and dramatic shifts in business requirements.

Unlike the ambiguity of the film’s monoliths, the stories of monoliths in software go only one of two ways:

1. Your app organically grows to the point where you worry about the tight coupling between services and the gray area between frontend client(s) and backend business logic. You may have an operational scare, like a recently introduced bug preventing users from saving their data, which becomes the last straw: All new services are to be developed in separate repositories and connected through an API layer.
2. Second, you’re forced to splinter your complex legacy monolith into dozens, maybe hundreds, of discrete APIs as part of a “cloud native transformation” strategy your CTO got sold at a conference from a startup that promises to have packaged together all the leading open source technology into an “all-in-one software delivery platform.”

In the beginning, your monolith isn’t often mourned. Microservices simplify the way you scope the impact of major or breaking changes across your app, and your peers love that they can develop new APIs in anything but JavaScript. Once again, everyone starts to feel like contributors, writing new features and solving complex optimization puzzles, not sitting in hours-long meetings to negotiate each major release of the old monolith.

The death of your monolith smooths over some hiccups around organizing developers and engineers, but it also introduces new technical requirements you hadn’t quite scoped initially. You start wondering: Was all this time we spent refactoring and configuring infrastructure, then dealing with the challenges of microservices, necessary, given the complexity of the engineering problems we face?

You picked microservices because you thought it would solve your technical problems when it’s best suited to solve organizational and collaborative issues that your company almost certainly doesn’t have. By letting your monolith die, you injected a lot of unnecessary complexity into your system, like:

• Your local development environments, which you could once launch with a single command, now require far more complex operations, like setting up a local Kubernetes cluster or paying a platform as a service handsomely to launch new instances on every run of your CI/CD pipeline.
• Configuration and orchestration work — aka operations — steals time from your core competency: building new features and layering quality, security and good policy into your app.
• You understand your system deeply, but know very little about its surroundings, leading to incidents where you or others are responding to incidents without any holistic understanding of how the app functions. You’re forced to debug through unknown waters without anyone mentoring you.
• Testing reaches time, cost and complexity limits when you need to spin up an ephemeral deployment of the entire application on every CI run, with snowballing cost and complexity.
• Shared libraries inadvertently create version fragmentation instead of simplifying common transformations and functions, as every change requires extensive testing on every service that depends upon it, unless you use the version that last worked for you.

You went from “writing bad code to building bad infrastructure,” as Kelsey Hightower warned, in search of the engineering discipline you wish you had from Day 1. Or, as David Heinemeier Hansson, CTO of Basecamp and creator of Ruby on Rails, argues, you added needless complication: “Replacing method calls and module separations with network invocations and service partitioning within a single, coherent team and application is madness in almost all cases.”

Now, in this “madness,” you start wishing you hadn’t killed off your monolith. Maybe, if you’re one of the lucky few, like the Segment team within Twilio, you can try again with a new type of monolith: one you build with intention.

A Different Story: The Maturation of an Intentional Monolith

This story begins one of two ways too:

1. You commit to this architecture from Day 1, not because monoliths are the default way to begin developing an app, but because you know the challenges ahead if you don’t necessitate the complexity of microservices.
2. You’ve tried microservices and failed to achieve the promises of collaborative nirvana, scalability and operational simplicity given the size and requirements of your company. You’ve made the painful realization that microservices injected too much complexity into the equation and are carefully designing a new consolidated service.

As you continue down the monolithic road, this architecture scales successfully because you’re aware of the common fears and misconceptions.

“Developers don’t like working in monoliths.” There is no official polling for which developers prefer, but so long as the monolith remains performant, the developer experience of working in a single repository errs on the side of simplicity. With services separated by folders and calls — not IaC and languages and repositories — you can synchronize even major changes into a single pull request for better velocity and quality.

When the Segment team transitioned to an intentional monolith, they jumped from 32 improvements in their shared libraries in a year to 46 … in just the following six months.

“A monolith is just a black box that no one understands.” Many say the same about microservices.

“Monoliths are slow.” Monoliths aren’t a great candidate for vertical scaling, but they can be uniquely fast. Data transfer stays within process memory, cheaper and faster than machine-to-machine communication, and doesn’t have to hop through the orchestration logic of distributed services. When the time comes to scale horizontally, you rely on services with more predictable performance and cost, like AWS EC2, rather than betting on serverless options — and there are plenty of application performance monitoring tools, like Scout APM, specifically designed to help you identify optimization opportunities down to specific lines of code.

“Monoliths require so much more testing.” This seems like a good problem. As Hightower suggested, writing infrastructure isn’t the solution to writing bad code, which often comes down to writing poor tests. Or not enough.

“You don’t get to play with new (cloud native) tech!” Yes, a monolith won’t work on the latest serverless or container orchestration fad, but you still have options. For example, you can deploy a service mesh like Linkerd or Istio to connect your monolith to external services via mTLS. You can improve scalability with a load balancer and a multicloud deployment, or connect your app to an OpenTelemetry Collector to send logs, metrics and traces to one of the multitudes of observability solutions available today.

Aware of these misconceptions and their realities, you can also implement strategies for overcoming the genuine challenges of building a monolith:

• You ensure all services use a single version of a given dependency to prevent fragmentation.
• You find or build a layer between your infrastructure and the outside world like Segment did with its Centrifuge project, for queuing requests/messages/events and dealing with failure gracefully rather than hoping your monolith doesn’t crash.
• You build robust test suites with mocking or recorded HTTP interactions for fast, deterministic tests that minimize the risk of any small bug taking down some or all of your app.
• You eliminate abstraction and unnecessary conceptual models in favor of simplicity, recognizing the microservices architecture solves human coordination problems at the enterprise scale — likely not a current issue where you work now.
• In the words of Hansson, you “integrate your systems until it’s impossible for one person to hold it all in their head.” If Basecamp hasn’t reached that point, you probably won’t either.

The key is that you’re not building a monolith because it’s the default. That’s where even the best-intentioned monoliths go to die. You’re building a monolith with intention because it solves the first-order engineering problems your organization has right now. Because it’s still possible to hold your app in your head. Because you don’t want to have to start from scratch again once microservices fall out of favor and you realize you understand so little about what you’ve built.

Build one with intention and see for yourself: Monoliths are the right choice almost every time.

One of the biggest benefits of monoliths comes down to who benefits the most from ballooning costs due to microservices architecture. Stay tuned for the second piece in our series, but until then, a hint: It’s not you.