Governance Engineering Breaks Down the Silos in Regulated Software

For as long as we can remember, the big conversation in software engineering has been about tension between dev and ops. The core chronic conflict came down to two tribes who spoke different languages, had different values and were rewarded by different incentives. What’s worse is that these different factions also were siloed into separate organizational units with little incentive to collaborate.

👁 Image

This realization gave rise to the DevOps movement, ushering in a new culture of collaboration, automation, measurement, lean and sharing,

But, if you’re delivering software in a regulated industry like financial services or healthcare you might also be experiencing a new kind of conflict.

In these industries, DevOps comes with extra challenges that are still unsolved. Because, while engineering teams are technically capable of deploying multiple times per day, they have governance concerns that are still being performed by legacy manual processes for compliance, audit and security. And these governance requirements are set and monitored by folks in security, change management, audit and risk management in siloed parts of the organization.

The good news is that the solution to this type of problem is known: DevOps.  Or as Mark Twain said, “History never repeats itself, but it does often rhyme.”

Regulated Software

In today’s interconnected world, software permeates every aspect of our lives. It enables us to share cat pictures, read cinema reviews and (my favorite) play Wordle. However, software’s influence extends far beyond entertainment; it powers our financial systems, propels our vehicles and even controls life-saving medications.

When software of this importance fails, the consequences can be significant. Companies developing critical software must navigate substantial risks, driven by legal requirements and the imperative of maintaining their brand reputation.

This complex task of managing these risks is known as software governance. Yet, in many organizations, the individuals responsible for ensuring software governance operate in isolated silos, disconnected from each other and the broader context.

The Challenge of Governance Silos

Within organizations operating in high-stakes environments, specialized governance processes and roles have been established to manage risks effectively. These roles encompass a diverse range of specialists, including risk officers, change managers, security experts, compliance and legal professionals, and quality officers. Despite their different titles and responsibilities, their overarching objective remains the same: to safeguard against the company’s risks.

👁 Software governance teams speak in terms of risk, values safety, and are rewarded by compliance

Each specialist group speaks the language of risk, values safety and is rewarded by the company according to compliance. These individuals possess the authority to establish standards, policies, and guardrails to mitigate risks and are often tasked with inspecting and ensuring adherence to these standards.

What’s surprising about this siloed arrangement is that often it is like this on purpose. For instance, financial institutions adopt the Three Lines of Defense strategy, which provides independent oversight for risk management.

However, a notable challenge arises from this setup: these governance specialists rarely have a comprehensive understanding of the technical risks involved. Their expertise lies in their respective domains of risk management, compliance and legal frameworks, which may not encompass the intricacies of the software systems themselves.

This knowledge gap poses a significant handicap in ensuring effective governance, as it hampers the ability of these specialists to assess and address the specific technical risks that underlie regulated software.

The Challenge of Engineering Silos

As organizations recognize the outsize value that technology brings, every company has become a software company.

👁 Software engineering teams speak in terms of tech, value freedom, and are rewarded by speed

Engineers, who form the backbone of software development, speak the language of technology. They value freedom in their work and are rewarded by the speed of their delivery. However, they often find themselves in a situation in which they don’t speak the same language as the governance specialists.

Technologists struggle to understand why there is an abundance of red tape and bureaucratic processes surrounding their software development. They may feel disconnected from the objectives and constraints imposed by governance and compliance requirements. And they largely feel disempowered to influence or change these processes.

The Wall of Confusion in Governance

This chasm in language, values and rewards leads to a disconnect between the engineering teams and the governance specialists resulting in a chronic breakdown — the wall of confusion.

👁 The conflict at the heart of governance engineering

Governance silos set rules that engineering doesn’t understand or control

One of the key issues of the wall of confusion stems from rules and processes that engineering teams often struggle to understand or control. Examples of such rules include segregation of duties and change approvals. These directives are often imposed without clear context or explanation regarding the underlying risks.  What’s worse, often the implementation of these rules gets ossified in legacy, one-size-fits-all processes that don’t keep up with other tech improvements.

👁 Governance teams set rules that don't make sense to engineering

All of this leads to frustration and confusion among engineers. Without a comprehensive understanding of why these rules are in place, or the specific risks they aim to mitigate, engineers can perceive them as unnecessary bureaucratic hurdles. This lack of context and transparency can breed resistance, non-compliance and poor governance.

Engineering delivers compliance evidence that governance doesn’t understand

And the confusion runs both ways! When it comes time to validate compliance through audit, the evidence provided is in the form of tickets, docker image shas and git commits that are impossible to navigate to a non-engineer.

👁 Engineering teams provide evidence that doesn't make sense to governance teams

So simple questions from an auditor like “Can you tell me every change to production?” quickly escalate into spelunking a multitude of incomprehensible CI logs.

All this results in poor risk management, massive amounts of toil and frustration at audit, and ultimately clogs and demotivates engineering.

Toward a Better Approach with Governance Engineering

The fantastic news is this: we have seen the problem before, and we already know the solution. This exact chronic problem was what used to describe the tension between dev and ops — and the answer is DevOps!

The solution is to bring these two disciplines together to collaborate on risk management using the knowledge from both sides of the wall. A combination of Culture, Automation, Lean, Measurement and Sharing (CALMS) and holistic thinking can have an outsize positive impact.

And we already see a lot of first steps towards bringing governance and engineering together. Books have been written on the subject, and the beginnings of a community are forming.

What’s missing is a name for this. And then Bill Bensing had an insight that he shared in a talk at DOES Vegas last year: What if we applied SRE principles to software governance? Or, in his words:

“Governance engineering is what happens when you ask a software engineer to design a governance team.”

So what is governance engineering? Well, it’s DevOps of course! ;-)  Just this time including governance folks into the fold.

If you’d like to learn more about how folks are working with governance engineering in real life, you can connect with the community over at the Governance Engineering LinkedIn group.