Mitigate Risk Beyond the Supply Chain with Runtime Monitoring

Responsible organizations already spend a lot of time and resources building secure pipelines, implementing best practices and testing applications for vulnerabilities. Despite all these efforts, there’s a hidden risk that is often overlooked in the supply chain story: off-pipeline changes.

What happens when software changes aren’t properly vetted and approved through the established security process? What if they introduce vulnerabilities that go undetected until it’s too late? What if a nefarious insider or attacker runs workloads in your infra?

I want to emphasize the importance of tracking and monitoring all deployments, especially those that bypass the standard security protocols. Let’s discuss some of the risks and consequences of unapproved deployments, and explore some best practices for keeping your system secure and compliant, no matter how fast you need to move.

How Secure Are Golden Pipelines/Golden Paths?

For today’s software organizations, security has never been more top of mind. On one side there is the growing threat of being hacked by malicious actors, which is discussed in CrowdStrike’s 2023 Global Threat Report. And on the other, there is a wave of cybersecurity regulation from the government to mitigate such cybersecurity vulnerabilities. Software organizations feel the heat from both sides as they work to improve their security posture in ways that will also achieve audit and compliance with new rules.

To meet these challenges we hear a lot about the software supply chain and the importance of securing it with DevSecOps approaches and golden paths to production. Since the Biden administration’s executive order in 2021, we’ve heard a lot about software bill of materials (SBOMs), and new products, services and vendors have sprung up to help us with them. But, do DevSecOps and supply chain approaches meet our cybersecurity challenges head on?

In regulated industries, governance in development is a key focus when it comes to securing the software delivery pipeline. Teams are often required to follow strict guidelines to ensure the security and integrity of their systems.

Many organizations do this with what’s often referred to as a “golden pipeline” that incorporates all of the necessary security checks like pull-request checks, SAST, DAST, software composition analysis and so on. A typical example of a golden pipeline looks something like this:

👁 Image

Most teams follow these golden pipelines to production to ensure that their software delivery process is secure and compliant. And there’s a lot of benefits to following this approach:

• It takes the burden off each team to figure out how to comply.
• It allows all the stakeholders to collaborate and reach a consensus.
• It makes audit and change management easier.

However, while DevSecOps processes like these have been effective in controlling and monitoring the delivery pipeline, they do not secure the entire software development life cycle. Bad actors can bypass these controls and introduce unapproved changes directly into production, putting the entire organization at risk. Let’s probe a bit further and find out where the vulnerabilities are.

The Limitations of DevSecOps Pipelines

DevSecOps pipelines and golden paths are put in place to ensure that changes made to a system follow a defined process and are authorized before deployment. This helps maintain system stability, ensure compliance and mitigate risks.

But pipeline controls have one obvious limitation when it comes to ensuring the security and compliance of an entire software system. They can only ensure security and compliance for changes that have gone through the pipeline. They do not account for bad actors who access production by going around the golden path.

There are several key security questions that cannot be answered in the pipeline:

• How do we discover workloads that haven’t gone through our pipeline?
• What happens if an internal developer has the keys to production?
• What happens if we are breached?
• What happens if our deployment process has silent failures?

Think of a golden pipeline as a river running into a lake. Monitoring what’s flowing in the river does not guarantee the quality of the water in the lake. You need to monitor the quality of the water in the lake too!

👁 Image

Closing the Security Loop with Runtime Monitoring

These security questions that can’t be answered in the pipeline can be answered in runtime. Monitoring production is the only way to be sure about what’s actually running in your environments and adds an extra layer of security on golden pipelines. It’s also quite straightforward:

• Record every approved deployment in the pipeline.
• Monitor (and record) what is running in production.
• Raise the alarm if any unexpected workloads are running.

👁 Image

By monitoring and recording the state of production, organizations can detect any unauthorized changes that have bypassed the change management process. This closes the loop on security, ensures continuous compliance in your environments, and greatly mitigates threats and risks.

The Benefits for Security Audits (and Auditors)

For many teams, particularly in regulated industries like finance and health care, software delivery must be managed according to a specific process to manage risks. And to ensure that the process is followed for each change, a change management process is added for verification before deployment. Traditionally, it is a gated process of manual controls and approvals.

👁 Image

Change management implemented as a gate has a lot of negatives:

• Increases lead time for changes
• Leads to larger and riskier releases
• Requires manual audits to ensure compliance

But taking a golden pipeline approach to the change management process builds compliance into every change. By recording the evidence for every test, security scan, pull request, etc., as they happen in the pipeline, proof that the delivery process is being followed and risk controls are being met can be gathered automatically. It’s more reliable, faster and removes the need for a manual gate.

👁 Image

This approach turns subjective processes into objective, high-efficacy risk management. It also has the added benefit of enabling automated, continuous auditing. You don’t have to spend time gathering the paperwork for an audit trail when all of the evidence has already been captured in your pipelines and then securely stored.

Toward Autonomous Security and Governance

Autonomous governance is an alternative approach to traditional change management processes. It brings compliance, risk, security and development teams together to define standards and automate controls, evidence gathering, and approval work. It leaves humans in the loop for governance while providing audit-ready, continuous compliance for deliveries.

👁 Image

With autonomous governance, organizations can identify and address non-compliant changes immediately instead of waiting for audit time. It’s faster, more effective and provides better risk mitigation than traditional change management processes.

Conclusion

DevSecOps approaches are effective in controlling security in the delivery pipeline, but they fall short in securing the entire software development life cycle. Monitoring production is a powerful addition that will help you to properly secure a DevSecOps approach. By keeping a close eye on production, organizations can detect any unauthorized changes and address them immediately.

With runtime monitoring and autonomous governance, organizations can bring compliance, risk, security and development teams together. In this way, the whole tech organization can work to define standards and automate controls, evidence gathering and approval work to ensure a much more secure DevOps environment.