Mitigating Risk With Automation: Do’s and Don’ts

Automation is a key competency for many teams in any organization. Enabling teams to complete repeatable tasks reliably and with minimal manual intervention creates more space in the workday for more complex or interesting work. We know we can’t solve difficult macroeconomic environments with automation. At best, we can hedge a bit and hope to keep the lights on more efficiently.

Beyond just getting more done, automation provides teams with powerful risk mitigation across various workflows and processes.

Why We Automate

Automation projects often get started with a misguided “do more with less” mindset. The “less” can mean many things: smaller budgets, shorter timeframes and/or reduced headcount. A team can’t do more with less, so they will be forced to prioritize and focus on what projects mean the most to their users. Smaller teams often still accomplish fewer tasks overall but funnel efforts more intentionally toward the biggest wins.

Other automation efforts are begun to mitigate toil, the types of operational work that naturally happens in technical environments. For folks working in a site reliability engineering-based environment, toil is a pretty specific set of tasks. Toil includes things that are manual, repetitive and grow as the environment grows. Google also defines toil tasks as inherently automatable — they are often excellent targets for automation because they are tasks that are well-understood and repetitive.

We can also automate strategically, using automation to enable teams to perform tasks more reliably, to provide a mechanism for delegating tasks and to mitigate various risks to the business.

What Risks Automation Will Mitigate

Digital businesses have a number of risks that can hurt customer experience or revenue streams. Strategic automation in key areas can help manage some of those risks.

One of the big organizational benefits of automation is that it is an encapsulation of the expertise of the team. Good automation preserves institutional knowledge. An automated component, script or tool is a composite of the organization’s learning about the task being automated.

Having this experience preserved in an automation artifact makes it available when the person considered to be the expert isn’t. Subject matter experts like to take vacations just like everyone else! They also move on to new positions or leave the organization. Automation is an SME that doesn’t leave, reducing the risk that an incident will take longer to resolve in the absence of an SME.

Teams can also consider automation to be an unflappable team member in the face of a stressful incident. Handling outages affecting customers is challenging and not everyone on your team will handle high-stress situations well. Nerves can affect the way a responder approaches the incident and can have a negative effect on their ability to troubleshoot and process information about the systems.

Having automation in place to produce reports, scan for errors and run simple mitigation procedures like restarts or resource allocation adjustments reduces the risk of a flustered responder losing track of what they are looking for or how to run a task.

Finally, automation is a key component in mitigating security vulnerabilities. Vulnerability reports include recommendations from the vendor on how to patch or otherwise shore up a vulnerable system. Deploying security fixes in a timely manner is crucial for protecting systems from being breached by bad actors.

What Risks Automation Can Introduce

Many teams worry that having more automation — and more code — introduces more potential for technical debt in their environments.

Maintaining automation is important. Having automation that only partially works or is outdated relative to the systems it is meant to operate on is worse in many ways than having no automation at all. When responders and other team members come to rely on automation and don’t realize it no longer matches the systems that are deployed, poorly maintained automation can cause more damage than it was intended to remediate.

Automation should be maintained alongside the systems it is intended to act upon and updated as the services change. Major changes like operating system upgrades or platform migrations will obviously require automation to be updated, but other changes like new libraries or dependencies can also mean adjusting scripts or tools. Making a regular habit of auditing and testing automation at regular intervals or project milestones will help keep automation in sync with its services.