Customer-Facing Incidents on the Rise, IT Leaders Say

Fifty-nine percent of IT leaders report that the number of high-priority, customer-facing digital incidents rose over the last year, according to a new study from PagerDuty.

The number of incidents increased by 43%, the survey participants said.

Meanwhile, only 7% reported a declining number of incidents.

These outages hurt companies’ brands and revenue streams. In fact, 90% of IT leaders believe outages have reduced customer trust. Yet 69% of those surveyed believe that their board and management are failing to invest in technology and processes to protect customer trust.

The PagerDuty survey was conducted by the market research firm Censuswide between May 31 and June 6, with 500 IT decision-makers who are responsible for IT operations at organizations with at least 1,000 employees.

Why Customer-Facing Incidents Are Rising

The report suggests that increased complexity, the rapid expansion of digital services and insufficient investment in IT infrastructure maintenance are driving the increase in incidents. This doesn’t mean that people aren’t doing their jobs. Ninety-one percent of respondents believe their IT team is effective at operations and promptly resolving technical issues.

Most customer-facing incidents are related to downtime and not a security incident. According to a 2024 report from Extrahop released in April, 51% cybersecurity and IT leaders surveyed said more than half of cybersecurity incidents at their organization are due to poor cyber hygiene. That same report found that, on average, critical vulnerabilities cause 56 hours of downtime.

From one perspective, outages at data centers and cloud service providers may actually be on the decline. The Uptime Institute’s 2023 survey of data center managers found that 55% experienced an outage in the last three years. That’s far fewer than the 69% reported in the 2021 version of the same study. Power problems continue to be the prime culprit when data centers go down.

Reducing the Prevalence, Cost of Incident Response

Automating incident response is part of the solution. According to PagerDuty, 86% of IT leaders’ organizations are making strides toward fully automating the end-to-end incident response process.

The PagerDuty survey looked at the status of automating 12 different incident response tasks. Both alert triage and data collection are fully automated, according to 40% of IT leaders. Remediation steps were the least likely task to be fully automated, cited by just 27% of respondents.

Automating the full process is important because of the difficulty involved with coordinating all the different teams involved. In fact, per the PagerDuty study, organizations that are closer to this goal have fewer customer-facing incidents and can resolve those issues faster:

• Overall, 33% of IT leaders said they have fully automated five or more processes. These organizations have fewer customer-facing incidents and quicker time to resolution.
• In contrast, the 19% that have at least five manual processes were more likely than the study average to have more incidents over the last year (29 versus 25) and longer time to resolution (four hours versus 2.7 hours).

Besides automating the entire incident response process, PagerDuty sees generative AI and artificial intelligence for IT operations (AIOps) pushing forward the automation of incident response even further. For example, the GenAI assistant in PagerDuty Operations Cloud can draft post-incident reviews.

Help Needed Coordinating Incident Response

Alignment across IT teams is the primary barrier to automating end-to-end incident response, according to 24% of IT leaders in the PagerDuty study, with another 12% citing agreement among executive leaders.

This finding jibes with what other studies have found:

• Catchpoint’s most recent survey of people involved with site reliability engineering found that the most commonly cited difficulties with recent incidents were diagnosing the problem (53%) and escalating to, or coordinating between, responsible parties (39%).
• Getting the right teams collaborating and engaged is one of the two biggest incident response and management challenges, according to a 2024 report by BigPanda, based on an Enterprise Management Associates survey of IT professionals.
• That same BigPanda report found that the most common reasons for increases in outages’ duration, cost and impact were human error, increased reliance on IT due to digital transformation and networking complexity.