 |
VOOZH | about |
|
VOOZH | about |
We’re so glad you’re here. You can expect all the best TNS content to arrive Monday through Friday to keep you on top of the news and at the top of your game.
Check your inbox for a confirmation email where you can adjust your preferences and even join additional groups.
Follow TNS on your favorite social media networks.
Become a TNS follower on LinkedIn.
Check out the latest featured and trending stories while you wait for your first TNS newsletter.
Over half of Fortune 500 companies experience a minimum of 1.6 hours of unplanned downtime every week. This creates unhappy customers, piling pressure on stretched teams and giving IT leaders major headaches. To keep up, IT leaders have adopted more complex modern digital strategies. This infrastructure supports critical customer-facing services, yet these IT systems also create risk. As more and more software makes it into production and operational burdens increase, incidents become inevitable. Digital leaders want to know how to reduce the cost of incidents so that they have the resources to focus on innovative projects. For many, the answer is end-to-end event-driven automation.
End-to-end event-driven automation is a critical component of artificial intelligence for IT operations (AIOps). An event is anything sent by a trusted source that communicates an abnormality in system function, such as incoming data from a monitoring tool.
When automation is applied at the event level, it can create many efficiencies for responders. First, it can help make event data more understandable to responders by adding all the context and diagnostics to an event before responders get involved. This means there is no time between a notification and the responder getting the information needed to do the next action — or even executing the next action for the responder.
Applying automation at the event level helps avoid customer impact and accelerate mean time to resolution (MTTR). For well-understood issues, automatic remediation can resolve incidents without any responder intervention. Even if some responder actions are required, event-driven automation can be the difference between a major incident taking 4.5 hours to go from detected to resolved or it taking 15 minutes. That can equate to significant cost savings for your business.
End-to-end event-driven automation can also drive value across the IT function, from support teams to developers. Consider the network operations center (NOC). Event-driven automation could help normalize incoming event data and automate routing events to the right team based on a predetermined set of criteria. Without end-to-end event-driven automation, NOC teams can be overwhelmed with event data and waste time trying to determine which alerts matter and who should receive them.
There’s also value for site reliability engineers (SREs), whose job includes helping other teams work more efficiently. They could set event transformation and routing at ingestion to automate an event’s full journey. This can make incident response less cumbersome and free teams to implement auto-remediation. Engineering teams can also benefit from auto-remediation and intelligent routing so that only relevant incidents are directed their way.
Major incident management teams can also benefit from incidents being detected early and automatically routed to them with diagnostic information and normalized data. This leg up during critical incident response can protect brand reputation, reduce service level agreement (SLA) and downtime costs, and dramatically reduce MTTR.
When implementing end-to-end event-driven automation, focus on early wins to drive greater organizational buy-in and momentum. A crawl, walk, run approach may work best.
In the first (crawl) stage, consider the quick wins that could come from suppressing alert noise. By stopping an incident from sending a notification until a certain number of events arrive and signal a problem, organizations can avoid alert overload and empower responders to work undistracted. Pausing notifications for transient alerts will drive the same benefits for incident responders. This is best used for incidents with well-defined conditions that change state frequently (also known as “flapping incidents”). For example, an organization could pause some high CPU usage incidents for five minutes so that only prolonged high usage triggers an incident.
The next stage is to “walk” by ensuring events, alerts and incidents are paired with as much information as possible. Event enrichment can accelerate triage by populating events with relevant contextual information and normalizing event data so that all events look the same across teams. Alert enrichment means defining the severity of the alert it should be created with so that it is escalated to the right team and with the right priority level. Incident enrichment allows users to add notes to incidents when they’re created, such as the potential root cause of an incident and how a responder should proceed. These notes can also be published in knowledge base articles and internal wikis.
Finally, it’s time to “run” by automating the diagnostic process and even remediating well-understood incidents without responder intervention. One way of doing this is through webhooks, which enable users to define custom headers and payload body fields that trigger when an incident is created. Another is through third-party automated incident resolution capabilities that present first responders with prebuilt job templates and plugin integrations. Jobs can be invoked automatically or at the click of a mouse by these responders, empowering them to do work previously reserved for subject matter experts.
For end-to-end event-driven automation initiatives to generate the support and momentum they need, IT leaders must be able to demonstrate the value of these projects to the organization. Fortunately, small signs of success should start to appear immediately. Quantitative value is easiest to measure with certain metrics. Consider MTTR for services undergoing end-to-end event-driven automation and compare them with services that aren’t automated. Measuring SLA penalties could also provide useful insight.
Qualitative value is more challenging to measure, although key performance indicators (KPIs) do exist. Attrition rates, exit interviews and employee surveys can all prove useful data sources to measure success. Qualitative value can be one of the first signs of success in automation because teams are excited about improving their work-life balance and carving out more time for value-adding initiatives.
