Incident Response - how great companies do it

February 12, 2020

An incident response plan is a pre-devised action stratagem for IT teams on how to respond to critical IT events efficiently. As modern applications continue to grow in scale and complexity, there will be more people working on more interdependent systems, consequently, the question is not if a system will fail, but when, and how best to respond. A part of what makes systems more complex is customers demanding 24x7 availability from businesses today, reliability has become a competitive edge in an always-on world.

Critical events are high-pressure situations with high stakes, incident response processes help curtail some of the pressure. A well-rounded incident response plan ensures that members of incident response teams can organize all facets of incident response (stakeholder communication, public response, etc) while also focusing on the resolution of the issue. The primary goals of efficient teams should be to curtail damage, reduce MTTR, and bring down the loss of revenue.

The four aspects of incident management to keep in mind while formulating an IRP are: detect, acknowledge, triage and learn.

Detection- Identifying the issue is the first step, communicating effectively is the second. Teams that detect incidents early usually end up resolving them before it snowballs into a high-level catastrophe. Not all obscure incidents can be given individual attention, which is where log data comes in, a good log management application is essential to the tool kit of every production team. Log data can be analyzed after incidents to find out if there were minor indicators before something crashes and also tracked for the future. Log monitoring can be integrated with a good alert management application for instantaneous alerts when there are irregularities in the data.

Acknowledgment- Teams that detect issues quickly but ultimately cannot get the message across the right people fail to thrive. According to a SysAid survey named the future of ITSM, 92% of people in IT departments feel uninvolved in their organization’s DevOps activities due to lack of proper communication between teams. Notification chatter from different quarters of an organization during a major incident can make processes chaotic and slow. Breaking down team siloes through effective communication is still as relevant as ever, especially for incident management. Establishing communication protocols ensures that information is relayed through predetermined channels to principal stakeholders, business leaders and vested customers putting them at ease ensuring that core technology team members can focus on fixing the issue at hand without hindrance. The use of an alert management platform helps streamline the communication process and guarantees that the right subject matter experts are notified at the right time.

Resolution (triage)- Write, run and constantly update the game plan. Large companies like Netflix and Amazon regularly test the stability of their systems by deliberately injecting chaos. This helps teams stay fresh and improve their response times. Do not let your documentation get stale and have refresher sessions when new members of the team join so that everyone is aware of the plan. Effective games plans should also have:

Incident roles: The importance of this cannot be stressed enough, having predefined incident roles like Incident commander, communication lead, etc ensures no contradictory decisions are made during the storm, there is bolstering of stakeholder trust and improved team cohesion.
Well defined escalation policies: Based on the system being monitored, automated escalation policies will have to be defined for notifying one or several people. Customizable escalation policies that can easily be configured with schedules for optimum speed.
Task Templates: Templates ensure that every member of the team, even in the absence of senior, more experienced guidance will have an idea of how to go about getting started.

Learn- Blameless post mortems help teams understand the root cause of major incidents without finding fault with individual members. Blamelessness encourages people to come forward and talk about their mistakes without fear of repercussions. This fosters an internal culture of trust and learning.

Google’s famed SRE book highlights what are the types of triggers that warrants a detailed study:

User-visible downtime or degradation beyond a certain threshold Data loss of any kind On-call engineer intervention (release rollback, rerouting of traffic, etc A resolution time above some threshold A monitoring failure (which usually implies manual incident discovery)

Pioneers of blameless post mortems, Etsy encourages organizations to document everything. The discussion sessions during post-mortems will bring about new perspectives and ideas on how to best implement changes to strengthen stability. To this end they have even created a tool called Morgue to help production teams with the documentation process. It allows for adding information about the event with graphs, timelines, log data and communication screenshots.

Incident management runbooks are unique to every organization, factors may include tools being used, number of teams and type of business. Finding a one size fits all plan is not possible, organizations need to be open to change, to constantly learn and upgrade. Document everything and unify data from all services for maximum overview.

Zenduty is a cutting edge incident management platform designed by developers keeping the well-being of engineers in mind. Sign up for free here.

Monitoring with New Relic- Everything you need to go to get started

January 27, 2020

Amrit Balraj

DevOps is an organizational philosophy that enables continuous delivery and continuous deployment with a focus on continuous testing, automation and collaboration among dev teams, business, and operations teams. Consequently, continuous monitoring is also a key phase of the DevOps lifecycle, which is where application performance monitoring tools come into the picture. APM tools enable developers to monitor user experience in real-time with an eye on the health and stability of their applications. They enable tech teams to make informed decisions based on performance metrics and also improves observability which is integral for DevOps teams.

DevOps teams use APM tools early in the product lifecycle to identify potential threats to stability and resolve them before production. New Relic is one such application performance service designed to monitor web applications in real-time. New Relic makes troubleshooting, analyzing and scaling a painless process for engineering teams.

New Relic is designed to help modern production teams move faster without fearing high-cost downtimes. With New Relic Insights, users can assimilate data from multiple sources and analyze customer trends, create metric charts and create customizable dashboards.

Notable features of New Relic:

Users gain deep insight into the performance of their applications through customizable data-rich dashboards with New Relic Insights. Business-specific attributes can be added to each dashboard giving developers the ability to fine-tune their applications through analysis of indexed data. New Relic also offers detailed tracking of key transactions that are important to the organization. The platform enhances observability across teams which is a key aspect for modern DevOps teams.

Scalability analysis is a highly important factor, especially for startups, where a sudden increase in traffic can cause a site to crash. New Relic enables production teams to analyze site traffic and conduct load tests on their applications to ensure smooth scalability. Modern systems are dynamically changing, New Relic ensures that users can keep an eye on response times and can predict when to upgrade capacity.

Alerting is another important aspect of any APM . New Relic allows users to configure alerts when there are errors causing downtime of latency issues. Apart from customizable email or text notifications, New Relic can be integrated with alert management tools like Zenduty , VictorOps, and Xmatters.

New Relic is a feature-rich performance monitoring software that comes with a bit of a learning curve to familiarize. This article just scratches the surface of the potential that New Relic offers engineering teams.

The company has expanded its number of solutions beyond its flagship APM product and offers additional plug-ins for a host of enterprise requirements. One of the best features of all, you will start seeing metrics populating a dashboard within minutes of set. Users can also start a deep analytical X-Ray Session to start troubleshooting and optimizing your functions.

Grafana- Everything you need to know

January 21, 2020

Amrit Balraj

Grafana is an open-source platform for data visualization, monitoring, and analysis. It's designed around providing context-rich visualizations, mainly though graphs but also supports other ways to present data through pluggable panel architecture. Every dashboard is versatile and custom-buildable for specific projects of software development or business requirement. Grafana’s beautiful dashboards are one of the reasons Grafana is so popular with users. Visualizations in Grafana are called panels, and users can create custom dashboards with panels for different data sources. Grafana supports graphs, tables, heatmap and freetext panel types. Grafana users can also make use of a well established ecosystem of user-made dashboards for different types of data and sources.

In 2020, even small organizations are generating enormous amounts of data which makes tools for monitoring and analyzing big data integral to every workflow. Grafana is a popular solution due to its ease of use, visually pleasing graphs and ability to integrate with a variety of databases like Graphite, Logz.io, Influx DB, MySQL, PostgreSQL, ElasticSearch etc. An added benefit of it being open source is that developers can write custom plugins based on their requirements.

Some of the cool features that Grafana users love are:

Customizable dashboards- Grafana dashboards are feature rich and can be configured to display data from a variety of databases using varied visualization options like histograms, heat maps, and/or charts. Users love the flexibility and use of use that Grafana offers.

Native support for a wide variety of databases. Grafana is also open source with support for custom coded plugins based on project requirements.

Cloud monitoring- Cloud solutions like AWS have built in data gathering infrastructure which makes Grafana the perfect solution for dashboard creation and visualization.

Some of the cons of using Grafana are:

Lack of storage options - Grafana is strictly a visualization tool and lacks storage as a part of it’s core functionality. Prometheus is way ahead of Grafana when it comes to storage functionality because of it’s dimensional model, however Grafana can be integrated with Prometheus quite easily.

Limited alerting and incident tracking capabilities- Grafana doesn’t have well defined alerting or incident tracking capabilities as it’s functionality is more concentrated on data visualization.

Some users find that the initial set up of Grafana is confusing when compared to other visualization tools like Kibana or Knowi.

Some real world use cases for Grafana are:

DigitalOcean used Grafana to upgrade their aging, disparate data visualization tools across all of their teams to improve collaboration. Grafana offered a cost-effective easy to use data visualization solution.

Stack-Overflow required a platform to generate self-service dashboards to visualize data from OpenTSDB, Elasticsearch, and their alerting tools.

Grafana also has Grafana Enterprise designed for large organizations and their enterprise data visualization requirements. Grafana Enterprise includes 24x7 customer support and training from the core Grafana development team.

Grafana is the ideal solution for visualizing system data. Streamlined installation and configuration ensure accessibility and observability for all users. The beautiful and customizable interface makes the experience of monitoring application performance rewarding and empowering for all users.

Find the guide for Zenduty+Grafana integration here

Site reliability engineering- Predictions for 2020

January 10, 2020

Amrit Balraj

As we head into 2020, it's clear that DevOps has finally crossed the divide and gone mainstream. With DevOps firmly ingrained as a standard practice, we now look at how it will evolve. DevOps is driving more overall alignment between development and operations teams than has ever existed in the past. For developers, that means building and delivering impeccable apps to market quickly. Operations teams can focus on a more Ops-centric approach for ensuring streamlined product releases and post-production experience for customers. The two groups can now more readily focus on the things they do best while believing that the other team is in sync.

As agile development methodologies continues to grow and organizations shift focus to improve quality without compromising speed, visibility across the software development life cycle will become of prime importance. In 2020, teams will increasingly be dependent on software that proactively monitors, and even anticipates, potential incidents. Let’s look at the top trends which will influence the world of tech this year.

Kubernetes

According to The State of Kubernetes Adoption and Security report published by Alcide which is based on 200 responses from DevOps engineers, cloud architects, and security professionals, 45% of companies are now running Kubernetes. As more organizations adopt Kubernetes the demand for experienced engineers is on the rise. The rapidly evolving landscape of Kubernetes also means that pros, as well as beginners, have a lot of learning to do in 2020.

SRE

Site Reliability Engineering, coined by Google engineers in 2003, has been around for 15 years or more. However, the technological ideology is still relatively misunderstood. As per the report linked above, based on a survey of 118 SREs globally, 64% of respondents indicate the SRE role has only existed in their companies for 3 years or less. With automation and observability becoming a key factor for more efficient and rapid deployments, an SRE job profile has become one of the most sought after in tech this year.

Microservices

Microservices architecture will become even more mainstream in 2020 as evident in this Google trends graph. The main reason for this is that organizations are constantly pushing the envelope developing complex software and microservices that offer developmental speed, seamless scaling and smaller release cycles in a highly competitive marketplace. Successful companies like Spotify, Amazon, and Netflix have achieved long term growth and sustainable workflows through early adoption, which is something to bear in mind this year.

Serverless

Serverless architecture is helping developers to focus on their applications without having to manage servers. Companies are turning to FaaS and BaaS in 2020 as they ultimately provide business value through billing based on consumption while building resilient systems. Serverless computing, on a greater scale, is also about reducing your carbon footprint through limited usage which makes it more sustainable for the planet in the long run.

Security

Security breaches continue to disrupt service and raise costs for companies on a global scale. This is leading to more companies embedding security into DevOps practices (or DevSecOps) to counter breaches in their defenses. DevSecOps is about implementing security early in the development process and including everyone is aware of best practices.

Microservices and cloud native applications offer unique vulnerabilities, understanding them is crucial to be better prepared for the future. DevSecOps improve collaboration across teams so that there is improved visualization and observability.

This is by no means an all-inclusive list of predictions as software development is quite large and complex. It is essential for new and long-time organizations to stay aware of the ever-changing landscape of technology and adopt them as per their requirements. One of the great things when looking ahead is seeing all of the choices we have for custom building your path to growth.

Be Zen.

Incident Alert Routing — Getting woken up only by alerts that matter to you

January 1, 2020

Vishwa Krishnakumar

Incident Alert Routing — Getting woken up only by alerts that matter to you

Site reliability engineers have one of, if not the, toughest roles in any organization. While dealing with incidents is one part of the job, the other is to build reliable systems. Google’s SRE book sums this approach nicely.

At least 50% of each SRE’s time should be spent on engineering project work that will either reduce future toil or add service features. Feature development typically focuses on improving reliability, performance, or utilization, which often reduces toil as a second-order effect. We share this 50% goal because toil tends to expand if left unchecked and can quickly fill 100% of everyone’s time. The work of reducing toil and scaling up services is the “Engineering” in Site Reliability Engineering. Engineering work is what enables the SRE organization to scale up sublinearly with service size and to manage services more efficiently than either a pure Dev team or a pure Ops team.

One of the most important challenges for an SRE when it comes to balancing work between firefighting and toil reduction is the issue of alert noise. An alert, for a particular engineer, can be considered as noisy if the alert does not provide any information about an event that has already happened, or an event that is about to happen, that significantly affects the end-user or customer, and the alert is not actionable on the part of the said engineer. Alert noise, not only distracts your engineers from their engineering duties but also causes needless stress, increases anxiety and exacerbates mental and physical health issues that your engineers may already be facing. It will come as no surprise that organizations with the noisiest alerts also have the highest engineering churn rates in the industry.

Every Monitoring or IT management tool sends a lot of alerts daily. While some alerts are important and actionable, most of them are noise. Nobody likes to wake up to an alert at 2 AM in the morning unless it’s a major incident or an outright catastrophe. We took a close look at the incident management services and how they help teams reduce alert noise. We were surprised to see that all of them have implemented just bare-bones features to mitigate alert noise, and despite being in the market for over 7–10 years, have failed to see alert noise reduction as something that needed to be solved in a meaningful way.

How Zenduty can help you shrink a river of alerts to a trickle

When we started building Zenduty, solving the problem of alert noise was top on our list of priorities. And so we did, with Alert Rules. Following are the criterions on which you can route your alerts on Zenduty along with operations like <, ≤, >, ≥, ==, !=, contains(substring), not contains, regex match, is empty, is not empty, in(multi-select), not-in(multi-select), between and not between:

Payload Search — Every alert from a monitoring integration sends its own custom payload along with the alert parameters. For example, an critical alert from Jenkins will also contain a payload like — {“job”: “build-prod”, “job-number”:5, “status”: “SUCCESS”}. You can run a key search on the alert payload and use the above operators on the value
Incident Message and Incident Summary— Route if you find a keyword in the alert message or summary
Alert Time of the day —Route depending on the time of the day
Alert Day of the week — Route depending on the day of the week
Alert Date — Route if the incident falls on or between specific dates
Alert Type — Route depending on the type of the alert(critical, error, warning, acknowledged, resolved and info)

Following are the actions you can take if the alert routing conditions are met:

Suppress — Suppress the incident
Change Alert Type to —Let’s say the alert type of the incoming alert is critical(which will create an incident), and you do not want the alert to create an incident, you can change the alert type to “info”.
Add Note — Automatically add a not to the resultant incident that will help the on-call engineers and incident commander to determine the RCA and triaging steps
Route to Escalation Policy — Route to an escalation policy that is not the default on the service affected. This may be useful in cases where you might need a more aggressive or less aggressive escalation depending on the type of alert that has come in.
Assign User — Assign the incident to a specific user. This is useful where a known component is affected and the fastest way to resolve the incident is by assigning it to a known subject matter expert
Change Incident Urgency — change the urgency of the incident. If the alert creates a low priority incident and the your conditions detect a potentially high priority incident, you can conditionally change the priority to high and vice-versa.
Change Alert Message — create your own custom Incident Title from the alert payload
Change Alert Summary — create your own custom Incident Summary from the alert payload
Change Alert Entity ID — change the alert entity_id to implement custom deduplication
Assign Role to User — after the alert creates an incident, assign a particular role to a particular user
Assign incident tag — automatically add a tag to the newly created incident depending on the matching condition. For example, if a backend component is affected, add a tag “backend”
Add incident task and template(coming soon) — Automatically add an incident task template or a task to the incident

The Zenduty Alert Rules allow you to fine-tune your incident alerting at many levels and make sure that the right team members are alerted when specific conditions are met, add notes and tags, route to a different escalation policy than the service default policy, customize the incident title and summary, and most importantly suppress the incident.

If you’re looking for an end-to-end incident management platform with awesome alert routing and response management capabilities, do give Zenduty a spin and leave us your feedback in the comments below.

And finally, be zen!

Making on-call superheros

December 27, 2019

Amrit Balraj

Building a world-class service is as much about maintaining software as it is about developing it. On-call engineers are typically responsible for ensuring the reliability and availability of your service i,e your reputation, and source of revenue. Robust on-call schedules ensure that the right people are ready-to-go during times of crisis.

Organizations continue to depend on on-call schedules and incident response processes that are a source of stress/anxiety or panic to employees. In the long run, they lose great employees due to redundant on-call strategies, which ultimately affect their turnover. Protecting your employees from unnecessary drama and noise has to be prioritized while supporting maximum growth and maintainability.

Some of the common organization mistakes are:

Lack of flexibility- Incidents happen when you least expect them to, which means that there will be times when schedule changes will have to be made to accommodate employees who are unavailable due to personal emergencies. Having a schedule with flexibility means that employees will feel happier at the same time making the team more adaptable to different kinds of situations.
Knowledge Silos- Ownership of different aspects of a service is divided across teams with limited internal communication. Lack of subject matter expertise within on-call teams leads to engineers tackling issues they are not experienced in, a sure-fire recipe for disaster.
Leaning on operations- This is another common mistake that happens to organizations that scale too quickly without plans for maintaining growth. Reliance on one team to handle the maintenance of infrastructure leads to pager fatigue and burnout.
Work-life imbalance- Employees are not encouraged to have a healthy work-life balance leading to stressful, unhappy employees. Balanced employees lead to productive work, leading to rapid decision making in high-intensity situations.

An effective on-call schedule is one that values employee time and their happiness with a focus on functionality. A team with an effective on-call strategies views every event as a learning experience. Here are some best practices:

Have primary and secondary responders, followed by higher level executives like managers. Define escalation policies for contingencies like family emergencies or if one responder sleeps through their alerts. Everyone is human and individuals are going to miss notifications on occasion.
Make a playbook to help future responders to scrutinize to understand impact and steps taken to resolve it. Accuracy is key while writing these, conduct 30 minute review meetings every week to find ways to improve your incident response plans.
Use the right tools, utilize automated monitoring tools which unify notifications coming from all of your alerting tools. This will help your engineers cut down on noise and focus on making snap decisions.
Listen to your engineers, smart employees tend to be strong willed and opinionated. Since they are also the ones who work intimately with the infrastructure, their insights will be valuable.
Encourage employees to have a healthy work-life balance, you are inherently responsible for their well being. Ultimately the performance of an employee is dependant on their happiness. Spread awareness about the importance of mental health days and channelling their energies to hobbies
Embrace DevOps as a philosophy rather than just DevOps titles with traditional Ops. DevOps encourages shared responsibilities and knowledge within organizations. Internal collaboration reduces friction within teams, improving communication and visibility, reducing overall stress.

Not everyone is an expert on DevOps ideology and not everyone will be versed with every single software in your system. However, support each other as a team and help channel personal grow, overcome challenges, and ultimately build great tech.

Further reading sources: On-Calliday: A guide to unsucking your on-call experience

Being an On-Call Engineer: A Google SRE Perspective

Rob Ewaschuk’s Philosophy on Alerting

Incident Response 2.0 — The Zenduty Incident Command System(ICS)

December 15, 2019

Vishwa Krishnakumar

Incident Response 2.0 — The Zenduty Incident Command System(ICS)

Zenduty - Slack Incident Command System integration

We are super excited today to introduce our latest Zenduty integration with Slack, which we are calling the Zenduty Slack Incident Command System(Slack-ICS). This was many months in the making and went through multiple iterations and it is something we believe will redefine proactive incident management and response. The Slack-ICS was really something we thought of the moment Slack released its game-changing **Block-Kit UI Framework, and let us upgrade our Slack integration from somewhat of a Bruce Banner to the incredible Hulk. A special shoutout to the DevRel team at Slack, especially Emily Pinkerton, Gregg Rybicki and [Aman Gupta] for helping us navigate through the Block kit framework and selecting the right combination of UI ingredients.

The Slack-ICS was the next logical implementation in our Zenduty-ICS solution set after Incident Roles, Incident Tasks, Task Templates, and Incident Tags. The goal of the Zenduty-ICS system is to implement a recursive separation of responsibilities, a recognized chain of command, to manage and assign incident tasks by roles or an ad-hoc basis, and establish a clear path to resolution post-incident acknowledgment.

The before(pre-ICS) experience

Before our Zenduty-ICS framework was shipped early November, our Slack user experience was, in my opinion, minimal —but pretty much at par with our competitors. An incident is triggered, alerts were sent to phone, SMS, Email, Slack, and a message was sent to a dedicated Slack channel. Actions on this Slack message were restricted to acknowledging, resolving and reassigning, with alerts being sent for any changes to the state of the incident.

There were a couple of problems, especially involving critical incidents and multiple responders/stakeholders:

If there’s no proper team-wide incident response training whereby someone declares themselves as commander, nobody will know who is leading the incident. (Ack’er != Commander).
There were no standard operating procedures or checklists in place. For teams that have been through training, users might declare roles in the channel and scurry over to an incident response document or a runbook.
There was no way for a commander(if designated) to delegate tasks. There were mechanisms to assign concrete roles, tasks were assigned ad-hoc, follow-throughs were forgotten, nobody knew who was working on what unless they read through a bunch of message threads.

Incident Management 2.0 — The Zenduty-ICS experience

Last month, we introduced the four central pieces of the Zenduty incident response framework: Tasks, Roles, Tags, and Task Templates.

These four pieces combined will give anyone the entire high-level overview of the progress of any incident. Whenever an incident is triggered in Zenduty, the on-call engineer, after receiving the page, may acknowledge the incident and then claim the incident commander or ask someone else to take command. The incident commander will then add relevant responders, invite users to take up specific roles, assign tasks to those roles and update the incident status on the go.

Incident Roles

Team ICS roles

Depending on your org structure and business, you can define roles like :

Incident Commander — leads the incident, and assigns roles and tasks to different people
Comms Lead — communicates about the incident to customers and stakeholders
Ops Lead — making changes and pushing the fix

The Task Template

A sample checklist of tasks auto-assigned for incidents for a service

The task template is the starting point for any incident. In the task template, you can define standard operating procedures and point them to specific roles. When a particular service encounters an incident, the task from the task template will automatically be added to the incident task list and after the incident commander assigns roles to specific individuals, these individuals will automatically inherit the tasks in the incident. At any given point, anybody in the team or company will know exactly who is leading the incident and who is dealing with specific aspects of the incident.

Incident Tags

Incident tags let you classify the incident based on component, service, priority, customers or any other segment. These tags can then be used in your analytics dashboard to measure tag-wise service, team, and user performance.

Incident tagging for better classification

Anatomy of an incident with ICS

There are two ways in which an incident can be created on Zenduty— automatically through a service integration/API or manually by a user via the Zenduty dashboard or Slack slash command. Once an incident is triggered, Zenduty will automatically run the escalation policies and Email/SMS/Call/Slack the folks on-call. In the meantime, Zenduty will create a dedicated Slack channel for the incident, a Jira ticket, a Zoom/Webex/Hangout room and a Statuspage incident, the links to which are added to the incident’s Comms section. Zenduty will then automatically take all the tasks from the task template associated with the service and add them to the incident tasks. If any alert rules and actions are configured, it can automatically assign a user or tags as well.

Easy links for Incident communications

Once the Slack channel is created, Zenduty will automatically add the on-call engineer. The on-call engineer can acknowledge the alert and then check the service context alerts for some preliminary RCA, and start roping in additional engineers(by adding them as responders and Zenduty will notify them via SMS/Call/Email/Slack)to assist with the incident. Zenduty automatically adds these responders to the incident channel. At this point, the on-call engineer can assign themselves as the incident commander, or someone else, depending on the situation. The incident commander then proceeds to assigns different roles to people.

Incident roles

Tasks from task template, auto-assigned during role assignment

Once there are sufficient SMEs in the channel, each person with a role will start working on their assigned tasks. The incident commander may choose to assign additional tasks to a role or an ad-hoc member. Zenduty will notify the assignee whenever a task is assigned to them.

Stakeholders can follow the progress of major incidents in their pwn dedicated channels and Zenduty will auto-update the incident message as new incident updates keep pouring in.

Once an incident is completely resolved, Zenduty will automatically archive the channel and close any associated Jira issues or Statuspage incidents.

Best Practices for Incident Management

The following are the best practices as prescribed by Google in their Google SRE book(highly recommended).

Prioritize: Stop the bleeding, restore service, and preserve the evidence for root-causing.

Prepare: Develop and document your incident management procedures in advance, in consultation with incident participants.

Trust: Give full autonomy within the assigned role to all incident participants.

Introspect: Pay attention to your emotional state while responding to an incident. If you start to feel panicky or overwhelmed, solicit more support.

Consider alternatives: Periodically consider your options and re-evaluate whether it still makes sense to continue what you’re doing or whether you should be taking another tack in incident response.

Practice: Use the process routinely so it becomes second nature.

Change it around: Were you the incident commander the last time? Take on a different role this time. Encourage every team member to acquire familiarity with each role.

Here’s a video showing the entire incident lifecycle with Zenduty ICS in action.

Zenduty — SRE Puzzle of the week — Forgotten password

December 11, 2019

Vishwa Krishnakumar

Zenduty — SRE Puzzle of the week — Forgotten password

Let’s say you have a safe, and you forgot the password. You know it is a four digit-password, and you don’t want to type 10,000 4-digit numbers into the safe.

The safe has the property that if the password is “1234”, you could type “01234” “85624359571234” and it would open; it only cares about the last 4 numbers pushed.

Write a function(in the comments section below) to produce a sequence of numbers to enter into the keypad. Do share the joy of puzzle-solving with your friends.

Incident Alerts — Reducing Alert Noise

December 10, 2019

Vishwa Krishnakumar

Incident Alert Routing — Reducing Alert Noise

Our monitoring tools are the sentinels of our systems. They scan every aspect of the health of our systems, and alert you whenever anything goes awry. As sysadmins, it is expremely important to design your alerting infrastructure in a way that maximizes your signal to noise ratio.

To borrow a quote from the great Captain Edmund Blackadder, finding a root cause of in incident is like looking for a tiny piece of hay within a stack full of needles. Alert noise from a badly configured system can kill your company in many ways. To begin with, the noisier the system, the higher the probability of missing a critical alert. Second, noisy alerts can result in serious mental and physical health issues within your operations teams, and consequently, higher churn and loss of in-house expertise. It is therefore important that you constantly tweak your systems to weed out alerts that are not important or actionable. Here are three ways:

Grouping alerts by component

Most monitoring tools provide a range of metrics to alert on whenever the metric crosses a threshold. Instead of sending one alert per metric, identify a metric that can point to the root cause instead of symptomatic metrics. A good approach would be to group a set of metrics together and fire an alert if a component is down.

Analyze, analyze, analyze

Alert noise reduction is a continuous process. One must always look at the entire alert data at the end of every month and segment them into actionable/important alerts and suppressable alerts. Analyze correlations between different kinds of alerts, look at alerts that correlates with multiple alerts and pick an indicative metric to alert on.

Logical operations

If your monitoring tools have logical operators that can act on variables in real time, now is the time to put them to work. Instead of sending an alert for everytime a threshold crosses 90%, send an alert if the 90% threshold was crossed 50% of the time in the last 15 minutes. If your system generates multiple alerts for the same root cause, use multiple variables for threshold breaches within a set time-window(depending on your SLAs) - if metric x,y and z have crossed their thresholds for 50% of the time in the last 15 minutes, raise an alert.

We recently shipped Zenduty Alert Rules, which allow you to fine-tune your incident alerting at many levels and make sure that the right team members are alerted when specific conditions are met, add notes and tags, route to a different escalation policy than the service default policy, customize the incident title and summary, and most importantly suppress the incident.

And finally, be zen!

On-call doesn't have to be stressful

November 29, 2019

Amrit Balraj

“Being on-call is a critical duty that many operations and engineering teams must undertake to keep their services reliable and available. However, there are several pitfalls in the organization of on-call rotations and responsibilities that can lead to serious consequences for the services and the teams if not avoided. This chapter describes the primary tenets of the approach to on-call that Google’s Site Reliability Engineers (SREs) have developed over the years, and explains how that approach has led to reliable services and sustainable workload over time.” - Google SRE book

On-call schedules are a necessary evil while maintaining the reliability of your services. Being on-call is a foreboding experience, even when things are going well, it can end up making you feel drained and exhausted. The culture of DevOps fosters rapid deployment, SRE and continuous integration which creates a need for quick remediation of incidents when they occur. Several organizational initiatives help make the on-call experience more bearable like flexible working hours, remote-friendly offices, etc. There are several measures that you can take to make your experience better:

Optimize your environment Ensure that your home is a well organized on-call environment, with all of your devices at easily accessible places. Even if you are woken up from deep sleep, your brain shouldn’t have to waste time figuring out where everything is. This will vastly improve your response time when notifications are blowing up your sleep.

Clean up your notifications Scrubbing up your notifications is an effective tactic for on-call as notifications will be flying at you from one or multiple sources. Set up customized notification rules depending upon the severity of the incidents and SLAs.

Urgent notifications from family members can be filtered through based on rules you’ve set eliminating chaos during times of duress.

Personal Care On-call shifts can stress you out, even when everything is going according to plan. Prioritize your mental health, take mini-breaks whenever you can, stretch, grab a snack or get some fresh air. Night shifts can throw a spanner into your sleep schedule, however, regular sleep is vital. Your brain will incrementally slow down without proper sleep which might cause you to make more mistakes, ultimately affecting your response efficiency.

Distressed routines can also lead to stress eating which will ruin your physical well being in the long run. Make sure to get enough rest, sunlight and exercise to help you find your zen during on-call schedules.

Having your incident management procedure Additionally, with your organization’s incident management plan, have your plan during contingencies. An incident management plan is only as strong as the sum of all it’s parts.

Have a personal checklist or questionnaire to tick off when you go through the motions. Post-incident focus on the end-user experience and work backward, quickly identify what went wrong, time to detection and resolution for future post-mortem meetings. A big part of making the on-call experience better is building more resilient infrastructure through learning from our experiences.

Set up the right tools Using the right tools will go a long way in helping you keep your focus on the issues that need to be prioritized. Zenduty is a state of the art incident management platform which gives you a unified view of all your notifications from your infrastructure where every member of the team is aware of the incident no matter where they are.

The platform also provides incident analytical metrics to help you prepare for post-mortem meetings and leverage your incident management plans. With Zenduty you can improve your team’s MTTR while helping them focus on alert that matter.

Custom escalation routes can be defined pre-incident depending on work schedules along with pre-defined incident roles so that everyone is aware of who is working on what eliminating chaos.

Zenduty has support for 100+ services like Sentry, Freshdesk, AWS cloud watch, New Relic and Zabbix.

Ultimately it all comes down to your physical and mental well-being, prioritizing this will make on-call much easier to manage. A perfect system is not one that never fails, but one which provides a sustainable environment for response and restoration.