Using context to triage change-triggered incidents

May 27, 2020

One of the first things incident managers do when they get an alert page from Zenduty is to check the “Context” tab of the incident. Incident context is extremely critical to get a first responder’s view of what happened and what could possibly have caused it. Context tells you what happened before an incident. In the case of 40–50% of all incidents, Zenduty’s incident context can tell you within 5–10 seconds, what could be the cause of an incident.

What causes incidents?

A few days back, I read a message from a thread in the Hangops Slack and I couldn’t help but chuckle.

Nowadays whenever I see the root cause was […] my brain automatically replaces with our false confidence convinced us that […]

Greg Sanker(@gtsanker) of Oregon State EIS did a small Twiter survey a few years ago.

Some of the best insights around the causes of incidents can be found in Subbu Allamaraju’s blog. In his study, he identified five key reasons behind incidents:

Top 5 triggers behind incidents based on a sample of several hundred incidents. These are (a) change, (b) drift, c) unknown, (d) infrastructure, and (e) certificates. Credits: Subbu Alllamaraju’s [blog](https://m.subbu.org/incidents-trends-from-the-trenches-e2f8497d52ed) “Incidents — Trends from the Trenches”

What changed before an incident?

As you can see in the pie chart above, the single biggest contributing factor to incidents is Change. The first thing every incident responder looks for after having inspected the incident alert is Context around what changes before an incident was triggered. This was the feedback we got from our pilot customers, which is why we decided to bake in context into Zenduty from the get-go.

Plug:) If you’re a NOC, SRE or ITOps team and looking for an incident alerting, on-call management, and incident response service, sign up today on Zenduty for a 14-day trial.

Incident Context in Zenduty

Zenduty allows you to add “Context” integrations - alert sources those that will send contextual information, but not create any incidents in your service. These can be your Git records(Github, Bitbucket), CI/CD(Jenkins, Travis, CircleCI), upstream alert sources from services of other teams, config change events within your cloud(AWS, Azure, or GCP), or any other tool that will give your team information about your service. Here’s how it works.

Setup your service on Zenduty(guide) and add context integrations to your service. Whenever an incident is triggered within your service, you will be able to see all the contextual alerts(info) from your sources.

Database error caused by incorrect config change

The image above shows an incident triggered by an incorrect database configuration change. The context for this incident comes from GitHub and Heroku(Cloud/CI/CD) which showed that a config change was pushed and deployed to master which caused the incident 15 minutes post-deployment. This context allows incident responders to deep-dive into possible root causes due to recent changes and also loop in the personnel who were part of the change into the incident response process. Context is especially useful when brought into your comms channels where all teams already reside.

Incident Context in Zenduty, if configured well, can help you back-track your incidents to a change in 10–20% of all your incidents. Context can also greatly help in further reducing your resolution time by ensuring that all possible change information is in your fingertips the moment you start triaging your Sev0 incident.

If you’re a NOC, SRE or ITOps team and looking for solid incident alerting, on-call, and incident response platform, do give Zenduty’s 14-day trial a spin.

Real-time alerts from Zabbix and escalation with Zenduty

May 21, 2020

Vishwa Krishnakumar

Recently, one of our customers, a 20-member NOC team of a large B2C company, had set up Zabbix to monitor a network of over 1000+ servers, routers, and switches. The NOC team wanted to set up alerting, on-call scheduling, and an escalation matrix whenever a critical network component encountered any downtime. The NOC team used Slack as the primary communication channel and Zoom for real-time communication.

Preheating the oven

For NOC teams like these running a very large operation, setting up alerting can be very tricky. It is imperative that only alerts with the highest criticality are escalated to the NOC on-engineers and therefore the selection of right metrics is critical. Two things to note before we move on to the Zabbix-Zenduty integration:

The short item update interval dogma stinks. You will need to increase the item update interval to something saner than the (often) recommended “as frequent as possible”. I’d recommend at least 5 minutes — that’s 20 times the default of 30 seconds.
The Zabbix notification system is surprisingly naive — this is where Zenduty shines. With alert suppression, alert collation, and maintenance modes, Zenduty can stop the flood of Zabbix alerts. To prevent cascading alerts, disable in all triggers “Multiple PROBLEM events generation”.

The Setup — Zabbix alerting with Zenduty

A little background on Zenduty — Zenduty is an end-to-end incident alerting(SMS, Phone/IVR, Slack, MS Teams, Email, Android/iOS Push Notifications), on-call scheduling, and response orchestration platform that helps NOC teams respond to and resolve critical downtime in the least possible time and provide your customers with industry-leading SLAs and reliability.

I won’t bore you with the details of the setup — you can see the setup documentation here. What I will show is how you can setup the escalation policies and how the incident lifecycle would look like.

TL;DR: Setup a team on Zenduty. Setup a service within your team. Add the Zabbix integration to your service.

Sign up on Zenduty here to get real-time alerts from your Zabbix setup.

Setting up escalation policies

Escalation policies dictate who should be alerted first, then second, then third, and so on, until someone responds to a Zabbix alert. In Zenduty, for each escalation “step”, you can either add a user(s) or an on-call schedule(read how you can setup on-call schedules on Zenduty here. video).

How it all comes together

Handling Comms and Collaboration

What else can you do with your Zabbix incidents on Zenduty?

There are a bunch of actionable things you can do for your Zabbix-generated incidents:

Custom route alerts depending on host or service with Zenduty’s alert rules
Assign custom incident priorities and SLA alerts for operations managers
Assign playbooks(or “task templates”) to your incidents outlining remediation steps for critical downtime alerts from Zabbix
Automatically create comms channels for every critical Zabbix alert — Zoom, Jira, Statuspage, Conference bridge, Slack

Get started on Zenduty for free here! No cc required.

Zenduty’s powerful all in one scheduling, centralization, integration and notification tool and helps you manage all your production Zabbix alerts in a single place, and get cross-channel(SMS, Phone/IVR, Slack, Microsoft Teams, Android/iOS Push notifications and Emaill) alerts and respond with speed, and resolve critical incidents before they affect your customers.

I hope you enjoyed this blog. Sign up on Zenduty for free and get started with our Zabbix integration. Feel free to leave any comments on the blog in the comments section below.

Accelerating your Zendesk customer support response times by 50% and meeting support SLAs

May 10, 2020

Vishwa Krishnakumar

Zendesk is one of the most popular ticketing support and customer service platforms available in the market. Two metrics that measure the effectiveness of your customer support are the response and resolution times — how soon are you able to respond to a customer ticket, and how soon are you able to mobilize relevant personnel, perform necessary remediation tasks and finally resolve the ticket.

Response time + remediation time = resolution time

As businesses close more deals and add more accounts, it is still imperative for businesses to maintain their SLA levels and resolve customer support tickets within SLA timeframes. Having a solid support team is great, but supporting hundreds or thousands of users in the most efficient, cost-effective way while maintaining SLAs continues to be a challenge for the majority of companies.

An SLA policy ( service level agreement) lets you set standards of performance for your support team. They’re like a target, or a deadline, within which your team is expected to respond and resolve tickets.

We spoke with VP Operations, support managers, whose companies are using Zendesk as their primary CRM. The biggest pain point were around ***ensuring ***that tickets are resolved by support agents without violating SLAs. While Zendesk does a fantastic job of moving tickets forward, teams often ended up missing SLA violation(ticket reply, ticket resolution etc.) alert emails. What they needed was a proactive alerting system that would alert them through an automated phone call, an SMS, Slack or Teams message whenever a customer ticket was about to violate or had already violated the SLA thresholds, so that they can step in and help the agent in resolving the ticket within the SLA timeframe. This is how Zenduty helped them.

Step 1: Go to the Zendesk-Zenduty setup documentation and set up your two-way outgoing integration

Step 2: Setup Escalation policy on Zenduty

In your Zenduty Teams page, create your escalation policy for critical Zendesk tickets. Escalation policies define who needs to be alerted(successively) when a ticket has not been responded to within a certain timeframe.

Step 3: Setup SLAs on Zenduty

In your Zenduty Teams page, create SLA definitions for different ticket priorities. Within your SLA definitions, define when and who should be notified before or after an SLA is violated. There are two SLA time conditions that can be used — time to acknowledge and time to resolve.

While the escalation policies ensure that the support operations team is alerted of critical tickets, the SLA policies ensure that the management and various stakeholders are alerted whenever a critical ticket violates SLA policies for that ticket.

While SLA breaches might not be common, it is always important to keep the customers informed, revisit your SLAs, do postmortems diligently, and most importantly set up a feedback loop with other teams to enable continuous improvement.

To get SLA alerts for your Zendesk tickets, sign up for a free Zenduty account here and leave your comment or questions below.

Monitoring service health and downtime events within your Google Cloud with Zenduty

May 1, 2020

Vishwa Krishnakumar

Google Cloud Platform (GCP) is a collection of Google’s computing resources, made available via services to the general public as a public cloud offering.

The GCP resources consist of physical hardware infrastructure — computers, hard disk drives, solid-state drives, and networking — contained within Google’s globally distributed data centers, where any of the components are custom designed using patterns similar to those available in the Open Compute Project.

Google Monitoring(formerly Stackdriver) is a monitoring service that provides IT teams with performance data about applications and virtual machines running on the Google Cloud Platform.

GCP Monitoring Operations performs monitoring, logging, and diagnostics to help businesses ensure optimal performance and availability. The service gathers performance metrics and metadata from multiple cloud accounts and allows IT teams to view that data through custom dashboards, charts, and reports.

GCP Monitoring Operations is natively integrated with Google Cloud Platform and hosted on Google infrastructure. In addition, it can pull performance data from open source systems, such as Cassandra, Apache Web Server, and ElasticSearch.

Monitoring service health in real-time with Zenduty

The cost and impact of system downtime are rising rapidly due to the business’s increasing dependence on data and technology. System downtime can cause loss of opportunities, productivity loss, confidence erosion, potential employee overtime costs to recover work lost and catching up, service level agreement penalties, supply chain ripple effects, and most importantly, a damaged reputation. Effective communication with staff, customers, and service providers are very important to manage the impact of downtime.

Dispatching critical GCP alerts

A key feature of GCP Monitoring is its alerting service which enables you to set up alerts to monitor the metrics and log data for the entire stack across your infrastructure. GCP Operations/Monitoring dispatches these alerts via Email. However, for critical metrics indicative of degraded user experience, email may not be good enough to elicit a prompt response and resolution from your reliability and NOC teams. For high availability services with SLA timeframes in minutes, you need to be able to promptly alert the right engineers and teams about a critical issue, and also gather responders, subject matter experts and communicate to the relevant internal and external stakeholders while at the same time, keeping a watchful eye on the SLAs.

This is where Zenduty can help you.

Integrating GCP Operations/Monitoring alerts with Zenduty

Zenduty acts as a dispatcher for the alerts generated by GCP. Zenduty determines the right engineers to notify based on on-call schedules and escalations and notifies via using email, text messages (SMS), phone calls, Slack, Microsoft Teams, and Android & iOS push notifications.

Zenduty will quickly alert you when your GCP Monitoring alarms are triggered and will include detailed JSON details provided by the GCP Operations Monitoring. This makes it straightforward and easy to diagnose issues directly from Zenduty or your chat platforms like Slack and Microsoft Teams, or on the go with Android/iOS without having to log into multiple services.

Zenduty provides you with everything you need you to minimize your mean time to recovery with advanced routing rules, flexible scheduling, analytics and reporting, integrated ChatOps (with Slack and Teams), stakeholder communications, and SLA alerts.

To get real-time alerts from your Google Cloud resources, head over to the integration documentation here.

Tip: Signup for a free Zenduty trial here and get real-time downtime alerts from GCP. To schedule a demo with our team, click here.

Custom route alerts to the right team or person

There are multiple ways of routing specific GCP alerts to specific teams or individuals, and they can be used in combination.

Routing through predefined Escalation Policies — you can create escalation policies that will define whom to notify and who to escalate the alert to if an alert is not acted upon within a specific timeframe(escalation). You can add individuals or on-call schedules/rotations with different users to an escalation policy. You can then map your escalation policy to your GCP service in Zenduty. All GCP alerts will flow through this escalation policy.

Using Custom Alert Routing Rules

Routing rules in Zenduty give you unparalleled flexibility when it comes to context-specific routing of alerts. In some situations, alerts can bypass your default escalation policy and go straight to either a specific engineer or to another escalation policy belonging to a specific team.

Collaborating on GCP alerts in Slack and Teams

Zenduty’s Slack and Team’s integrations let you manage your entire incident lifecycle from within your team channels. Zenduty also automatically creates an incident-specific conference bridge(Zoom, Teams, Webex), Jira ticket(two-way integrated) and Statuspage.

Attaching playbooks for better incident preparedness

Most teams have comprehensive playbooks in their Git or Docs or Confluence pages. But they are often hard to find and hard to execute in a transparent manner. Zenduty’s task templates lets you itemize your playbooks into role-specific tasks and automatically adds relevant tasks to your incident. All that your on-call engineer has to do is respond to the alert and look at the tasks tab to get started with triaging the incident.

Signup for a free trial!

If you’re looking for an end-to-end incident management platform, give Zenduty’s 14-day free trial a spin. Resolve your infrastructure issues before they affect your customers, increase reliability and offer industry-leading SLAs. And as always, stay Zen!

Sending Azure Monitor outage notifications to Microsoft Teams

April 24, 2020

Vishwa Krishnakumar

Microsoft Azure is a cloud computing service providing infrastructure as a service (IaaS), software as a service (SaaS) and platform as a service (PaaS) supporting multiple Microsoft Specific and third-party services and systems with 90+ compliance offerings and trusted by 95% of Fortune 500 companies to base their business on.

What is a system downtime and how does it affect me or my business?

According to Webopedia, “Downtime refers to periods of time during which a computer system, server, or network is shut off or unavailable for use. Systems go through periods of downtime for a variety of reasons, including power or hardware failure, system crashes, hacker attacks, system reboots, operating system and/or software updates, lack of network connectivity, and more”. While downtimes may be planned for maintenance, in most cases they are unplanned and unexpected.

The cost and impact of system downtime are rising rapidly due to the business’s increasing dependence on data and technology. System downtime can cause loss of opportunities, productivity loss, confidence erosion, potential employee overtime costs to recover work lost and catching up, service level agreement penalties, supply chain ripple effects, and most importantly, a damaged reputation. Effective communication with staff, customers, and service providers are very important to manage the impact of downtime.

Dispatching critical Azure alerts

A key feature of Azure is its alerting service which enables you to set up alerts to monitor the metrics and log data for the entire stack across your infrastructure. Azure dispatches these alerts via Email. However, for critical metrics indicative of degraded user experience, email may not be good enough to elicit a prompt response and resolution from your reliability and NOC teams. For high availability services with SLA timeframes in minutes, you need to be able to promptly alert the right engineers and teams about a critical issue, and also gather responders, subject matter experts and communicate to the relevant internal and external stakeholders while at the same time, keeping a watchful eye on the SLAs. This is where Zenduty can help you.

Integrating Azure alerts with Zenduty

Zenduty acts as a dispatcher for the alerts generated by Azure. Zenduty determines the right engineers to notify based on on-call schedules and escalations and notifies via using email, text messages (SMS), phone calls, Slack, Microsoft Teams, and Android & iOS push notifications.

Zenduty provides you with everything you need you to minimize your mean time to recovery with advanced routing rules, flexible scheduling, analytics and reporting, integrated ChatOps (with Slack and Microsoft Teams), stakeholder communications, and SLA alerts.

Steps to integrate Microsoft Teams and Azure with Zenduty

Below is a step-by-step guide to integrate Azure alerts with Zenduty.

In Zenduty:

Create a Zenduty account here
Create a team(one is created by default on signup)
In your team, create a service and add an Azure integration.
In the Azure integration config page, find and copy the webhook URL for later use.
Back in the service page, navigate to the Integrations tab and in the Outgoing integrations section, click on Add integration. Select Microsoft Teams Connector. Click on Configure for the connector integration. You will see an input box. We will fill this input box in Step 9.
Open Microsoft Teams in another tab. In the Apps, search for Zenduty. Add Zenduty to a Teams channel and also add as a connector.
Navigate to Chat and click on Zenduty(bot). In the chat window, type ‘login’.

Authenticate your Teams account. Zenduty is now configured for 1–1 alerts from Azure.
To send Azure alerts to a Teams channel, navigate to the Teams channel where you want to send the alerts. Click on the “…” button next to the channel and select Connectors. Select Zenduty from the application. Click on configure. Copy the webhook URL and paste in the MS Teams connector outgoing webhook configure section in Zenduty that we created in Step 4.

Configuration in Microsoft Azure:

Open the Azure portal and click on Alerts. Search on the search bar if not directly visible.

Create a new alert rule.

Select the resource which you want to monitor. Here we shall monitor a Virtual Machine.

Select the condition based on which the alert is to be triggered. Here we create an alert that is triggered when the CPU utilization of a Virtual Machine crosses 50%.

Create an Action Group (or use a previously created action group)

Enter the Action group name and short name:

Enter an action name and select Webhook under Action Type. A dialog box appears in the side.

Enter the webhook URL copied earlier from the Zenduty integration page.

Enable the common alert schema by toggling the button to yes and then click on OK.

Click on OK to add the action group.

The action group has now been created. It should visible under the section of Action Groups

Fill in the alert details appropriately. The alert rule name along with the severity will appear as the alert message and the alert description will appear as the summary in Zenduty

Click on create alert rule. You have now successfully created the alert.

Conclusion

Using the combination of Azure’s Application Insights along with webhook integration to Zenduty is a simple way to provide 24/7 monitoring and notification. Many aspects of application performance can be monitored with Azure alerts, including custom events detected within the application code. Now all essential Azure alerts and incidents will appear on the Zenduty dashboard enabling you to make use of all the power and capabilities of Zenduty to resolve incidents quickly and efficiently.

Azure service health alerts and escalation with Zenduty

April 23, 2020

Vishwa Krishnakumar

Microsoft Azure is a cloud computing service providing infrastructure as a service (IaaS), software as a service (SaaS) and platform as a service (PaaS) supporting multiple Microsoft Specific and third-party services and systems with 90+ compliance offerings and trusted by 95% of Fortune 500 companies to base their business on.

What is a system downtime and how does it affect me or my business?

According to Webopedia, “Downtime refers to periods of time during which a computer system, server, or network is shut off or unavailable for use. Systems go through periods of downtime for a variety of reasons, including power or hardware failure, system crashes, hacker attacks, system reboots, operating system and/or software updates, lack of network connectivity, and more”. While downtimes may be planned for maintenance, in most cases they are unplanned and unexpected.

The cost and impact of system downtime are rising rapidly due to the business’s increasing dependence on data and technology. System downtime can cause loss of opportunities, productivity loss, confidence erosion, potential employee overtime costs to recover work lost and catching up, service level agreement penalties, supply chain ripple effects, and most importantly, a damaged reputation. Effective communication with staff, customers, and service providers are very important to manage the impact of downtime.

Dispatching critical Azure alerts

A key feature of Azure is its alerting service which enables you to set up alerts to monitor the metrics and log data for the entire stack across your infrastructure. Azure dispatches these alerts via Email. However, for critical metrics indicative of degraded user experience, email may not be good enough to elicit a prompt response and resolution from your reliability and NOC teams. For high availability services with SLA timeframes in minutes, you need to be able to promptly alert the right engineers and teams about a critical issue, and also gather responders, subject matter experts and communicate to the relevant internal and external stakeholders while at the same time, keeping a watchful eye on the SLAs. This is where Zenduty can help you.

Integrating Azure alerts with Zenduty

Zenduty acts as a dispatcher for the alerts generated by Azure. Zenduty determines the right engineers to notify based on on-call schedules and escalations and notifies via using email, text messages (SMS), phone calls, Slack, Microsoft Teams, and Android & iOS push notifications.

Zenduty provides you with everything you need you to minimize your mean time to recovery with advanced routing rules, flexible scheduling, analytics and reporting, integrated ChatOps (with Slack and Teams), stakeholder communications, and SLA alerts.

Steps to integrate Azure with Zenduty

Below is a step-by-step guide to integrate Azure alerts with Zenduty.

In Zenduty:

Create a Zenduty account here
Create a team(one is created by default on signup)
In your team, create a service and add an Azure integration.
In the Azure integration config page, find and copy the webhook URL for later use.

In Microsoft Azure:

Open the Azure portal and click on Alerts. Search on the search bar if not directly visible.

Create a new alert rule.

Select the resource which you want to monitor. Here we shall monitor a Virtual Machine.

Select the condition based on which the alert is to be triggered. Here we create an alert that is triggered when the CPU utilization of a Virtual Machine crosses 50%.

Create an Action Group (or use a previously created action group)

Enter the Action group name and short name:

Enter an action name and select Webhook under Action Type. A dialog box appears in the side.

Enter the webhook URL copied earlier from the Zenduty integration page.

Enable the common alert schema by toggling the button to yes and then click on OK.

Click on OK to add the action group.

The action group has now been created. It should visible under the section of Action Groups

Fill in the alert details appropriately. The alert rule name along with the severity will appear as the alert message and the alert description will appear as the summary in Zenduty

Click on create alert rule. You have now successfully created the alert.

Conclusion

Using the combination of Azure’s Application Insights along with webhook integration to Zenduty is a simple way to provide 24/7 monitoring and notification. Many aspects of application performance can be monitored with Azure alerts, including custom events detected within the application code. Now all essential Azure alerts and incidents will appear on the Zenduty dashboard enabling you to make use of all the power and capabilities of Zenduty to resolve incidents quickly and efficiently.

Grafana alerts and incident escalation with Zenduty

April 22, 2020

Vishwa Krishnakumar

Grafana is one of the most popular open-source visualization tools that can be used on top of a variety of different data stores but is most commonly used together with Graphite, InfluxDB, Prometheus, Elasticsearch, Prometheus, AWS CloudWatch, and many others. Reliability engineers use Grafana is its ability to bring together several data sources together in a unified dashboard and increase the observability of your production systems.

Grafana is also a popular alerting integration source on Zenduty. Teams that monitor a bunch of metrics across hundreds of servers or applications use Zenduty as a way to alert their on-call engineering teams(SMS/Phone/Slack/Microsoft Teams/Push/Email) whenever a Grafana metric crosses a threshold and ensure swift remediation of critical alerts. The following are the steps to set up Grafana with Zenduty. Detailed instructions can be found here.

Step 1: Sign up on Zenduty. Create a team. Within the team, create a service. Navigate to your Service page and add a Grafana integration. Point the service to one of your escalation policies.

Step 2: In the integration configuration page, copy the webhook URL.

Step 3: Go to your Grafana web page and navigate to “Notification Channels”. Add a new channel. Select type as “Webhook” and give it the name “Zenduty”. Paste the webhook URL previously copied in step 2 and save the notification channel.

Step 4: Go to the graph which you want to monitor for threshold violations, click on “Edit”. In the Alerts tab, setup the configuration for when an alert should be fired.

Step 5: In the “Notifications” tab, add Zenduty as a notification method.

Whenever aa Grafana metric crosses a threshold, Zenduty will trigger an incident and execute the escalation policy and notify the relevant engineers by their set notification channels.

You can also send the Grafana alerts to Slack and Teams channels and manage your Grafana incident end-to-end within your favorite communications channels.

Meeting customer support SLAs on Freshdesk using proactive alerting and escalations with Zenduty

April 14, 2020

Vishwa Krishnakumar

Meeting customer support SLAs on Freshdesk using proactive alerting and escalations with Zenduty

As businesses close more deals and add more accounts, it is still imperative for businesses to maintain their SLA levels and resolve customer support tickets within SLA timeframes. Having a solid support team is great, but supporting hundreds or thousands of users in the most efficient, cost-effective way while maintaining SLAs continues to be a challenge for the majority of companies.

An SLA policy ( service level agreement) lets you set standards of performance for your support team. They’re like a target, or a deadline, within which your team is expected to respond and resolve tickets.

Few of our users, typically account managers, VPs, sales managers, whose companies are using Freshdesk as their primary CRM often faced challenges around ***ensuring ***that tickets are resolved by support agents without violating SLAs. While Freshdesk does a fantastic job of moving tickets forward, teams often ended up missing SLA violation(ticket reply, ticket resolution etc.) alert emails. What they needed was a proactive alerting system that would alert them through an automated phone call, an SMS, Slack or Teams message whenever a customer ticket was about to violate or had already violated the SLA thresholds, so that they can step in and help the agent in resolving the ticket within the SLA timeframe. This is how Zenduty helped them.

Step 1: Go to the Freshdesk-Zenduty setup documentation and set up your two-way outgoing integration

Step 2: Setup Escalation policy on Zenduty

In your Zenduty Teams page, create your escalation policy for critical Freshdesk tickets. Escalation policies define who needs to be alerted(successively) when a ticket has not been responded to within a certain timeframe.

Step 3: Setup SLAs on Zenduty

In your Zenduty Teams page, create SLA definitions for different ticket priorities. Within your SLA definitions, define when and who should be notified before or after an SLA is violated. There are two SLA time conditions that can be used — time to acknowledge and time to resolve.

While the escalation policies ensure that the support operations team is alerted of critical tickets, the SLA policies ensure that the management and various stakeholders are alerted whenever a critical ticket violates SLA policies for that ticket.

While SLS breaches might not be common, it is always important to keep the customers informed, revisit your SLAs, do postmortems diligently, and most importantly setup a feedback loop with other teams to enable continuous improvement.

Efficient task management for remote/work-from-home teams

March 21, 2020

Amrit Balraj

Task management best practices for remote/work-from-home teams

As COVID-19 continues to impact communities globally with health care professionals working tirelessly to prepare for emergencies and prevent the further spread of the pandemic, technology companies are also doing their part. Twitter, Google, and Amazon have issued directives instructing employees to work from home as the companies themselves move to pull out of tech events while hosting their own events virtually.

With employees working from home, this means that teams will be distributed for the foreseeable future. This also means that critical incident management can be challenging, but an important challenge to overcome. Being prepared for anything is literally the purpose of a well-rounded incident response plan, including the abrupt distribution of incident management teams.

What are some strategies to ensure that remote incident management teams are better prepared to react during critical events?

Let’s look at some best practices when teams work remotely:

Setting up communication channels:Excellent incident response teams know how integral an uninterrupted communication channel is. Communication is key for successful collaboration and the flow of relevant information for quick resolution of issues. Remote teams need to set up reliable means of communication for members to get in touch with each other with all possible haste.
Assemble the best toolchain for your team:Another aspect of systematic incident response teamwork are the tools used by teams. Today’s incident management platforms are designed to help workers no matter where they are in the world. Production teams today use a daunting arsenal of applications for monitoring, tracking, logging, analytical and alerting tools for staying updated on the health of their services. Constant notifications can lead to alert fatigue and slower response times, ultimately hitting their organizations where it hurts.
Practice blamelessness:Unanticipated change can be taxing for teams, when things inevitably go wrong it might be tempting to assign blame. Pointing fingers builds distrust within teams. Rather, now would be a perfect opportunity for learning.

Organizations have to rapidly adopt best practices for adapting to changes as employees get used to working and responding to critical events from home. In spite of the general pace of the world slowing down customers still expect services to stay up. Reliability is still as important as ever and teams that identify and adapt to the changing dynamic will stay ahead of their competition in the long run.

Along with the employers, the employees also have to find ways to cope with working from home. Employees who are new to remote work can find working in isolation distracting and unproductive. There are literally 100s of blog posts on the internet on this topic, we’ve gone through the bulk of them and compiled some of the most agreed upon ones:

Get ready for work like every day:Take a shower, get dressed and set up a home office. It can be tempting to work in your pajamas but long time remote workers unanimously agree that it’s counterproductive to do so. Having a dedicated home set up for work also allows you to step away and take breaks, which brings us to our point number two

Take adequate breaks:Schedule breaks, walk your dog, exercise, make a pot of coffee, whatever helps you refresh your mind and stretch your limbs. This helps keep your sanity in check while you grapple with a particularly resilient bug or writer’s block

Talk to your co-workers:Working from home for weeks can feel isolating, which is why communication is key. Set up stand-up meetings once every day, hear your colleagues speak about what they’re doing and vice versa. For more informal conversations, set up video or voice calls with the internal communication tool of your org, with anyone, it can be a random colleague or someone specific. Communication can help you focus on your work goals for the day and ensure that everyone is on the same page as well.

Set up an end-time:Do not over-exert your mind and body with ceaseless work, set up clear cut starting and ending time and let everyone know on Slack when you start and stop working.
Hydrate:This one is a no-brainer but people often forget to drink water when they get totally engrossed in their projects. Set reminders if you have to, keep your water bottle near your laptop or workstation where it is visible.

Do share this article with your coworkers and as always, stay safe.

Incident management for remote/WFH teams

March 19, 2020

Vishwa Krishnakumar

Incident management for remote/WFH teams

As the world tries to battle COVID-19, most of our customers here at Zenduty have started implementing social distancing measures within their companies by asking all their employees, including the NOC, SRE, ITOps, Support, and software engineering teams to work remotely or from home. While that may appear to be a drastic change in your day-to-day operations, it need not disrupt your reliability and support operations. With the right incident response practices, teams can be just as effective remotely as they are on-site. Here are some tips to supercharge your incident response with Zenduty if you’re part of a remote operations team.

Tip 1: Set up an incident command channel

Since everybody is remotely communicating on Slack or Teams, your favorite team chat application is the obvious choice for being the “command center” of incidents. Now, depending on the service/component affected or the severity of the incident, you can setup outgoing integrations with Slack/Teams and send your incident data to a specific channel — let’s call it the “incidents” channel. You can also go for the multi-channel(one-channel-per-incident) configuration if you’re worried about noise in your channels. Once an incident is triggered, the on-call engineer will acknowledge the incident, look at the incident context and if necessary, add responders through the Zenduty bot. Zenduty will notify the responders via SMS/IVR/Push/Email/Slack/Teams and add the responders to the channel automatically.

Incident notification in Slack channel

Tip 2: Setup incident roles

If you haven’t already done so, now would be a good time to experiment with setting up an incident command system — a practice inspired by the US NIMS, that is followed by the SRE teams at Google, Dropbox, Fastly, Github, New Relic, Stripe and thousands of other experienced SRE teams globally. The goal of having incident roles is to have a recursive separation of responsibilities.

It’s important to make sure that everybody involved in the incident knows their role and doesn’t stray onto someone else’s turf. Somewhat counterintuitively, a clear separation of responsibilities — Google’s SRE handbook

Roles may include an Incident Commander(leads the incident), a Communications Lead(talks to stakeholders), Operations Lead(RCA, changes), DB Lead, Infra Lead etc. Setting up roles will help you effectively delegate specific tasks as well as increase transparency into your response operations. You can define the incident roles in Zenduty within your Team pages.

Team ICS roles

Tip 3: Setup Task templates/playbooks

Most teams have a 40–80% coverage of their incidents within their playbooks, which they host on a team wiki or git or something like Confluence. A major challenge with remote teams is ensuring that the playbooks are followed to the tee by responders. Take a good look at your incident playbook document and create a task template comprising of discrete role-mapped tasks. Each playbook you have can be converted into aa task template on Zendutyv. Once you create the task template, go ahead and map those task templates to your services. As the incidents start pouring in, Zenduty will automatically take the tasks from the task template and append them to your incident tasks. As the incident commander assigns the roles to various responders, Zenduty will assign the role-mapped tasks to the respective responders.

A sample checklist of tasks auto-assigned for incidents for a service

Task templates can dramatically increase your incident preparedness, decrease on-call anxiety, reduce RCA/response errors and improve response times.

Tip 4: Comms integrations(Jira, Zoom, Statuspage)

In the ensuing confusion of a major incident, setting up Jira tickets, Zoom rooms and Status updates might not be a priority. Luckily, adding outgoing integrations on Zenduty with Jira, Zoom, and Statuspage will certainly help to reduce the friction. Once you setup these outgoing integrations, Zenduty will automatically create a linked(two-way) Jira ticket, a Zoom room and update your Statuspage. The links to these comms channels will appear on your incident page as well as your Slack messages.

Tip 5: Setting up Alert Rules

Remote on-call teams will invariably face challenges when it comes to finding the right person or team to triage or assign an incident. One way to bypass such scenarios is to use Alert Rules to override the escalation policies or assign specific subject matter experts to an incident depending on the nature or source of the alert. Alert rules can also help you suppress incidents.

Bringing all of this together

The above tips can help remote teams supercharge their incident response, improve incident preparedness as well as give your team the right confidence to handle critical incidents. While working remotely for an SRE, ITOps, NOC, Support role might be overwhelming at first, be rest assured that it only gets better with every passing day.

As the world gears up towards fighting COVID-19, we wish the patients a speedy recovery, and we continue to be inspired by our healthcare workers and by others who are caring for people around the world.