Efficient task management for remote/work-from-home teams

March 21, 2020

Amrit Balraj

Task management best practices for remote/work-from-home teams

As COVID-19 continues to impact communities globally with health care professionals working tirelessly to prepare for emergencies and prevent the further spread of the pandemic, technology companies are also doing their part. Twitter, Google, and Amazon have issued directives instructing employees to work from home as the companies themselves move to pull out of tech events while hosting their own events virtually.

With employees working from home, this means that teams will be distributed for the foreseeable future. This also means that critical incident management can be challenging, but an important challenge to overcome. Being prepared for anything is literally the purpose of a well-rounded incident response plan, including the abrupt distribution of incident management teams.

What are some strategies to ensure that remote incident management teams are better prepared to react during critical events?

Let’s look at some best practices when teams work remotely:

  • Setting up communication channels:Excellent incident response teams know how integral an uninterrupted communication channel is. Communication is key for successful collaboration and the flow of relevant information for quick resolution of issues. Remote teams need to set up reliable means of communication for members to get in touch with each other with all possible haste.

  • Assemble the best toolchain for your team:Another aspect of systematic incident response teamwork are the tools used by teams. Today’s incident management platforms are designed to help workers no matter where they are in the world. Production teams today use a daunting arsenal of applications for monitoring, tracking, logging, analytical and alerting tools for staying updated on the health of their services. Constant notifications can lead to alert fatigue and slower response times, ultimately hitting their organizations where it hurts.

  • Practice blamelessness:Unanticipated change can be taxing for teams, when things inevitably go wrong it might be tempting to assign blame. Pointing fingers builds distrust within teams. Rather, now would be a perfect opportunity for learning.

Organizations have to rapidly adopt best practices for adapting to changes as employees get used to working and responding to critical events from home. In spite of the general pace of the world slowing down customers still expect services to stay up. Reliability is still as important as ever and teams that identify and adapt to the changing dynamic will stay ahead of their competition in the long run.

Along with the employers, the employees also have to find ways to cope with working from home. Employees who are new to remote work can find working in isolation distracting and unproductive. There are literally 100s of blog posts on the internet on this topic, we’ve gone through the bulk of them and compiled some of the most agreed upon ones:

  • Get ready for work like every day:Take a shower, get dressed and set up a home office. It can be tempting to work in your pajamas but long time remote workers unanimously agree that it’s counterproductive to do so. Having a dedicated home set up for work also allows you to step away and take breaks, which brings us to our point number two

  • Take adequate breaks:Schedule breaks, walk your dog, exercise, make a pot of coffee, whatever helps you refresh your mind and stretch your limbs. This helps keep your sanity in check while you grapple with a particularly resilient bug or writer’s block

  • Talk to your co-workers:Working from home for weeks can feel isolating, which is why communication is key. Set up stand-up meetings once every day, hear your colleagues speak about what they’re doing and vice versa. For more informal conversations, set up video or voice calls with the internal communication tool of your org, with anyone, it can be a random colleague or someone specific. Communication can help you focus on your work goals for the day and ensure that everyone is on the same page as well.

  • Set up an end-time:Do not over-exert your mind and body with ceaseless work, set up clear cut starting and ending time and let everyone know on Slack when you start and stop working.

  • Hydrate:This one is a no-brainer but people often forget to drink water when they get totally engrossed in their projects. Set reminders if you have to, keep your water bottle near your laptop or workstation where it is visible.

Do share this article with your coworkers and as always, stay safe.

Incident management for remote/WFH teams

March 19, 2020

Vishwa Krishnakumar

Incident management for remote/WFH teams

As the world tries to battle COVID-19, most of our customers here at Zenduty have started implementing social distancing measures within their companies by asking all their employees, including the NOC, SRE, ITOps, Support, and software engineering teams to work remotely or from home. While that may appear to be a drastic change in your day-to-day operations, it need not disrupt your reliability and support operations. With the right incident response practices, teams can be just as effective remotely as they are on-site. Here are some tips to supercharge your incident response with Zenduty if you’re part of a remote operations team.

Tip 1: Set up an incident command channel

Since everybody is remotely communicating on Slack or Teams, your favorite team chat application is the obvious choice for being the “command center” of incidents. Now, depending on the service/component affected or the severity of the incident, you can setup outgoing integrations with Slack/Teams and send your incident data to a specific channel — let’s call it the “incidents” channel. You can also go for the multi-channel(one-channel-per-incident) configuration if you’re worried about noise in your channels. Once an incident is triggered, the on-call engineer will acknowledge the incident, look at the incident context and if necessary, add responders through the Zenduty bot. Zenduty will notify the responders via SMS/IVR/Push/Email/Slack/Teams and add the responders to the channel automatically.

Incident notification in Slack channel

Tip 2: Setup incident roles

If you haven’t already done so, now would be a good time to experiment with setting up an incident command system — a practice inspired by the US NIMS, that is followed by the SRE teams at Google, Dropbox, Fastly, Github, New Relic, Stripe and thousands of other experienced SRE teams globally. The goal of having incident roles is to have a recursive separation of responsibilities.

It’s important to make sure that everybody involved in the incident knows their role and doesn’t stray onto someone else’s turf. Somewhat counterintuitively, a clear separation of responsibilities — Google’s SRE handbook

Roles may include an Incident Commander(leads the incident), a Communications Lead(talks to stakeholders), Operations Lead(RCA, changes), DB Lead, Infra Lead etc. Setting up roles will help you effectively delegate specific tasks as well as increase transparency into your response operations. You can define the incident roles in Zenduty within your Team pages.

Team ICS roles

Tip 3: Setup Task templates/playbooks

Most teams have a 40–80% coverage of their incidents within their playbooks, which they host on a team wiki or git or something like Confluence. A major challenge with remote teams is ensuring that the playbooks are followed to the tee by responders. Take a good look at your incident playbook document and create a task template comprising of discrete role-mapped tasks. Each playbook you have can be converted into aa task template on Zendutyv. Once you create the task template, go ahead and map those task templates to your services. As the incidents start pouring in, Zenduty will automatically take the tasks from the task template and append them to your incident tasks. As the incident commander assigns the roles to various responders, Zenduty will assign the role-mapped tasks to the respective responders.

A sample checklist of tasks auto-assigned for incidents for a service

Task templates can dramatically increase your incident preparedness, decrease on-call anxiety, reduce RCA/response errors and improve response times.

Tip 4: Comms integrations(Jira, Zoom, Statuspage)

In the ensuing confusion of a major incident, setting up Jira tickets, Zoom rooms and Status updates might not be a priority. Luckily, adding outgoing integrations on Zenduty with Jira, Zoom, and Statuspage will certainly help to reduce the friction. Once you setup these outgoing integrations, Zenduty will automatically create a linked(two-way) Jira ticket, a Zoom room and update your Statuspage. The links to these comms channels will appear on your incident page as well as your Slack messages.

Tip 5: Setting up Alert Rules

Remote on-call teams will invariably face challenges when it comes to finding the right person or team to triage or assign an incident. One way to bypass such scenarios is to use Alert Rules to override the escalation policies or assign specific subject matter experts to an incident depending on the nature or source of the alert. Alert rules can also help you suppress incidents.

Bringing all of this together

The above tips can help remote teams supercharge their incident response, improve incident preparedness as well as give your team the right confidence to handle critical incidents. While working remotely for an SRE, ITOps, NOC, Support role might be overwhelming at first, be rest assured that it only gets better with every passing day.

As the world gears up towards fighting COVID-19, we wish the patients a speedy recovery, and we continue to be inspired by our healthcare workers and by others who are caring for people around the world.

5 reasons why Zenduty is a great alternative to Opsgenie, Pagerduty, and VictorOps

March 18, 2020

Amrit Balraj

Reason 1: Zenduty’s Custom Routing Rule(s)!

Every integration(with a monitoring source) within O/P/V sends alerts according to a single escalation policy that is linked to that monitoring source(service), no matter what the nature of the alert is, what time of the day or week it is triggered, what the severity of the alert is, which component is affected.

Pagerduty’s routing options

Paagerduty’s action options

For instance, if you have a Prometheus/Grafana monitoring system that is monitoring components built or maintained by multiple teams, and you integrate that alert source with O/P/V every alert generated by Prometheus will always go to the same escalation policy and alert a standard on-call rotations. If the on-call engineer does not have the expertise in dealing with the incident, he or she will have to bring in the team or the person that’s responsible for the component/service affected. While O/P/V might let you filter the alert, the custom actions are limited to suppressing the incident depending on the alert message.

But what if, depending on the alert data that the monitoring source sends, the alert also went to a specific team with the subject matter experts and the on-call engineer? Or what if the alert automatically bypassed the on-call engineer and went straight to the team? Enter Zenduty’s Custom Routing Rules.

Routing rules in Zenduty

  1. Payload Search — Every alert from a monitoring integration sends its own custom payload along with the alert parameters. For example, a critical alert from Jenkins will also contain a payload like — {“job”: “build-prod”, “job-number”:5, “status”: “SUCCESS”}. You can run a key search on the alert payload and use the above operators on the value
  2. Incident Message and Incident Summary — Route if you find a keyword in the alert message or summary
  3. Alert Time of the day — Route depending on the time of the day
  4. Alert Day of the week — Route depending on the day of the week
  5. Alert Date — Route if the incident falls on or between specific dates
  6. Alert Type — Route depending on the type of the alert(critical, error, warning, acknowledged, resolved and info)

Following are the actions you can take if the alert routing conditions are met:

  1. Suppress — Suppress the incident
  2. Add Note — Automatically add a not to the resultant incident that will help the on-call engineers and incident commander to determine the RCA and triaging steps
  3. Route to a custom Escalation Policy — Route to an escalation policy that is not the default on the service affected. This may be useful in cases where you might need a more aggressive or less aggressive escalation depending on the type of alert that has come in.
  4. Assign to a specific user — Assign the incident to a specific user. This is useful where a known component is affected and the fastest way to resolve the incident is by assigning it to a known subject matter expert
  5. Change Incident Urgency — change the urgency of the incident. If the alert creates a low priority incident and your conditions detect a potentially high priority incident, you can conditionally change the priority to high and vice-versa.
  6. Change Alert Message — create your own custom Incident Title from the alert payload
  7. Change Alert Summary — create your own custom Incident Summary from the alert payload
  8. Change Alert Entity ID — change the alert entity_id to implement custom deduplication
  9. Assign Role to User — after the alert creates an incident, assign a particular role to a particular user
  10. Assign incident tag — automatically add a tag to the newly created incident depending on the matching condition. For example, if a backend component is affected, add a tag “backend”
  11. Add incident task and template(coming soon) — Automatically add an incident task template or a task to the incident The Zenduty Alert Rules allow you to fine-tune your incident alerting at many levels and make sure that the right team members are alerted when specific conditions are met, add notes and tags, route the incident alert to a different escalation policy than the service default policy, customize the incident title and summary, and most importantly suppress the incident.

Reason 2: Zenduty offers a solid and frictionless path to incident resolution

Every critical incident has four main stages: The detection, the alert, the response(triage->investigate->mitigate), and the postmortem. While Opsgenie/Pagerduty/VictorOps(O/P/V) have done a great job with incident alerting, escalations, and on-call scheduling, they never really focused on the incident response side of things. It is no wonder that 79%+ of SREs feel anxiety and stress during and after an incident. What do you do once you receiving that dreaded page? For one, you can stop dreading those pages by improving your preparedness and by having a solid incident response process in place.

Companies like Google, Dropbox, Slack, and Shopify have pioneered the incident response by borrowing best practices from the Incident Command System(ICS) developed by the US National Incident Management System(NIMS). Here’s a video featuring Honeycomb’s Liz Fong Jones and Google’s Seth Vargo explaining the ICS practiced at Google. At Zenduty, we took a good look at the incident response best practices by the companies that we and the world admire for their reliability and brought those best practices into our product, which we collectively refer to as the Zenduty ICS.

There are four central pieces of the Zenduty ICS: Incident Roles, Tasks, Tags, and Task Templates. These four pieces combined will give anyone the entire high-level overview of the progress of any incident. Whenever an incident is triggered in Zenduty, the on-call engineer, after receiving the page, may acknowledge the incident and then claim the incident commander or ask someone else to take command. The incident commander will then add relevant responders, invite users to take up specific roles, assign tasks to those roles and update the incident status on the go.

Depending on your org structure and business, you can define roles like :

  1. Incident Commander — leads the incident, and assigns roles and tasks to different people
  2. Comms Lead — communicates about the incident to customers and stakeholders
  3. Ops Lead — making changes and pushing the fix

Incident Tags

Incident tags let you classify the incident based on component, service, priority, customers or any other segment. These tags can then be used in your analytics dashboard to measure tag-wise service, team, and user performance.

Incident tagging for better classification

Reason 3: Task Templates(Incident Playbooks)

A sample checklist of tasks auto-assigned for incidents for a service

The task template is the starting point for any incident. In the task template, you can define standard operating procedures and point them to specific roles. When a particular service encounters an incident, the task from the task template will automatically be added to the incident task list and after the incident commander assigns roles to specific individuals, these individuals will automatically inherit the tasks in the incident. At any given point, anybody in the team or company will know exactly who is leading the incident and who is dealing with specific aspects of the incident.

Reason 4: End-to-end incident management in Slack and Teams

Zenduty’s Slack and Teams integrations/bots allow you to manage your incidents end-to-end from your favorite chat application. Track your incident tasks, assign roles, fetch alerts and context, add responders and ack/resolve incidents.

Zenduty — Slack Incident Command System integration

Easy links for Incident communications

Reason 5: Fair and affordable pricing

At $0(Freemium), $5(Starter) and $15(Growth) per user per month, Zenduty is the most affordable alternative in the market. Zenduty offers you all the alerting capabilities offered by O/P/V in addition to powerful incident response capabilities. Compare with Opsgenie, Pagerduty, VictorOps.

If you’re looking for an end-to-end incident management platform with awesome alert routing and response management capabilities, do give Zenduty a spin and leave us your feedback in the comments below.

And finally, be zen!

Incident management with Microsoft Teams and Zenduty

March 12, 2020

Amrit Balraj

Teams is Microsoft’s versatile chat and collaboration solution for enterprise communication. Teams come bundled with Office365, offering chat, file sharing, and a host of other collaborative features. The platform also integrates with a host of popular project management applications, chatbots, and alert management platform makes it a hot favorite of production teams.

Organizations are realizing the potential of ChatOps for incident management and response. Teams integrate team-communication and incident response processes in one place, making it an incredibly useful tool in war-rooms. Teams can be seamlessly incorporated into incident management workflows and help support teams collaborate, track and manage IT incidents with complete visibility during critical events.

Some useful features of Teams:

  1. Robust internal chat capability with customizable channels and tags, best-in-class voice and video conferencing capabilities with support for up to 250 users. Teams also provide the feature of recording meetings making it invaluable in the post-mortem stage of incident management.
  2. Collaborative development platform with seamless integration with a host of web applications for application performance monitoring, incident management, data visualization, and analytical tools. Fits in the Office365 environment with 24/7 support.
  3. Customizable teams function, users can define teams based on internal departments. Teams consist of channels which is where the members can chat, have meetings, share files and collaborate on what they’re working on. Channels also have customizable tabs that can be used for quick access to important files, applications and collective information used by teams.

Zenduty is an end-to-end incident management application that unifies real-time alerts from 100+ application performance monitoring, data visualization, analytical and support systems and helps manage incidents, delegate tasks from within Teams and meet SLAs.

Integration features include:

  1. Fully customizable notifications with on-call schedules implementation features.
  2. Custom escalation policies to ensure incidents are acknowledged and resolved within SLA limits.
  3. Pre-definable incident roles and task templates to ensure effective delegation of response actions and elimination of chaos.
  4. Personal and channel alerts for service-level incidents.
  5. Intelligent alert context to accelerate RCA.
  6. Customizable alert routing based on on-call schedules.
  7. Response automation for building self-healing systems.
  8. Incident tags for classification of incidents.

Find the documentation for Zenduty+Microsoft Teams integration here.

Microsoft Teams is an excellent collaboration and communication platform with a plethora of features designed for organized teams to maximize productivity and streamline workflows. The Zenduty+Microsoft Teams integration will help teams leverage incident response and reduce background noise during highly critical events, reducing MTTR and consequently strengthening customer trust.

Zenduty is a cutting edge incident management platform designed by developers keeping the well-being of engineers in mind. Sign up for free here.

Application peformance monitoring with Datadog 2020

March 10, 2020

Amrit Balraj

Datadog is an application performance monitoring and analytical SaaS for cloud infrastructure. Datadog enables DevOps teams, SREs and IT operation teams to optimize their systems for uptime and availability. Modern services generate massive amounts of data from all of the different services and technologies, Datadog supports over 400+ integrations and collects data for improving visibility across dynamic production environments.

Every business, irrespective of the vertical and nature of business, does a substantial amount of their business online and consequently is dependent on uptime and network health of their applications. According to a report in 2018 by IDC, sponsored by data-storage company Seagate, more than 175 zettabytes of data will be created in 2025. Enterprises are projected to generate 80% of that data. Not all data will be stored or tracked, but enterprise will certainly see a growing data pile needing to be addressed from a variety of angles, including what Datadog offers. So, the relevance of its service is only increasing. Datadog offers end-to-end tracking of performance metrics, log files and a horde of other information on fully customizable dashboards. Datadog’s adaptable dashboards are designed to track, search and visualize vital metrics in beautiful graphs that instantly provide insights to teams. The user can build a real-time interactive dashboard, collaborate, and optimize application performance.

One of the most distinctive features of Datadog is the options visualizations of alerts via colored hexagonal elements, each of them representing the service or container being monitored. These hexagons change colors based on the severity of alerts coming in. Users can click on one to instantly view the cause of the event.

Datadog, like any good APM, comes with customizable alerting rules, based on real-time changes.It also includes the ability to integrate with advanced teaming and alert management applications such as Zenduty. Users of Datadog can integrate over 350+ built-in integrations with open-source solutions, cloud native monitoring services and log management applications. These integration connectors provide the ability to monitor code changes, track administrative actions, view performance of physical devices or apps, or even facilitate communication in ways that make the most sense for your business.

Datadog is a comprehensive SaaS solution for companies looking for real-time monitoring tools, particularly those with the analysis and development requirements which can leverage the most out of the data being tracked and visualized.

Zenduty is a cutting edge incident management platform designed by developers keeping the well-being of engineers in mind. Sign up for free here.

Incident Response - how great companies do it

February 12, 2020

Amrit Balraj

An incident response plan is a pre-devised action stratagem for IT teams on how to respond to critical IT events efficiently. As modern applications continue to grow in scale and complexity, there will be more people working on more interdependent systems, consequently, the question is not if a system will fail, but when, and how best to respond. A part of what makes systems more complex is customers demanding 24x7 availability from businesses today, reliability has become a competitive edge in an always-on world.

Critical events are high-pressure situations with high stakes, incident response processes help curtail some of the pressure. A well-rounded incident response plan ensures that members of incident response teams can organize all facets of incident response (stakeholder communication, public response, etc) while also focusing on the resolution of the issue. The primary goals of efficient teams should be to curtail damage, reduce MTTR, and bring down the loss of revenue.

The four aspects of incident management to keep in mind while formulating an IRP are: detect, acknowledge, triage and learn.

Detection- Identifying the issue is the first step, communicating effectively is the second. Teams that detect incidents early usually end up resolving them before it snowballs into a high-level catastrophe. Not all obscure incidents can be given individual attention, which is where log data comes in, a good log management application is essential to the tool kit of every production team. Log data can be analyzed after incidents to find out if there were minor indicators before something crashes and also tracked for the future. Log monitoring can be integrated with a good alert management application for instantaneous alerts when there are irregularities in the data.

Acknowledgment- Teams that detect issues quickly but ultimately cannot get the message across the right people fail to thrive. According to a SysAid survey named the future of ITSM, 92% of people in IT departments feel uninvolved in their organization’s DevOps activities due to lack of proper communication between teams. Notification chatter from different quarters of an organization during a major incident can make processes chaotic and slow. Breaking down team siloes through effective communication is still as relevant as ever, especially for incident management. Establishing communication protocols ensures that information is relayed through predetermined channels to principal stakeholders, business leaders and vested customers putting them at ease ensuring that core technology team members can focus on fixing the issue at hand without hindrance. The use of an alert management platform helps streamline the communication process and guarantees that the right subject matter experts are notified at the right time.

Resolution (triage)- Write, run and constantly update the game plan. Large companies like Netflix and Amazon regularly test the stability of their systems by deliberately injecting chaos. This helps teams stay fresh and improve their response times. Do not let your documentation get stale and have refresher sessions when new members of the team join so that everyone is aware of the plan. Effective games plans should also have:

  • Incident roles: The importance of this cannot be stressed enough, having predefined incident roles like Incident commander, communication lead, etc ensures no contradictory decisions are made during the storm, there is bolstering of stakeholder trust and improved team cohesion.
  • Well defined escalation policies: Based on the system being monitored, automated escalation policies will have to be defined for notifying one or several people. Customizable escalation policies that can easily be configured with schedules for optimum speed.
  • Task Templates: Templates ensure that every member of the team, even in the absence of senior, more experienced guidance will have an idea of how to go about getting started.

Learn- Blameless post mortems help teams understand the root cause of major incidents without finding fault with individual members. Blamelessness encourages people to come forward and talk about their mistakes without fear of repercussions. This fosters an internal culture of trust and learning.

Google’s famed SRE book highlights what are the types of triggers that warrants a detailed study:

User-visible downtime or degradation beyond a certain threshold Data loss of any kind On-call engineer intervention (release rollback, rerouting of traffic, etc A resolution time above some threshold A monitoring failure (which usually implies manual incident discovery)

Pioneers of blameless post mortems, Etsy encourages organizations to document everything. The discussion sessions during post-mortems will bring about new perspectives and ideas on how to best implement changes to strengthen stability. To this end they have even created a tool called Morgue to help production teams with the documentation process. It allows for adding information about the event with graphs, timelines, log data and communication screenshots.

Incident management runbooks are unique to every organization, factors may include tools being used, number of teams and type of business. Finding a one size fits all plan is not possible, organizations need to be open to change, to constantly learn and upgrade. Document everything and unify data from all services for maximum overview.

Zenduty is a cutting edge incident management platform designed by developers keeping the well-being of engineers in mind. Sign up for free here.

Monitoring with New Relic- Everything you need to go to get started

January 27, 2020

Amrit Balraj

DevOps is an organizational philosophy that enables continuous delivery and continuous deployment with a focus on continuous testing, automation and collaboration among dev teams, business, and operations teams. Consequently, continuous monitoring is also a key phase of the DevOps lifecycle, which is where application performance monitoring tools come into the picture. APM tools enable developers to monitor user experience in real-time with an eye on the health and stability of their applications. They enable tech teams to make informed decisions based on performance metrics and also improves observability which is integral for DevOps teams.

DevOps teams use APM tools early in the product lifecycle to identify potential threats to stability and resolve them before production. New Relic is one such application performance service designed to monitor web applications in real-time. New Relic makes troubleshooting, analyzing and scaling a painless process for engineering teams.

New Relic is designed to help modern production teams move faster without fearing high-cost downtimes. With New Relic Insights, users can assimilate data from multiple sources and analyze customer trends, create metric charts and create customizable dashboards.

Notable features of New Relic:

Users gain deep insight into the performance of their applications through customizable data-rich dashboards with New Relic Insights. Business-specific attributes can be added to each dashboard giving developers the ability to fine-tune their applications through analysis of indexed data. New Relic also offers detailed tracking of key transactions that are important to the organization. The platform enhances observability across teams which is a key aspect for modern DevOps teams.

Scalability analysis is a highly important factor, especially for startups, where a sudden increase in traffic can cause a site to crash. New Relic enables production teams to analyze site traffic and conduct load tests on their applications to ensure smooth scalability. Modern systems are dynamically changing, New Relic ensures that users can keep an eye on response times and can predict when to upgrade capacity.

Alerting is another important aspect of any APM . New Relic allows users to configure alerts when there are errors causing downtime of latency issues. Apart from customizable email or text notifications, New Relic can be integrated with alert management tools like Zenduty , VictorOps, and Xmatters.

New Relic is a feature-rich performance monitoring software that comes with a bit of a learning curve to familiarize. This article just scratches the surface of the potential that New Relic offers engineering teams.

The company has expanded its number of solutions beyond its flagship APM product and offers additional plug-ins for a host of enterprise requirements. One of the best features of all, you will start seeing metrics populating a dashboard within minutes of set. Users can also start a deep analytical X-Ray Session to start troubleshooting and optimizing your functions.

Grafana- Everything you need to know

January 21, 2020

Amrit Balraj

Grafana is an open-source platform for data visualization, monitoring, and analysis. It's designed around providing context-rich visualizations, mainly though graphs but also supports other ways to present data through pluggable panel architecture. Every dashboard is versatile and custom-buildable for specific projects of software development or business requirement. Grafana’s beautiful dashboards are one of the reasons Grafana is so popular with users. Visualizations in Grafana are called panels, and users can create custom dashboards with panels for different data sources. Grafana supports graphs, tables, heatmap and freetext panel types. Grafana users can also make use of a well established ecosystem of user-made dashboards for different types of data and sources.

In 2020, even small organizations are generating enormous amounts of data which makes tools for monitoring and analyzing big data integral to every workflow. Grafana is a popular solution due to its ease of use, visually pleasing graphs and ability to integrate with a variety of databases like Graphite, Logz.io, Influx DB, MySQL, PostgreSQL, ElasticSearch etc. An added benefit of it being open source is that developers can write custom plugins based on their requirements.

Some of the cool features that Grafana users love are:

Customizable dashboards- Grafana dashboards are feature rich and can be configured to display data from a variety of databases using varied visualization options like histograms, heat maps, and/or charts. Users love the flexibility and use of use that Grafana offers.

Native support for a wide variety of databases. Grafana is also open source with support for custom coded plugins based on project requirements.

Cloud monitoring- Cloud solutions like AWS have built in data gathering infrastructure which makes Grafana the perfect solution for dashboard creation and visualization.

Some of the cons of using Grafana are:

Lack of storage options - Grafana is strictly a visualization tool and lacks storage as a part of it’s core functionality. Prometheus is way ahead of Grafana when it comes to storage functionality because of it’s dimensional model, however Grafana can be integrated with Prometheus quite easily.

Limited alerting and incident tracking capabilities- Grafana doesn’t have well defined alerting or incident tracking capabilities as it’s functionality is more concentrated on data visualization.

Some users find that the initial set up of Grafana is confusing when compared to other visualization tools like Kibana or Knowi.

Some real world use cases for Grafana are:

DigitalOcean used Grafana to upgrade their aging, disparate data visualization tools across all of their teams to improve collaboration. Grafana offered a cost-effective easy to use data visualization solution.

Stack-Overflow required a platform to generate self-service dashboards to visualize data from OpenTSDB, Elasticsearch, and their alerting tools.

Grafana also has Grafana Enterprise designed for large organizations and their enterprise data visualization requirements. Grafana Enterprise includes 24x7 customer support and training from the core Grafana development team.

Grafana is the ideal solution for visualizing system data. Streamlined installation and configuration ensure accessibility and observability for all users. The beautiful and customizable interface makes the experience of monitoring application performance rewarding and empowering for all users.

Find the guide for Zenduty+Grafana integration here

Site reliability engineering- Predictions for 2020

January 10, 2020

Amrit Balraj

As we head into 2020, it's clear that DevOps has finally crossed the divide and gone mainstream. With DevOps firmly ingrained as a standard practice, we now look at how it will evolve. DevOps is driving more overall alignment between development and operations teams than has ever existed in the past. For developers, that means building and delivering impeccable apps to market quickly. Operations teams can focus on a more Ops-centric approach for ensuring streamlined product releases and post-production experience for customers. The two groups can now more readily focus on the things they do best while believing that the other team is in sync.

As agile development methodologies continues to grow and organizations shift focus to improve quality without compromising speed, visibility across the software development life cycle will become of prime importance. In 2020, teams will increasingly be dependent on software that proactively monitors, and even anticipates, potential incidents. Let’s look at the top trends which will influence the world of tech this year.

Kubernetes

According to The State of Kubernetes Adoption and Security report published by Alcide which is based on 200 responses from DevOps engineers, cloud architects, and security professionals, 45% of companies are now running Kubernetes. As more organizations adopt Kubernetes the demand for experienced engineers is on the rise. The rapidly evolving landscape of Kubernetes also means that pros, as well as beginners, have a lot of learning to do in 2020.

SRE

Site Reliability Engineering, coined by Google engineers in 2003, has been around for 15 years or more. However, the technological ideology is still relatively misunderstood. As per the report linked above, based on a survey of 118 SREs globally, 64% of respondents indicate the SRE role has only existed in their companies for 3 years or less. With automation and observability becoming a key factor for more efficient and rapid deployments, an SRE job profile has become one of the most sought after in tech this year.

Microservices

Microservices architecture will become even more mainstream in 2020 as evident in this Google trends graph. The main reason for this is that organizations are constantly pushing the envelope developing complex software and microservices that offer developmental speed, seamless scaling and smaller release cycles in a highly competitive marketplace. Successful companies like Spotify, Amazon, and Netflix have achieved long term growth and sustainable workflows through early adoption, which is something to bear in mind this year.

Serverless

Serverless architecture is helping developers to focus on their applications without having to manage servers. Companies are turning to FaaS and BaaS in 2020 as they ultimately provide business value through billing based on consumption while building resilient systems. Serverless computing, on a greater scale, is also about reducing your carbon footprint through limited usage which makes it more sustainable for the planet in the long run.

Security

Security breaches continue to disrupt service and raise costs for companies on a global scale. This is leading to more companies embedding security into DevOps practices (or DevSecOps) to counter breaches in their defenses. DevSecOps is about implementing security early in the development process and including everyone is aware of best practices.

Microservices and cloud native applications offer unique vulnerabilities, understanding them is crucial to be better prepared for the future. DevSecOps improve collaboration across teams so that there is improved visualization and observability.

This is by no means an all-inclusive list of predictions as software development is quite large and complex. It is essential for new and long-time organizations to stay aware of the ever-changing landscape of technology and adopt them as per their requirements. One of the great things when looking ahead is seeing all of the choices we have for custom building your path to growth.

Be Zen.

Incident Alert Routing — Getting woken up only by alerts that matter to you

January 1, 2020

Vishwa Krishnakumar

Incident Alert Routing — Getting woken up only by alerts that matter to you

Site reliability engineers have one of, if not the, toughest roles in any organization. While dealing with incidents is one part of the job, the other is to build reliable systems. Google’s SRE book sums this approach nicely.

At least 50% of each SRE’s time should be spent on engineering project work that will either reduce future toil or add service features. Feature development typically focuses on improving reliability, performance, or utilization, which often reduces toil as a second-order effect. We share this 50% goal because toil tends to expand if left unchecked and can quickly fill 100% of everyone’s time. The work of reducing toil and scaling up services is the “Engineering” in Site Reliability Engineering. Engineering work is what enables the SRE organization to scale up sublinearly with service size and to manage services more efficiently than either a pure Dev team or a pure Ops team.

One of the most important challenges for an SRE when it comes to balancing work between firefighting and toil reduction is the issue of alert noise. An alert, for a particular engineer, can be considered as noisy if the alert does not provide any information about an event that has already happened, or an event that is about to happen, that significantly affects the end-user or customer, and the alert is not actionable on the part of the said engineer. Alert noise, not only distracts your engineers from their engineering duties but also causes needless stress, increases anxiety and exacerbates mental and physical health issues that your engineers may already be facing. It will come as no surprise that organizations with the noisiest alerts also have the highest engineering churn rates in the industry.

Every Monitoring or IT management tool sends a lot of alerts daily. While some alerts are important and actionable, most of them are noise. Nobody likes to wake up to an alert at 2 AM in the morning unless it’s a major incident or an outright catastrophe. We took a close look at the incident management services and how they help teams reduce alert noise. We were surprised to see that all of them have implemented just bare-bones features to mitigate alert noise, and despite being in the market for over 7–10 years, have failed to see alert noise reduction as something that needed to be solved in a meaningful way.

How Zenduty can help you shrink a river of alerts to a trickle

When we started building Zenduty, solving the problem of alert noise was top on our list of priorities. And so we did, with Alert Rules. Following are the criterions on which you can route your alerts on Zenduty along with operations like <, ≤, >, ≥, ==, !=, contains(substring), not contains, regex match, is empty, is not empty, in(multi-select), not-in(multi-select), between and not between:

  1. Payload Search — Every alert from a monitoring integration sends its own custom payload along with the alert parameters. For example, an critical alert from Jenkins will also contain a payload like — {“job”: “build-prod”, “job-number”:5, “status”: “SUCCESS”}. You can run a key search on the alert payload and use the above operators on the value

  2. Incident Message and Incident Summary— Route if you find a keyword in the alert message or summary

  3. Alert Time of the day —Route depending on the time of the day

  4. Alert Day of the week — Route depending on the day of the week

  5. Alert Date — Route if the incident falls on or between specific dates

  6. Alert Type — Route depending on the type of the alert(critical, error, warning, acknowledged, resolved and info)

Following are the actions you can take if the alert routing conditions are met:

  1. Suppress — Suppress the incident

  2. Change Alert Type to —Let’s say the alert type of the incoming alert is critical(which will create an incident), and you do not want the alert to create an incident, you can change the alert type to “info”.

  3. Add Note — Automatically add a not to the resultant incident that will help the on-call engineers and incident commander to determine the RCA and triaging steps

  4. Route to Escalation Policy — Route to an escalation policy that is not the default on the service affected. This may be useful in cases where you might need a more aggressive or less aggressive escalation depending on the type of alert that has come in.

  5. Assign User — Assign the incident to a specific user. This is useful where a known component is affected and the fastest way to resolve the incident is by assigning it to a known subject matter expert

  6. Change Incident Urgency — change the urgency of the incident. If the alert creates a low priority incident and the your conditions detect a potentially high priority incident, you can conditionally change the priority to high and vice-versa.

  7. Change Alert Message — create your own custom Incident Title from the alert payload

  8. Change Alert Summary — create your own custom Incident Summary from the alert payload

  9. Change Alert Entity ID — change the alert entity_id to implement custom deduplication

  10. Assign Role to User — after the alert creates an incident, assign a particular role to a particular user

  11. Assign incident tag — automatically add a tag to the newly created incident depending on the matching condition. For example, if a backend component is affected, add a tag “backend”

  12. Add incident task and template(coming soon) — Automatically add an incident task template or a task to the incident

The Zenduty Alert Rules allow you to fine-tune your incident alerting at many levels and make sure that the right team members are alerted when specific conditions are met, add notes and tags, route to a different escalation policy than the service default policy, customize the incident title and summary, and most importantly suppress the incident.

If you’re looking for an end-to-end incident management platform with awesome alert routing and response management capabilities, do give Zenduty a spin and leave us your feedback in the comments below.

And finally, be zen!

Next →