Relationships between Operation and Devlopment Teams

October 16, 2019

Modern businesses are evolving rapidly with the advent of cloud, CI/CD and microservices. However, there still exists an extensive and obvious divide between principle business stakeholders and developmental teams. Development teams are often unaware of the challenges faced by operations teams and vice-versa.

This is where a need for adoption of DevOps principles comes into the picture. DevOps which came into existence as the natural successor to Agile practices in software development. For a versatile delivery process that is adaptive to continuous change and delivery, there is a need for the introduction of the agile concept of continuous feedback. Continuous feedback is a way where negative and positive feedback is accepted and implemented to production constantly.

For instance, daily standup meetings allow members of the development team to share status updates and highlight challenges in the way. Additionally, the sprint review meeting is the opportunity to deploy changes to a broader user group, including executives and end-users. In addition to evaluating goals for the project, the group can provide feedback on the current solution and the unforeseen needs before the next sprint meeting.

Feedback loops have to be a combination of agile best practices, automation tools, and efficient documentation. Testing is also an integral part of developing fast feedback loops. Implementing DevOps best practices will add to the overall worth of an organization, including the following:

Efficient code delivery
Quicker deployment to market
Higher-quality in output
Improved collaboration between developers and operations
A culture that unites business, development, and operations together for excellent responsiveness to market demands

To become necessary to customers, operations and developers need to be a collaborative unit. Development teams are required to design, deploy, test and troubleshoot their software swiftly and authentically. Operations teams are compelled to monitor and identify issues before they happen. The adoption of DevOps culture is an organizational change that requires a shift in mindset primarily. When successful communication channels are established between managers, developers, and operations through DevOps practices, there will be lesser conflicts of interest within teams, and an increase in overall quality and speed of delivery for the customers.

Zenduty is a cutting edge incident management platform designed by developers keeping the well-being of engineers in mind. Sign up for free here.

ChatOps-The future of collaboration

October 7, 2019

Amrit Balraj

ChatOps is the implementation of chatbots to unify communication and collaboration. Through ChatOps every single member of a team will be aware of what the other members are working on. It is the logical next step in the evolution of communication among teams after email and IM.

Projects of today are developed at a global scale with millions of people as potential users, this means that teams are larger and often work in shifts or even remotely. ChatOps tools can also be integrated with other tools of monitoring, diagnosis, notifications, and data mining to make any team more efficient and focused.

Github opened to door to the adoption of this concept way back in 2011 when they introduced Hubot . Initially, it was developed as a way to automate their internal company chat room, today it is an open-source highly customizable chatbot with custom scripts and support for a host of plug-ins. Jesse Newland, principal product engineer at Github defined Hubot as:

“Putting tools directly in the middle of conversation”

The major advantages of ChatOps can be summarized as:

Automation: RepetiSStive manual tasks can become tedious and consequently prone to error. Automation of tasks frees up a lot of precious man-hours, which will increase the overall productivity of the team. This is especially valuable in incident monitoring and initial response. ChatOps ensures a seamless communication channel where everyone can be brought up to speed on what updates are happening and where they can help.

Visibility: With ChatOps every team member’s work is in full view. If your team is distributed with different shift timings and/or locations chatbots make the workflow transparent. This benefits CI as workers can focus their time on day to day without trying to find out who introduced what and when

There are plenty of powerful bots in the market which are customizable as and how your team functions as a whole. Some of the more popular ones are:

Hublot
Err- developed with python.
Lita- developed with ruby.

Teams are always evolving to get faster, more productive and dynamic, ChatOps gives them tools to leverage their time. Through the integration of ChatOps platforms, you can have a unified communication channel where everyone can see what the workflow is. This is especially crucial in incident management as everyone can see the changes, fixes, and updates across the board.

Post Mortems- Bringing clarity to incident reviews

October 3, 2019

Amrit Balraj

An incident post mortem is known by many names- incident review, root cause analysis (RCA), learning review, but what do they entail?. A post mortem is a post-incident activity to help organizations understand how the incident happened and to learn from it. Service incidents are an unavoidable hurdle for any company when they do happen, the teams working will be wholly focussed on restoring service as quickly as possible.

Most times they don’t have the time and find the root of the problem as they focus on fixing the problem at hand. Organizations for the most part always have post-mortem meetings to assess what happened once a service has been restored. An incident resolution time-frame only ends once the entire team participates in the post-mortem meetings.

The entire team needs to participate in post mortems as people who worked on the frontlines of incident management can share their insight from experience and increase the collective knowledge of the entire team. Post mortems tend to be structured differently depending upon the organization and there is no single playbook to help you run them efficiently. Teams need to view post mortems as a learning exercise to understand what they have to do to avoid such an issue in the future.

Here are some guidelines to help your company conduct streamlined post mortems:

Document Everything : Post mortems are not just meetings where people talk, it is a record of everything that was done during an incident. The documentation from these meetings will be reference summaries for future teams to help them resolve downtimes when they happen. If all the steps taken while the incident is happening are documented immediately after it happens, it will serve as a game plan for engineers down the line when they are also facing an incident.

Devise a system to ensure that this documentation is available to the entire team for transparency and preparedness. This will serve as effective for stakeholder communication as well.

Foster Blamelessness: Industry leaders like Google talk about this extensively in their Site Reliability Engineering handbook, the importance of blamelessness during post mortems. Post mortems are time-consuming processes and need to be freed from unnecessary assigning of blame on individuals who may or may not be at fault. This creates an environment of fear with people unwilling to come forward and explain where they went wrong. Studies have shown that this negatively impacts the growth of an organization in the long run.

Blamelessness encourages team members to talk about the actions they took, assumptions they made and establish timeframes. This also builds trust among team members where they can openly talk about their concerns without fearing repercussions. A trusting team contributes to the strength of the system as a whole.

Track Post mortems: As companies perfect their incident review game they build up a database of valuable documentation for future incident response professionals. This can benefit current teams as well, especially if the tracking system has been set up for quick access during times of crisis.

Over time they can be reviewed by executives to understand patterns where potential weakness can be exploited.

Every technology worker knows that downtimes are unavoidable, they come when least expected, leaving behind a trail of chaos. Post mortems help teams to view incidents calmly without the stress of an ongoing incident. There are dozens of publications and handbooks by industry leaders to help organizations with best practices in post-incident reviews. ../

Zenduty is a cutting edge incident management platform designed by developers keeping the well-being of engineers in mind. Sign up for free here.

The importance of Incident Roles

September 30, 2019

Amrit Balraj

Modern technology organizations are required to be adaptive in their approach to incident management. A single project will have multiple teams working as different branches on integrated systems. Even if all the members have unified communication channels when an interruption occurs in the service there’s bound to be chaos. The frontline response team will have to be on their toes to get to the root issues at the first signs of trouble.

According to IBM’s Cost of a data breach report 2019 the average cost of an outage globally was 3.92 M. The US holds the highest average cost at $ 8.19 million. Any disruption in service is a potential breach in SLA agreements and puts a company’s profitability at stake. The report further highlights the long term costs of an incident linger for up to 2 years after an incident.

An integral part of any good response plan is preset incident roles. Incident roles ensure that the responsibilities are made clear pre-event to eliminate noise and consequent chaos. Team leads who manage day to day operations of the service will be aware of which team member will be effective in front line response roles.

The guidelines for developing a framework for incident response are:

Encourage on call engineers to take initiative to delegate important tasks to team members based on their core strengths.
Establish roles pre incident and test out scenarios with different team members playing roles they are comfortable with.
Keep your stakeholders informed with regular status updates. They need to know if an issue is ongoing or has been resolved.

With incident roles predefined, your IRPs can be adaptive and reiterated as you face new challenges and learn from experience. With Zenduty you can automate your IRP by defining incident roles before an incident which will put an end to chaos when an outage occurs. You can also create task templates as per your choice and assign them to your incident roles making incident mitigation a seamless procedure.

In the long run these steps will make your IRP more transparent and open to evaluation for strengthening your system as a whole. You can safely make decisions to streamline your service delivery and reduce the effects of an incident or outage.

Zenduty is a cutting edge incident management platform designed by developers keeping the well-being of engineers in mind. Sign up for free here.

Fostering blamelessness at the workplace

September 20, 2019

Amrit Balraj

An integral lesson every business (of any size) learns is that failure is inevitable at some point in the production cycle. There might be times where things go haywire at critical junctures sending teams scrambling to rectify the root issue and reinstate service. The underlying causes are often many and varied especially in large scale systems with complex architecture and interdependence. The initial response of any supervisor or manager is to figure out which individual of their team messed up and assign blame. Consequently the hapless team member may be reprimanded or reassigned based on the severity of the issue.

Is this the best way to handle an incident with team members?

In reality this type of blame assigning is actually counterproductive and will contribute to creating a workplace culture of fear. John Lunney and Sue Lueder who are senior site reliability engineers (SREs) at Google wrote the following in their paper “Postmortem Culture: Learning from Failure”

“ Blameless postmortems are a tenet of SRE culture. For a postmortem to be truly blameless, it must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior. A blamelessly written postmortem assumes that everyone involved in an incident had good intentions and did the right thing with the information they had. If a culture of finger pointing and shaming individuals or teams for doing the "wrong" thing prevails, people will not bring issues to light for fear of punishment.”

Team individuals fearing retribution for past errors will hesitate to escalate issues which ultimately harm the company in the long run. Fear leads to stress, stress leads to poor choices, leading to further chaos down the road.

How do we ensure that our team is performing unrestrained by fear?

Fostering blameless post mortems in the workplace is the key to understanding problems when they arise, getting on top of it and avoiding bigger problems from arising. The other great advantage of this is that it builds trust within teams which ensures they get the support they need to build great projects.

There are some ground rules:

Words have inherent power, instead of asking who/why/when, ask how an incident happens. Write down results from discussions to increase visibility across the team. This will help establish workflows and worm out system weaknesses.
Build a safe space for people who made the mistake to step up and talk about the mistakes they’ve made. This will instruct the rest of the team on how to avoid making the same mistake in the future strengthening the system as a whole.
Make sure the entire team understands blamelessness in the workplace. Conduct meetings to help them grasp the concept and understand that failure is a unique opportunity to learn.

Fundamentally it all comes down to creating an environment of trust and identifying areas for improvement across the board. This will also promote transparency within the organization and exponentially increase incident handling. At Zenduty we believe in creating an environment where every employee is treated with respect and incidents are viewed as opportunities for making our teams stronger.

Zenduty is a cutting edge incident management platform designed by developers keeping the well-being of engineers in mind. Sign up for free here.

The true cost of unreliability

September 13, 2019

Amrit Balraj

Every organization is different in the way it functions as a whole, with different approaches to operations management, functionality and legal structure. However every company whether big or small face unplanned downtimes from time to time.

There are multiple examples of large companies taking major hits due to an unintentional outage. An example of a significant crash of services is the Amazon Prime Day crash of 2018 which is estimated by Axios to have cost them $72m-$99m in lost sales. Prime members were unable to log on to the site to participate in the lightning deals of the day, causing a customer service nightmare along with a deep hole in their pockets.

If your company has an annual turnover of $1m, outages can cost from upto $1000-10,000 per hour of downtime. A study undertaken by software company ITIC displayed that one hour of downtime costs 98% of large enterprises more than $100,000 per hour of downtime. The costs multiply if downtime is experienced by business critical service components like payment, support, on boarding etc. These issues, if not detected and resolved immediately, can go from being a minor problem to potential public/customer relations disaster. Companies which deal in financial services, energy and data security are often the worst hit after a data outage. Always remember - Reliability has a direct correlation to your company’s growth, brand image and bottom line.

What are some strategies to improve the reliability of the services offered by your company?

Everyone prefers having their services smoothly around the clock, but outages are all part of the game. Your team might be building the best platform in the business with complex code and interdependent systems, but it pays to be cautious in the long run (literally).The most time tested methods to ensure that incidents don't take a chunk of your income is to:

Formulate realistic SLA goals, define controllable SLOs for uptime over a specified time, and establish clear SLIs and error budgets
Use monitoring and measuring tools to understand factors like rate of deployment, mean time to recover (MTTR) and quality assurance.
Build a solid CI/CD pipeline, standardize the deployment process, and automate testing as much as possible.
Construct an iron-clad incident response strategy with clear roles and responsibilities, incident checklists, automations and communication channels.
Learn from downtime - conduct blameless postmortems of critical incidents(when/what/why/how) and institutionalize best practices within your teams.

We are building Zenduty to help your company will be ahead of the curve when it comes to reliability and support. Zenduty serves as a single source of truth all of your alerts to help notify the right people minimizing confusion. Working with predefined parameters, Zenduty will you help your teams monitor the incident timeline increasing communication and visibility.

Zenduty is a cutting edge incident management platform designed by developers keeping the well-being of engineers in mind. Sign up for free here.

Real-time incident management with HetrixTools

August 1, 2019

Akshaya Krishnan

Incident management works best when all of your incidents and alerts can be tracked from a centralized hub. HetrixTools is a blacklist check and monitoring software that monitors if your domain or IP get blacklisted and enable to take action instantly.

The Zenduty-HetrixTools integration allows you to sync your HetrixTools tickets with Zenduty, and keep up with all the updates. The advanced monitoring allows you to keep a bird's eye view on all the updates.

How to manage Incident Response Efficiently

July 29, 2019

Jaya Jain

Organizations try to be proactive about incidents and down-time. An outage or downtime can be devastating to the bottom line of businesses, not to mention a poor experience for their customers and users.

Real-time incident management with Ghost Inspector

July 23, 2019

Akshaya Krishnan

Incident management works best when all of your incidents and alerts can be tracked from a centralized hub. Ghost Inspector helps you build or record automated website tests in your browser and continuously monitor websites for issues.

The Zenduty-Ghost Inspector integration allows you to sync your Ghost Inspector tickets with Zenduty, and keep up with all the updates. The advanced monitoring allows you to keep a bird's eye view on all the updates.

10 Superfoods to Boost Productivity

July 19, 2019

Akshaya Krishnan

Scientists have discovered a direct link between foods consumed by people and their productivity levels. This forces us to rethink our dietary habits and consume superfoods that are rich in vitamins and protein.

Here is a list of some superfoods that we can consume to increase our productivity.