CIO

What CIOs can learn in the wake of a major IT outage

The impact of system outages brings to light the stark reality that as cloud continues to evolve, there are serious challenges to its true resiliency to withstand unforced errors
 

Outages from Australian organisations across the financial, government, travel and telecommunications sectors seem to be occurring on a weekly basis often caused by hardware failures, software upgrades, human error and ransomware attacks - causing major service interruption.

The impact of these outages brings to light the stark reality that as cloud continues to evolve, there are serious challenges to its true resiliency to withstand unforced errors.

From reputational damage to lost revenue, lost productivity or other measures – one thing is certain, no business can afford an outage in today’s ultra-competitive business environment. There has never been a better time for CIOs to take a cold hard look at their business continuity and disaster recovery (DR) plans to see if they are truly prepared.

The fact remains, that the cost of investing in a truly resilient disaster recovery (DR) platform is exponentially less than the cost of having to fix the situation after the fact. To do this, increasingly CIOs are seeing the true value in having a layered approach to their cloud strategy - meaning a secondary (or more), geographically and meteorologically, off-premise recovery data centre. This ensures that should anything happen to your primary site, you will always have a secondary location to avoid or reduce the impact of an outage.

The cost of downtime

In a world where we increasingly rely on IT capabilities, with much of it supported in the cloud, some businesses are incredibly vulnerable to any down time, and can find themselves with their entire website, and consequently their business, offline for a significant amount of time.

Some business applications will be the hardest to have these multiple layers as they are designed in place and are not designed for portability. To solve this challenge, and have the right safety nets in place, many CIOs are now looking at how to build a more hybrid cloud, leveraging a managed service provider or their own data centres.

Downtime can have disastrous implications for a business, both financially and reputationally. In fact, Gartner estimates that on average, every minute of downtime will cost a business $5,600, which adds up to over $300k per hour.

Diversity adds to the resiliency-in-layers effect

Having a DR plan alone is not enough. Again, resiliency-in-layers is key here when it comes to business continuity. This means examining your vendors, locations and technologies to understand how to make this all heterogeneous.

Having diversity adds to the resiliency-in-layers effect by separating one action, activity, bug or catastrophic event from impacting the rest of the business environment. And you must test that plan regularly, to ensure that if you are hit, you avoid downtime by having the automation muscles built into your plan.

A successful DR infrastructure needs to be highly automated and continuously replicate data, allowing for applications to be quickly “rewound” to the seconds just before an outage. It must be able to meet recover point objective (RPO) defined by the business, with little to no loss of data or loss of application availability. Even a few seconds can cost you tens of thousands of dollars either in the way of lost revenue from an application being down and unable to transact or incurring fines from a compliance failure.

IT resilience – The case for hybrid cloud

Every CIO has their own organisational requirements. Some have compliance challenges and others may have data locality issues. For this reason DR plans can sometimes seem as unique as a fingerprint in how they are built, maintained and where they recover too. While IT is clearly moving towards cloud-based infrastructures, the centerpiece of this trend revolves around the ability to thrive through every permutation of a disaster being more than just natural causes, but even common power failures and human error.

Although each element within hybrid cloud has its own associated strengths and weaknesses - what is the best way to manage against technology service disruptions?

Here are the three pillars that help enterprise-class organisations achieve IT resilience:

1. You must have resiliency-in-layers, meaning a secondary (or more), geographically and meteorologically diverse, off-premise recovery data centre. This ensures that, should anything happen to your primary site, you will always have the redundant location to reduce the risk of an extended outage altogether.

Use a managed service provider (MSP) or cloud service provider (CSP). This switches the financial model to OpEx and allows you to leverage a ready-made infrastructure and service provider hired experts contractually obligated to deliver on the defined service level agreement (SLA).

2. Dip your toe into a public cloud infrastructure. Increasingly, organisations are rolling their own or leveraging MSP/CSP partners to “test drive” public cloud as a second or third site. Businesses must understand and match their data and application priority with the associated target and SLA requirements.

3. While every public cloud outage demonstrates that it’s not immune to catastrophes, looking at public cloud as a part of your resiliency-in-layers, hybrid-based plan can be a cost-effective way to get a third or more site and add some geo and meteorological diversity to your plan.

Organisations need to build and adopt tools and platforms with redundant, scalable, simple recovery and DR testing processes. The quicker a company can recover data, the less an effect it will have on the business with significant cost and time savings realised.

Andrew Martin is Asia Pacific vice president at Zerto.

Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.