Effective Disaster Recovery Tips
Disaster recovery is an area of IT Management that can be easy to overlook. As IT budgets remain tight, the design and implementation of an effective Disaster recovery strategy can be put on hold.
However, as organisations and their customers increasingly rely on database, servers, applications and data sources, maintaining business continuity and protecting the IT infrastructure within the business is a principal concern.
Whilst the importance of protecting data remains high, the amount of budget allocated to the purchase of failover servers and recovery software, not to mention the facility to store data copies off-site for true data protection, is usually fairly minimal.
Although system downtime and data loss from hardware/disk failure, system corruption, theft, fire and flood damage, power failure or human error are rare, many companies have to deal with a degree of downtime and data loss caused by at least one of these factors at some point. Xynomix estimate that over 80% of organisations have had to execute their disaster recovery plans to some extent at some point. A survey conducted by Symantec in June 2009 suggests that this figure may be even higher: http://www.symantec.com/about/news/release/article.jsp?prid=20090630_01
Xynomix are often drafted in to develop Disaster recovery, or ‘DR’ solutions, that cover all bases: Quick System Recovery; Minimal Data Loss; Reliable Backups; Minimal Management and the security of Off-Site data storage, often without the need to purchase further expensive hardware or licences. We know what makes a successful Disaster recovery strategy in terms of technology, but we also understand the thought processes that must occur within the organisation to ensure the delivery of an effective DR solution.
Whether your organisation is a small start up or a multi-national corporation, the design, implementation and testing of a disaster recovery plan takes careful and dedicated preparation. We hope that our five top tips for an effective Disaster recovery arrangement can help your organisation to evaluate the requirement for DR whilst emphasising the importance of internal preparations.
1) Establish the Organisation’s DR Requirements
Disaster recovery discussions are usually centred on Recovery Point Objectives [RPO] and Recovery Time Objectives [RTO]. RPO simply describes the acceptable amount of data loss measured in time and RTO refers to the amount of time within which a business process must be restored before causing extreme damage to the business.
Typical disaster recovery strategies are as follows:
- Tape backups taken at regular intervals and sent off-site
- On-site Disk backups either copied to an offsite disk
- On-site Disk backups taken directly by offsite disk
- Off-site data replication, usually utilising SAN
- Systems and data kept off-site, e.g. Virtualised DR
However, before beginning speculative discussions around the type of DR solutions available and desired RPO and RTO , it is a good idea to perform a system audit and ascertain what level of disaster recovery is both required and desired.
It is important to ask the following questions:
- How important is your data and how much can you stand to lose?
- If lost, how easy would it be to replace your data?
- How long could the company’s systems be down for without causing serious damage?
- Do some internal systems require more protection than others?
- How much budget do you have available for the implementation of a Disaster recovery strategy?
- What are your overall recovery requirements/expectations/goals?
When you agree internally on requirements and ambitions for disaster recovery, finding the appropriate DR arrangement will be far easier and you can be sure that the implemented solution is delivered to exact specifications.
It is worth remembering that requirements and expectations should be set in relation to system criticality and the resources that you have available. RTO and RPO requirements can be mapped in relation to the IT infrastructure and a suitable recovery strategy established for each system within budget constraints. Whilst zero RTO’s and RPO’s would of course be desirable, the level of Disaster recovery protection that would be involved to achieve these levels would make such a degree of high availability impractical.
2) Balance Cost and Benefit
Cost will always be a concern when implementing a disaster recovery strategy, but by investing in DR, you are protecting against the greater, unexpected financial loss that a ‘disaster’ would cause.
Your aim when planning disaster recovery should be to balance the cost of the strategy with the potential loss that you are protecting against. The cost of your Disaster recovery strategy should never outweigh the benefits of the level of system recovery on offer. For example, if portions of your systems and data are fairly expendable or can be recreated, would it really be worth paying a premium to keep a copy of the systems and data off-site? Whilst DR is a vital aspect of database system management for some, for others the cost of guarding against the risk of data loss is more than the actual worth of the data itself.
The cost/benefit balance can first be addressed by performing a Business Impact Analysis [BIA] to assess the level of risk facing business operations should a disaster occur. A good approach to your BIA is to catalogue and rank all mission-critical systems and analyse how much downtime they would be able to cope with before causing a great deal of harm to the business. Some sources call this length of time ‘Maximum Tolerable Downtime’ for obvious reasons! RPO and RTO mentioned above can be gleaned from conclusions drawn around MTD.
Conducted in conjunction with a risk assessment to determine system weaknesses and the consequent likelihood of system disruption, the BIA should provide an indication of how much resource should be allocated to a disaster recovery strategy. The cost of DR should be considered as part of the overall cost of data ownership. For example, if your system applications generate a certain amount of revenue for the organisation each month and the cost of Disaster recovery is less than that figure, systems are protected and still profitable. However, if the opposite occurs and DR costs are more than the revenue generated, the application’s value would need serious consideration.
3) Plan, Test, Evaluate
Each organisation will have different requirements to meet when planning their disaster recovery strategy. However, there are three key capabilities that your disaster recovery strategy must deliver on:
- Backups of data must be protected in an off-site location
- Backups must contain enough data in a protected off-site location to allow you to fully recover your business operations
- The recovery process should be capable of restoring business function within the stipulated timeframe or ‘MTD’ value
Once your organisation has designed and implemented a disaster recovery solution, it would be easy to assume that everything was taken care of and the team could move on to other projects.
However, DR is a ‘living’ series of processes. It encompasses the procedures for business operations whilst system restoration occurs after a disaster and recovering any data that has been lost and returning to normal business procedures. A DR team including representation from each business unit should be provided with hard copies and online copies of the DR plan.
The DR plan should be tested, subjected to an in-depth evaluation and updated on a regular basis, ideally whenever a change occurs within the business that could affect the plan: for example, if there are any changes in technology, infrastructure, personnel and procedures. It should be well documented at each juncture. If disaster strikes, the last thing your team will want to do is search for instructions on how to rebuild the backup environment, for example. Documentation needs to include contact details for vault services, administrators and access passwords, instructions on preparing the systems, restoring the systems and validating the data centre prior to putting it back online.
4) Ensure that DR requirements will be met
The Disaster recovery technology that you choose should be able to accommodate internal requirements and expectations for your DR solution. For example, if some of your systems are mission-critical and would require restoration in a matter of minutes, choosing tape backups would not be advised. Thorough research of various strategies and their restoration times should be conducted and the findings should then be aligned with your budget and expectations.
Xynomix are often approached by organisations that have chosen a DR option that simply does not meet their needs: it is important to remember that a cheaper option can become more expensive in the long run.
You should also have the personnel in place to support the DR strategy from the planning stage. Senior members of the organisation’s leadership should be wholly supportive and invested in the plan, and a planning team should be established that incorporates representatives from all parts of the business.
Staff should be briefed and given instructions for each stage of maintenance and system recovery. After all, a Disaster recovery strategy is only as good as the technology used and those delivering it. Systems are constantly evolving. The plan for reaction in event of disaster should be kept up to date and should accommodate changes in the system such as the addition of new applications, an increase in data volume and new equipment. Those responsible should test the plan over and over until it works seamlessly.
Having worked hard to establish an effective Disaster recovery strategy, the organisation as a whole should attempt to prevent it being called in to use. Physical measures such as the use of surge protectors and a backup generator, fire prevention mechanisms such as fire alarms and extinguishers, and RAID disk protection technology should be put into place and serviced regularly.
If you do not have the appropriate resources in-house, or if resources become stretched during the planning process, it may be appropriate to consider outsourced Virtualised, or ‘Cloud’ based DR for continuous off-site data protection and lower infrastructure costs.
5) Decide whether or not you should you outsource
In many cases, organisations elect to use an outsourced disaster recovery provider to provide a stand-by site and systems rather than using their own remote facilities. If your Disaster recovery strategy involves on or off-site tape backups, this may not apply. However, for higher level DR strategies utilising off-site systems and data replication, it can be beneficial to outsource. The design, implementation and management of a DR strategy in-house can be prohibitive due to the levels of investment in technology and training personnel that it entails.
There are a number of arguments in favour of outsourced Disaster recovery:
- DR Outsourcing companies will be able to pass on cost savings through economies of scale
- IT resources are free to focus on other business-critical tasks
- Increased expertise from those that are experienced in all aspects of DR
- Access to diverse technologies and DR strategies, such as off-site data hosting, without capital investment
- The latest DR technologies and principles applied to your environment
It is important to weigh up the benefits that outsourcing your Disaster recovery strategy holds against the costs involved. If the cost of the Disaster recovery solution will outweigh its value, sticking with an internally led DR strategy may be sensible.
Xynomix often talk to organisations that are worried about outsourcing DR due to data security. If you are considering outsourced Disaster recovery, you should look to providers that will adhere to your own strict data security measures as well as their own. When Xynomix become involved in the implementation of a Disaster recovery solution, our own security levels and procedures are almost always set at the same levels or higher than those internally.
DR methods such as Hosted DR utilising Virtualisation technology mean that secure and up to date Disaster recovery strategies can be low cost and effective. More information around Xynomix’ Hosted DR solution can be found here: http://www.xynomix.com/oracle-consultancy/xynomix-disaster-recovery]
We hope that you have found these tips useful. It is worth remembering that, whilst it involves a high level of planning and involvement from all areas of the business, when called upon, a disaster recovery strategy can be worth its weight in gold. By investing in DR, you can ensure that employees and customers can stay connected and the business will continue to move forward with minimum service disruption and downtime.
If you would like to discuss any aspect of disaster recovery, please:
