Mitigating Risks of the IT Disaster Recovery Test

Tuesday, July 20, 2010

Bozidar Spirovski

E973b16363b3de77b360563237df7e32

The IT Disaster Recovery Test as part of the Business Continuity testing is becoming an annual event for most IT departments. It is mandated by a lot of regulators, nearly insisted upon by internal audit and of course a very healthy thing to do.

But performing the IT DRP test without proper risk management can put your organization at significant risk.

To put things into perspective, let's analyze the steps, risks and countermeasures of an IT Disaster Recovery test:


DRP Test StepActivityRisksCountermeasures1. Failure of primary systemsIn order to perform a disaster situation, the Primary systems need to be caused to fail on some level
  1. Databases not closed properly/damaged due to forced shutdown or forced power failure
  2. Hardware components failing due to forced shutdown or power failure
  3. Spilt-brain cluster due to uncontrolled sequence of failures of servers and storage
  1. Full backup prior to the initiation of the DRP test
  2. Backup components and Vendor presence at ready during the entire test.
  3. Not performing a direct forced shutdown but forcing a network level isolation at the routers
2. Activation of Disaster Recovery systemsSevering any relation between the DR and the primary systems and running the DR systems as temporary primary
  1. Actual failure of primary system during the test
  2. Failure of the primary system while the DR system is concluded to be non-functional
  1. Full awareness of the test of every interested party - business custodians, directors of divisions and top management to initiate the real Business Continuity Plan
  2. Full backup prior to the initiation of the DRP test at DRP site, and full vendor support.
3. Reconfiguring the user environmentIntervening in the end-user environment in a way that will make them use the DR system
  1. Error in reconfiguration which may cause the end-user to input test data into the primary systems
  2. Error in reconfiguration which may cause the primary system to stop functioning.
  1. , 2. Scripted and documented steps of reconfiguration. All steps should be performed by 2 persons - one observing the others actions
4. Reverting to the primary systemsResuming the primary systems at some level and reestablishing the relation between the DR and the primary systems
  1. Error in reconfiguration which may cause the primary system to stop functioning.
  2. Copying of test data that was input into the DR test system back into the primary location3. Failure of primary systems during resumption
  1. Scripted and documented steps of reconfiguration. All steps should be performed by 2 persons - one observing the others actions.
  2. Fully controlled and documented process of resumption, which guarantees that only the primary system is data master.
  3. Full backup prior to the initiation of the DRP test, Backup components and Vendor presence at ready during the entire test.

With all these risks, is it more prudent to never perform an IT DRP test? - Absolutely NOT, and here is why:
  • Performing the IT DRP test actually confirms that things are running, and if something breaks, you are much more prepared for the next time.
  • Not performing the test will just make you think everything is great, until the incident occurs. And the incident is just as certain as death and taxes
So, perform the IT DRP test regularly, but with a whole set of countermeasures for the possible risks which can happen during the test. Of course you will miss some risks, but if you plan for 10 and miss 1 is much better then not planning at all!

Cross-posted from ShortInfosec
Possibly Related Articles:
11084
Enterprise Security
Enterprise Security Disaster Recovery
Post Rating I Like this!
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.