Editing Recovery-oriented computing (section)

==Online verification and recovery mechanisms==
Recovery mechanisms are ways in which the systems can recover from failures.  These recovery mechanisms should be well designed, meaning that they are reliable, effective and efficient.  These systems should be proactive in testing and verifying the behavior of the recovery mechanisms so should there be a real failure it is certain that these mechanisms will do what they are designed to do and aid in the recovery of the system.  These verifications should be performed even in production level equipment as this type of equipment is the most vital to have up.  There are two methods for performing these tests and both of these should be used.  The first method is directed tests in which the tests are set up and executed.  The other method is a random test in which they occur without warning.