{"id":300,"date":"2011-06-17T09:52:49","date_gmt":"2011-06-17T09:52:49","guid":{"rendered":"http:\/\/policybee.co.uk\/blog\/?p=300"},"modified":"2024-01-23T16:55:45","modified_gmt":"2024-01-23T16:55:45","slug":"disastrous-disaster-recovery","status":"publish","type":"post","link":"https:\/\/www.policybee.co.uk\/blog\/disastrous-disaster-recovery","title":{"rendered":"Disastrous disaster recovery"},"content":{"rendered":"\n

<\/p>\n\n\n\n

\"When<\/figure><\/div>\n\n\n\n

It\u2019s probably stating the obvious to say that comprehensive disaster recovery solutions are important to any business.<\/p>\n\n\n\n

Here at PolicyBee we need a pretty robust back-up procedure for two main reasons: 1) so we can keep trading when all else fails and 2) to keep our clients\u2019 data secure.<\/p>\n\n\n\n

We know we\u2019ll have to rely heavily on our \u2018plan B\u2019 if things do go wrong. So it\u2019s got to be right.<\/p>\n\n\n\n

Which begs the obvious question: what happens when it\u2019s not right?<\/p>\n\n\n\n

Well, after two years and \u00a3thousands spent trying to get our disaster recovery software to work exactly<\/em> the way we’d like it to, we’ve been forced to come up with an answer to that particular question.<\/p>\n\n\n\n

As a company responsible for holding client data, not having a reliable contingency plan in place is simply not an option. Heres’ what happened when we realised our disaster recovery solution wasn’t so much of a solution after all.<\/p>\n\n\n\n

Counting the cost<\/strong><\/h2>\n\n\n\n

Let\u2019s start at the beginning. We bought disaster recovery software (we\u2019ll call it LookAgain) recommended to us by a longstanding supplier of ours (we\u2019ll call them Suffolk IT) in May 2009. In simple terms, the software was designed to ‘mirror’ the files on the servers at PolicyBee HQ to a group of servers at our disaster recovery site.<\/p>\n\n\n\n

The software itself, the licences to use it and the maintenance package for the first year cost just shy of \u00a320,000. Turns out that was just for starters.<\/p>\n\n\n\n

Following installation, and more or less right from the start, we battled to keep the software up and the costs down. For example, every Windows update meant rebooting the servers; which in turn meant expensive engineer\u2019s time.<\/p>\n\n\n\n

And when we weren\u2019t turning the servers off and on again, we had problems with the software not \u2018mirroring\u2019 all our files \u2013 again costing us time and money to resolve.<\/p>\n\n\n\n

<\/p>\n\n\n\n

The beginning of the end<\/strong><\/h2>\n\n\n\n

In early 2011, we needed to retrieve some lost files from our recovery site. Unfortunately, they weren\u2019t there. The recovery server had fallen a few months behind and approximately 1,000 files were missing. Not great.<\/p>\n\n\n\n

We voiced our concerns about the reliability and escalating costs of LookAgain to Suffolk IT. That prompted them to conduct a thorough check every day for two weeks.<\/p>\n\n\n\n

All seemed well so they then lined up the recovery site to carry out a full test of the whole set-up at the beginning of May. The test ran…but 4 out of 5 servers failed to come online successfully.<\/p>\n\n\n\n

We figured enough was enough. So we pulled the plug and now have a much better plan B in place that we know we can rely on.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Back up, punk!<\/strong><\/h2>\n\n\n\n

You might wonder why we’re telling you this.<\/p>\n\n\n\n

Well, spending a shedload of cash on software that doesn\u2019t do what it\u2019s supposed to do, when it\u2019s supposed to do it, is a story in itself.<\/p>\n\n\n\n

The really important bit, however, is what we\u2019ve learned from all this. It\u2019s clear that from now on we should be:<\/p>\n\n\n\n