Initial diagnosis of suspected virus following the loss of 8TB of data ends up being caused by disgruntled employee deleting files in anger.
Our client is a global business to business advertising agency with over 600 employees. They have a strong reputation in the advertising sector, servicing leading blue chip brands.
The Disaster Recovery Solution
Plan B looks after all advertising imagery for this client, which is one of their most valuable assets. It totals 12TB of data, predominantly on one large file server. One of the difficulties agencies experience in recovery following a server failure is the unpacking and restoring of large amounts of backup data, which can extend recovery time to weeks. Plan B minimises this customer’s risk of IT downtime by building a replica system with all data already uploaded, offering a virtual standby system that is fully configured for immediate use when required. This offers our customer the ability to immediately failover to the standby recovery system which is ready for use, with the most up to date data present.
One morning, just before Christmas 201, Plan B received a call from a member of the technical team. They had lost around 8TB of data and needed it back quickly. The best way to achieve this was to invoke the Plan B recovery platform, as data transfer times for 8TB, even to disc, were estimated at days rather than hours. The initial diagnosis by the client was a virus, however a further 3TB had gone ‘missing’ during the invocation process, leading the Plan B technical team to suspect the cause was closer to home. This was because the deletions were different, the first a disc deletion, the second by a disc being reformatted. Agreeing the cause of the missing data could be an internal threat, staff were asked to leave the building and admin domain passwords changed. Unfortunately the root cause of the trauma had not been completely resolved by the change in admin domain passwords as the SAN admin passwords were not changed and the perpetrator managed to delete some of the LUNs on the SAN.
Plan B booted up the client’s virtual standby system and within just a few minutes of the initial call it was fully available to the client. Due to the nature of the problem, a high level call was made to the parent company in the USA to establish who could be trusted to handle the recovered system. It was only on their authority that we made their system available to the Head of IT only.
The plan going forward was to restore the client’s servers from the Plan B appliance rather than give employees access to the rescue platform. Because of the amounts of data involved, this was a very lengthy process so the Head of IT manually provided information to employees. As a safety net, in parallel to this it was agreed that Plan B start restoring the 12TB of data onto an ‘export server’ so that it could be physically couriered to site and deployed as a replacement to the original server. This would cover them if the initial data restoration took too long.
This process took 8 days over the Christmas break to right to disc (demonstrating how long they would have been without their systems if they didn’t use Plan B’s Pre-recovery service). It was fully tested and shipped to the customer where the Head of IT took receipt and installed it, providing a standalone local access of files whilst the data restoration to the newly built servers continued.
At the start of the New Year, service was starting to get back to normal as staff came back from the festive period. Unfortunately, however, the incident was not quite over as the Export server was left unprotected within the locked machine room. Someone subsequently accessed the machine room and deleted one of the VMs. At this stage the culprit had been identified but the damage still remained.
The client continued to provide services to their customers during this traumatic period, working from the Plan B’s protected systems. Having finally secured the perimeter, local services were rebuilt and fully restored and a short while afterwards the customer was moved back to their live system. The Plan B Disaster Recovery service resumed, protecting the new ‘live’ platform in the usual, dependable manner.
Plan B Disaster Recovery solution was able to save this client from experiencing severe consequences of their IT disaster, but only because:
- We Pre-recover. Recovering after the event they would have taken days rather than minutes due to the nature and longevity of the attack.
- We’re independent of their IT department. If they had been trying to run their own disaster recovery provision inside their management perimeter, it’s likely that the attacker would have destroyed that too.
This situation illustrates just how much an IT disaster can impair your performance and capabilities. Even though our client was highly capable of withdrawing access from employees appropriately under normal circumstances, they were under such immense pressure that they were unable to lock down their IT system effectively when they needed to. It is very common for the stress and pressure of an IT disaster to adversely affect performance of individuals, leading to further errors and impairing recovery. An external provider is much better suited to handle IT disasters for exactly this reason.