Hostroute blog
Statement: blue.srv2.com
Wednesday November 15, 2006
We were contacted today by the company that manages this server for us (Ultraspeed UK Ltd) and told that during an operating sytem reinstallation all of the Mysql data was lost from the server.
This problem was found when the new system was restored from the backups. On investigation it appears that the Mysql databases had been symlinked to the /home partition but the backup system was looking in the usual mysql partition for the databases. Under normal circumstances this would not have been a problem as we normally install new primary drives when doing system rebuilds, but for some reason they decided to reformat and reuse the existing primary drive. There was therefore no original disk to go to.
This means that:
All Mysql data from this server has been lost and is not recoverable, even by a data recovery company.
This is the first time we have ever lost any customers data, and its made worse by the fact that it was dowm to the negligence of a trusted contractor (i.e. not something we could have avoided happening ourselves).
Customers who have their own backups will be able to restore their own data. Our terms and conditions advise customers to have their own backups in case there is a catastrophic loss. This is the first time we have suffered such a loss.
At the moment there appears to be about fifty web sites affected. Of these probably only four or five have any significant amount of data in databases (the others being more easily re-creatable product lists or similar data which exists elsewhere or disused databases). The situation is made worse because this server has some of our oldest accounts on it (customers going back many years) and they may have more data associated with them than newer accounts. It will be 24 hours before we know the full impact of the incident.
Affected customers should enter a support ticket for further advice. We have a number of possible solutions which may help to ease the difficulties this is causing.
We would like to assure customers that this has never happened before and it will not happen again. The contractor involved has agreed to new procedures where drives will be replaced, not reformatted, and the old ones stored until it is known that the data on them is no longer required. They have also agreed to accept all liability in relation to this incident.
Update: Two customers were affected (one having three accounts on the server) so the total number of affected sites was four.
