VPSVille.ca explanation of the disappeared servers

Quoting the e-mail I and all affected customers received on October 13, 02008. I confirmed that my "hardware node" was gone. I'm posting this as a public service so anyone who's thinking of using VPSVille.ca is aware of this situation.

What Happened Here, Oct 8th.

We here at VPSVille have had a few days of frantic phone calls, hair-pulling, yelling, crying, screaming, sector-recovery crash courses and angry accusations. We've also looked closely at our logs and have pieced together what happened with our recent data loss incident.

Initially we thought it might be a case of gross negligence and some harsh words and harsher threats were exchanged, but we now believe it was just an honest, stupid mistake. Inexperience sunk the ship.

One of our new employee's made some crucial errors regarding security, and this was compounded by one of our techs travelling abroad and requiring some relaxation in our usually strict access policy. One firewall in front of another and both temporarily disabled = no firewall at all.

This unfortunately resulted in some malicious SOB with no real life to waltz into an almost unprotected segment of the network through a newly installed appliance with a default password and start deleting things. Several servers actually and a NAS unit assigned to back them up. One of the servers was our control node, which interfaces between the website and the other servers. Even our mail server was affected.

Some servers were running even after their files were deleted, through some strange quirk of Linux buffering that we still don't fully understand.

Its easy to blame a hacker of course, but we do feel that an unfortunate series of events combined to make their hack particularly easy. Far to easy for a network of this magnitude and importance to so many people.

This looked like a total loss, with no way to recover the data, so we sent out a somewhat panicked email to affected users that their accounts would just be canceled and refunded. Shortly afterwords our outgoing email died.

This resulted in a torrent of email, most of it surprising friendly, and much of it understandably angry and dismayed, telling us they wished to remain customers even if their data was not recovered.

Happily we were able to restore many servers with a sector recovery service. In light of this we have not arbitrarily canceled any accounts.

All affected accounts are re-enabled and have been credited. If we were able to restore your data we have sent you an email about it (hardware nodes rapier, cutlass and spear).

We sincerely express our condolences to affected customers for this unfortunate series of events, nobody should lose data at VPSVille, its an awful thing and we feel your pain.

Things do happen in life, but this won't happen again.

- Staff