Light offs
02/11/2009
How embarrassing. DoS followed by a really neat hack. We still don’t know what damage was caused.
Trying to bring the service online of the spare capacity.
November 2nd, 2009 at 10:37 pm
No, it’s not a hack. It was DoS attack followed by a malfunctioned protection algorithm. A h/w problem happened at the same time.
We are off to the data center.
November 3rd, 2009 at 1:07 am
Time is 6am GMT. We are restoring the data. It is hard to estimate how long it will take.
November 3rd, 2009 at 1:58 am
The web service is back on. User files resynced across all machines. Working on bringing all our databases in sync.
There is a good saying: your backup is only as good as your last restore. Looks like ours wasn’t that good after all.
November 3rd, 2009 at 3:02 am
It’s looking promising. Any minute now.
November 3rd, 2009 at 4:23 am
The site is back online.
Orders placed within 4 hrs of the crash had to be removed from the DB.
The files are still there, so if you have them in your emails they are recoverable.
We are trying to piece together the last 4 hrs.