October 24, 3:10pm EST: All data has been restored and the mail queues have finished flushing. However, we have found an issue with a few hundred accounts that requires us to manually rebuild the data structure of the mailbox. This work is ongoing and we expect it to be completed today. If you are still unable to access your email account, you are likely experiencing this issue. We will post another update when this work has been completed. – ross
October 23, 2:23pm EST: The data restore has been completed and the mail queues are still flushing. We are still engaged in cleanup work so that while all customer accounts have been re-enabled, we are still working on completing work on a small number of customer accounts whose mailboxes need manual attention. When the mail queues have completely emptied, you will have received all of the inbound mail that we were queuing during the outage. We will let you know when the queues have completely emptied out and when the manual recovery on the still affected accounts has been completed. - ross
10:00 AM - October 23rd: The data transfer for affected mailboxes has now completed and all mailboxes are back online. All queued mail has started flowing and the email queue should be fully flushed within the next 3 to 4 hours. We apologize for the inconvenience this has caused you. - Tony.
The restoration is running longer than expected but it should still be completed this today - there was more data to move than earlier indicated. Mail is being queued until the restore is done.
Mail is still being restored and the process is again expected to finish this morning. The overnight updates from the email team were rather sparse, so I will talk to a few people this morning to get you some better details. -ross.
The latest news from Tucows Email Team is as follows:
Notes: The restoration process is proceeding as planned with approximately 82% of the task completed. There is about 1% of the customer base on this server cluster with no access to Hosted Email mailboxes via webmail, IMAP or POP; their mail is being queued until the restore is done. The completion time is estimated to be in 8 hours (Tuesday, October 23, 2007 approximately 04:00 UTC), including a testing phase to ensure that all data is safely reinstated. We will provide the next update at (Tuesday, October 23, 2007 approximately 01:00 UTC).
So it seems that more accounts are affected that we previously understood, but the restore job is going faster than expected. Based on this update, it looks like the restoration is going to complete this evening at around 11pm EST (04:00 UTC). This is great news, all things considered, although as I'm sure you'd all agree, I wish would not have happened in the first place.
We're expecting another update in a few more hours. If it contains any further news, I will post here.
Update
The problem this is affecting some Netidentity email clients are due to a server failure. In order to ensure that no content or data is lost, 400 GB of data is being transferred to a new email server. The expected time of resolution has been revised to early Tuesday morning.
--------------------------------------------------------------
We are still experiencing an issue with our email service. The situation has been ongoing since October 19th at around 8pm EDT. Those customers whose mail service is affect will not have access to their email at this time. This issue does not affect all NetIdentity customers and does not affect any Domain Direct customers.
In order to fix the issue, our email administration team must take your account offline in order to restore your data. Once the accounts are restored we will be able to re-activate your mail service. We expect all accounts to be restored within the next 48 hours.
We will update the blog and the network status page as updates become available.
