Recovering from mail server disasters
If your best client were to lose their mail server and the emails on it, how crippling would that be? Most businesses both large and small depend on email. If a company relies on email, you need to have the tools and techniques ready to solve their problems for the day disaster strikes.The figures for just how many companies are risking everything are shocking; a survey by ApplicationContinuity.org in 2008 found that of the companies responding to the survey, less than half have a full disaster recovery plan in place, and less than half have implemented a high-availability solution of any type, even if the definition is left loose enough to include data backup solutions that would not provide high availability of email. When the survey looked at disaster avoidance, only 21% of companies have a disaster avoidance strategy, and almost a third of mid-sized companies rely on a single Exchange Server with no plans for what happens if that fails.
The findings are mystifying, particularly when taken alongside research from Neverfail that found that 25% of companies surveyed would experience a significant loss in employee productivity during email downtime and 19% would lose revenue. 34% would potentially damage their company or customer relationships, while 6% would be unable to maintain regulatory compliance.
Work out what you need – now
In the manner of all boy scouts, you need to be prepared. The time to discover you need one extra tool is when you’re pretending your number one customer’s server has caught fire, not when they’re calling you with the fire extinguisher in their other hand. For the purposes of this article, we’re assuming the mail server in question is completely unavailable – it was stolen over the weekend, or an irate user took a hammer to it. So how do you get back to a position where the users at your client have email? Given time, it’s straightforward to commission a new mail server, but it takes time and the users will be screaming for their emails now.
One solution to the problem just described is offered by companies such as Mimecast through their continuity services. These complement Mimecast’s email archiving service and are designed to provide continuous access to email services during the system failure, by providing an independent off-site system. While the customer’s own server is unavailable, the users can still access new incoming emails on the hosted servers, and can create new outgoing messages. Mimecast delivers the emails to a disaster recovery inbox that can be accessed by either Webmail or through an email client such as Outlook.
Once the mail server is back up and running, all queued emails and any new emails created by the users are delivered to the mail server.
Another company offering a similar facility to Mimecast is the Email Protection Agency. This is designed to be used alongside a customer’s normal primary email system during both disaster and normal operation. The Email Protection Agency captures all the email traffic and stores it for 28 days, so provides a short-term online backup as well as continuity service. If the local email server at the customer fails, the end users have access to their Internet email via a secure Web browser and are able to view, reply and forward emails. When the local server is restored, email is automatically re-synchronised with the primary email system.
MessageLabs offers email continuity that works with Outlook, Lotus Notes, Web browsers or BlackBerry devices. When a company’s email server is running normally, copies of all the emails are sent to the MessageLabs backup system. If they hit a problem, the back-up system is then activated over the phone or via the Web. MessageLabs lets administrators specify particular users who are affected, identifying them by mailbox, server, location or storage group, and the clients are then automatically connected to the back-up system. When the mail server is restored, all the emails, changes to calendar entries or contact updates are migrated back to the email server.
No server, no service
An email continuity service needs to be set up in advance. What if your customer’s email server has failed today without the luxury of an external system in place? The first thing to remember is that there are some simple steps they and you can take to minimise the disruption. Preventing email bounces stops customers and partners worrying; some ISPs will let you extend the resend delay on the relay so the company email isn’t bounced back to the sender immediately. The exact mechanism that you follow to do this varies, so you’ll need to check with your ISP.
It’s also worth checking whether your ISP provides a Webmail service – some such as Demon do, so the users can at least check their emails online, even if they usually work locally. Swapping to Web-based mail is obviously only an option for a small number of users, and it does carry the overhead of having unsynchronised emails when you later swap back to the normal server. However, it may save your bacon if the head honcho at your customer just has to check his emails and can’t wait while you fix the server.
When you’re trying to help a customer choose a backup system, the question you have to ask them is how much data they can afford to lose. Some companies would be unable to cope with losing minutes worth of emails; others would be OK with losing a day’s worth of data. Only your customer knows what really applies in their situation, and bitter experience suggests you should get the details in writing; that way, history can’t be rewritten. Whatever they decide, you should make sure it’s implemented as an automated system with logs and alerts. You can’t rely on someone remembering to do the backup; otherwise you get to the point where you need to recover the data and discover the last backup happened months ago.
In addition, don’t rely on a single backup system; at the very least have a system that takes local backups on the most frequent schedule, and off-site copies less often.
Where should the backup be?
When you’re trying to decide on a backup strategy that will keep your customer safe, there are two elements to consider – creating the backup and using it to restore data.
If the client is running their email server on a single standalone server, its message store will usually be either on the server’s internal storage or on directly-attached external storage. The message store will probably be small enough for it to be backed up onto a single tape on a tape backup device. This is simple, fairly cost effective, and you won’t spend forever setting it up and managing it. On the downside, the backup task will probably run locally on the server, so will share resources and could slow the mail server application down.
You may be tempted by the simplicity of online backups; no worrying about whether the tape drive is working properly, no phone calls about running out of space halfway through the process, just send it over the wire and let someone else deal with the practicalities. Online backup has some advantages, but for a quiet life you shouldn’t rely on this alone. Imagine what happens on the day you’re called in to restore your data, and because the server has gone down taking the Active Directory structure with it, you can’t actually get onto the Internet to retrieve the remote backup. Always make sure there’s at least some local backup as well. That way, you can retrieve the data even though you have no connection to the outside world.
Conversely, it’s worth having at least some data backed up online. That way, if the tape backup system fails to work when you need to recover the data, you’re not rushing off to the local computer store to get a matching tape drive just to get the data back. Instead, you can go online and download it. An online backup is also potentially easier to use. Offline backups may require you to work through several steps, making decisions about which tapes to use, in what order. If a frazzled client attempts to restore a system, they’re more likely than usual to make mistakes, so the simpler the restore operation the better.
Don’t forget PSTs
The other thing to ensure is that the backup collects all the data you want to keep safe. In the case of emails, the main danger lies in messages stored on the local hard disks of users. You may have set up the system to use a central email server such as Microsoft Exchange, and you may have set up the backup to save all the data stored in the Exchange message store, but how many of the users have data stored in local PST files on their hard disks? If you have mail quotas implemented, you can expect all users to have them (so email archiving services can solve two problems at once). Unless you back up those PST files as well as the main message store, you’re not taking a full backup.
PST files are notoriously easy to corrupt, so if users at the client have PSTs, plan ahead to avoid problems. Microsoft has a free Outlook add-in called Personal Folders Backup that works with all versions of Outlook. It will back up all the PST files a user has very simply, and automatically remind them to do backups on a regular basis after that. If they hit problems, the same tools can also be used to restore the Outlook data from previous backups. If you want a more centralised solution, products such as Zantaz EAS can be used to take central copies of all the PST files across your system; Mimecast also has an option for this.
If the customer can’t afford to run a redundant system with a spare mail server sitting idle, don’t forget the possibility of using a virtualised system, as it makes maximum use of available hardware and essentially gives them as many servers as they might ever need. Mail servers – even Exchange Server – run well in virtualised servers, and the virtualised operating system is easy to back up. You can have the mailstore backed up as frequently as required – once an hour, say – then take a snapshot of the entire virtual machine daily. If a problem occurs, it’s quick and easy to configure another virtual machine, and populate it with the snapshot. Your customer will be back on their way in a matter of minutes.
Backup – get the order right
One point that’s well worth drumming into the heads of whoever is responsible for taking backups at your clients is this: when they’re backing up their mail server, if they do incremental backups every day and a full backup once a week, they must remember to always do the incremental backup before the full backup. The reason is that once they’ve done a full backup, there’s no way for the mail server to work out what should have been included in the incremental backup, so that day’s incremental backup isn’t carried out. If for some reason the full backup fails, they’ve lost a day’s data, and most recovery systems won’t actually recover the data without a complete set of incremental backups.