How to Migrate 250,000 Emails with Zero Downtime

by Michael Cropper | Apr 3, 2016 | Developer, Retrospective | 0 comments

Rarely a day goes by without a new project with a new set of requirements, legacy systems and challenges which need to be overcome. This was one of these projects, where one of the many popular solutions we work through simply wasn’t going to meet the requirements on this occasion. I’ll leave the full reasons behind this story for a case study once we write this up. For now, to keep things simple we had to migrate around 250,000 emails from an old web server running IMAP technology and split these between two new places, the most used email accounts to run on the recommended Microsoft Exchange platform and the other email addresses to run on the older IMAP technology. There are many reasons why this had to be done, but I’ll skip over these for now.

So how exactly do you go about migrating 250,000 emails without losing access to anyones account, without any emails going astray during the migration and without losing any of the emails from the old system? Well, it requires quite a lot of planning, clear communication and most importantly a strategy that is going to work within the situation. This guide is for anyone else experiencing a large email migration project where it is essential everything continues running at all times. Did we get is 100% perfect, not quite. I’ve yet to experience any IT or software project be 100% perfect, but I’d like to think we did this with 99.5% perfection given the restrictions we were working with. This guide is going to cut straight to the chase and explain what the key stages of this process you need to go through to get ever closer to the perfect migration system.

Technologies

If you aren’t familiar with the differences between IMAP and Microsoft Exchange, then have a good read through our guide on the topic to get familiar. In short, when needing full traceability and efficient working with email technologies from anywhere in the world and on any device, quite simply you need to be using Microsoft Exchange. IMAP is not suitable for running a large organisation for a multitude of reasons.

With IMAP being the less advanced technology, this ultimately means that there are things that it doesn’t do as efficiently as Microsoft Exchange and most importantly, IMAP isn’t actually a piece of software per say. It is actually a protocol, a set of rules for how things are handled. What this means is that while there are a set of rules for how things are handled, as the protocol is implemented by software running on a web server, this in itself can lead to a whole host of problems depending on the how well the web server has been set up and configured, or how badly.  As such, it is difficult to predict with 100% accuracy exactly what is going to be needed without actually jumping in and getting on with it. This being said, you can certainly improve your odds by working through a structured process to maximise the efficiencies within the process.

Requirements Gathering

This has to be one of the most important aspects of any project. Before you jump in and start recommending any solutions, you need to fully understand the situation that is in place at the minute, the issues with this set up and the ideal functionality that is required within the organisation. Spend time talking to as many people as possible to understand the challenges they face and most importantly explaining the variety of options and their benefits. Once you have a full list of requirements you can then begin to plan the best solution.

Planning

With your set of requirements in place, you can review the options that are available to you and decide upon the best solution. There is no one-size fits all when it comes to technology, so make sure you are taking advice from a reputable company, it will save you a fortune in the long term. Or if you are the company implementing this process for someone else, make sure you don’t bite off more than you can chew, project like this can go really bad really quickly if you aren’t 100% sure what you’re doing.

In this specific example, our planning stage was a little more complex due to splitting all of the old IMAP emails between a new IMAP destination and Microsoft Exchange. This in itself required a change in email addresses for some of the users as you cannot use multiple technologies for emails on a single domain such as @example.com. In this case we had to set up a sub-domain for something.example.com which meant that the @example.com domain names could be powered by the superior Microsoft Exchange technology while the @something.example.com email addresses could be powered by the older IMAP technology.

To achieve this requires good web hosting and DNS management in the background. When you are on cheaper budget systems, you often cannot set up sub-domain delegation for MX records which is required to do this. If you aren’t sure what an MX record is, essentially this is a single piece of information which is used by ‘the internet’ which means that when an email is sent to hello@example.com, ‘the internet’ checks the MX record associated with example.com to see where it should forward the email onto. This is generally either Microsoft Exchange or an IMAP server somewhere, generally the one running on your web server at a specific port. As we were using two different technologies in the background, this meant that we needed to have two MX records which is not possible on a single domain, hence needing to set up an MX record at the sub-domain level too.

A key part of the planning process is communication which brings me on to the next section, before we look at the nitty gritty of how to actually migrate 250,000 emails with zero downtime.

Communication

Throughout any project, success or failure is generally down to lack of communication in one way or another. Migrating 250,000 emails is no different. Particularly when email is generally the most efficient way of speaking to people, you cannot rely on this should anything go wrong. In other words, you need another form of communication as a backup should things go pear shaped for whatever reason. Whether another email account on a 3rd party platform or the good old telephone.

As the technical person implementing this, the process behind migrating the large number of emails requires this information to be communicated to the key stake holders in a non-technical and easily understandable way – which can always be a challenge when working with naturally very technical aspects. So whatever you do, make sure you are over communicating with everything you are doing.

Implementation

So, let’s get down to the nitty gritty for how to migrate 250,000 emails with zero downtime.

Old Web Server Review

You need to know what you are starting with. Generally speaking, when you are migrating web servers for whatever reason, it is usually due to the old web server being a bit rubbish in whatever form. This often is coupled with a lack of flexibility that you are used to when working with industry leading technology. You must fully review the old web server, every aspect and speak to the key stake holders who have had anything to do with managing this in the past to fully understand any interesting configurations that may have been set up in the past which can be easily missed.

We’ve already covered about how to migrate a web server seamlessly with zero downtime, so have a read of that if you need more information on the topic. For now we’re going to look specifically at the email side of things. Remember, we wanted to migrate 250,000 emails from IMAP, half to Microsoft Exchange and half to the new web server which was also running IMAP. This required two significantly different approaches to achieving this task.

Prepare the DNS

First things first, we would always recommend migrating things in stages. So let’s assume that you are first going to migrate the contents of the web server, namely your websites, then secondly migrate your emails. Or vice versa. To do this requires you to prepare your DNS settings accordingly.

As DNS propagation can take up to 24 hrs to complete fully, you need to plan this step in advance. With good DNS providers, this is generally extremely quick, although do not count on this, plan in advance.

In our example, we were also migrating the DNS settings which were previous set at the old web hosting company. Instead we moved these to an independent setup, meaning that we could prepare these in advance so propagation could take place throughout the system prior to us changing nameservers at the registrar.

Registrar – Changing Nameservers

Once we had successfully added the DNS settings at the new DNS provider so that the website would run off the new web server and the emails would continue running off the old web server, we were set to go with changing the nameservers. Once we made this change, this meant that we could be confident that and downtime was kept to a minimum.

Ok, so now the first process had been completed, this meant that we could move onto the emails. To do so requires more planning and careful implementation.

Preparing Microsoft Exchange

If you have ever set up Microsoft Exchange before you will know that there are a few settings to configure at the DNS level. To configure Microsoft Exchange in preparation for when we needed it, we actually had to add some of the DNS settings on at the old web hosting company prior to switching the nameservers. This is required to give you the time to authorise your account along with configuring all of the relevant mailboxes and communicating with the individual users for the project.

This guide is not designed to look at how to configure Microsoft Exchange, so if you don’t know how to do that, get in touch and I’m sure we can help you in some way with what you’re working on. For now, assume that a significant amount of time was taken to get everything configured correctly. Interestingly, as part of this specific project, we had to hunt down a rogue Microsoft Exchange account that had been created many moons ago by someone within the organisation with zero information about what had been set up and why and most importantly who owned this. As the Microsoft Exchange system is extremely secure, there are safeguards in place to prevent someone else taking control of an Exchange domain if one has already been set up, meaning that we first had to hunt this down and regain control of this. I certainly hope you never have to do this as I can’t tell you how time consuming this is to do.

Ok, so now you have Exchange all prepared for your email migration project. Next is preparing the IMAP email accounts on your new web server.

Preparing the new IMAP email addresses

First things first, you need to create all of the accounts on the new web server for all of the email addresses which are going to be using IMAP. Make sure you do an audit up front on the old web server to understand how large the individual email accounts are. Often when you create an email account with a default size of say 1GB on IMAP, then for certain users this may not be enough space. You want to plan for this in advance as you do not want to find this out when you are half way through transferring emails only to have to start again.

Thankfully on good web server control panels such as cPanel, doing this initial research is simple and relatively quick. Unfortunately on poor control panels which often come with budget web hosting packages, finding this out before hand can be a little challenging and in some cases simply just not possible which is a little annoying.

Switching the New Email Addresses On

Now everything has been prepared with regards to your new email technologies, it’s time to switch the DNS settings to point all the email related activities to the correct sources. Make sure you don’t make any mistakes when doing this as everything will start to go wrong and emails could go astray.

Migrating 250,000 Emails

Ok, so now we have everything prepared, we’ve switched the emails on but there is nothing there other than emails which are now sent from this point forward. Jumping back a  little, it is actually possible to migrate the emails to Microsoft Exchange when configuring all of the accounts. There is a handy tool within Microsoft Exchange which isn’t 100% but can often connect to your old mail server and import the IMAP emails into Exchange which is a different protocol. When this works, this is perfect as you now have a system whereby all of the sent and received emails along with relevant folder structures that people use within their emails has been successfully ported over to Exchange. I’d recommend doing this section prior to switching your new emails on because it can take a while, as in days. So set this running and be patient. The larger number of mail boxes you are dealing with, the longer this will take. When migrating in batches, migrating around 20 mailboxes from IMAP to Exchange took around 8 hours to put this into perspective.

Now, as mentioned before, IMAP isn’t as good as Exchange when it comes to the underlying technology. Meaning that while there is a handy little tool that comes with Microsoft Exchange to do this task, the same is not true for IMAP. As IMAP runs of your web server, there are endless different configurations, setups and options which come into play. Meaning that you’re going to have to figure this bit out for yourself to a certain extent with a lot of testing various options.

Thankfully there is a seriously awesome tool called IMAPSync, which is designed to help with a lot of this process. At only £100 it is worth its weight in gold and anyone attempting to migrate any amount of emails between web servers should purchase this tool. The developer behind the scenes who built the tool is also extremely useful when helping with the usual configuration niggles that you face with things like this.

I’m not going to cover how to use this tool in detail as this is going to differ depending on your individual requirements. Suffice to say, that with a bit of testing for various configurations etc. you will get there in the end. Make sure you test things first on a test account on your new IMAP web server though so you can be confident that this is working as expected. The last thing you want is to be all gung-ho and start everything running only to find you did something wrong.

As a quick overview of IMAPSync, this tool requires you to install the software from the Yum repositories, upload a tarball to the web server under the Root account via SSH on the new web server, unzip this zipped folder, test a few configurations to make sure the scripts can actually access the internet and then you’re off to go. Something to bear in mind is that if you are having trouble getting the script to connect to the internet, it’s likely due to the firewall running on your web server, either physical or software based, which is blocking specific ports on the outbound side, generally port 143 for IMAP.

So there you go, once you’ve managed to run through this entire process, you’ll be in a situation where all of your IMAP emails are now also migrated to the new web server.

Sender Policy Framework

Something to bear in mind is that when setting up a IMAP emails, you will likely be blocked as spam if you don’t have SPF records set up at the DNS level. Leading technologies such as Microsoft Exchange will block emails from domains who don’t authenticate their self through SPF records by default as many of these emails are automatically generated spam. Something to bear in mind if you start getting issues with deliverability to certain email accounts or domains

Summary

We deal with email migration on a regular basis and you often can’t quantify the project before you get involved, it is extremely difficult to predict. Overall though, what I can say is that if you are going through a similar project, you need to be planning things effectively and implementing the right technologies to make sure everything works. For larger organisations, losing access to their emails for even an hour is a lifetime. With a well thought out strategy, detailed requirements gathering and planning your implementation will be much more straight forward. Follow a similar process when you are migrating emails and you too will be able to migrate a significant amount of emails with virtually zero downtime.

The only other thing I would finish with is that it is best if you can implement some of the critical items such as DNS changes at times which are less likely to have an impact. For example late in the evenings or at weekends. It’s not ideal for the person implementing this, although the benefits to the organisation you are working with is enormous, it will result in a situation where emails don’t end up getting lost in the ether while DNS propagation is taking place. It is also recommended to keep your users informed at all stages and most importantly tell them when not to send emails when you are making DNS changes, users often work at all hours of the day when you may be expecting them not to be, so keep in touch with people throughout.