by Michael Cropper | May 9, 2016 | Developer, Technical, WordPress |
Accelerated Mobile Pages, AMP for short, is an open source project designed to make the web faster. For people accessing web content on mobile devices, speed is a challenge for many users and with over 50% of content accessed on the web via mobile devices, this is more important now than ever.
The concept of Accelerated Mobile Pages is all about stripping out irrelevant styling and fancy JavaScript technologies to make the page load much faster, with the most important aspect, the content, loading virtually instantly.
If you’re interested in the finer details behind the project, have a read all about it here, https://www.ampproject.org/. The technical aspects behind the project are quite significant as are the underlying details about how your web browser loads content as standard.
Accelerated Mobile Pages AMP Speed Test
So we thought we’d put AMP to the test to see just how much faster it really is for WordPress in comparison to a rather bloated website which requires a bit of TLC, like most WordPress websites on the whole. The results below we repeated on the same website multiple times and cannot believe the performance increases we saw. The Accelerated Mobile Pages plugin for WordPress is available for download from the WordPress repository. A note on the plugin at the time of writing, it only supports Posts in WordPress, i.e. your blog posts. Pages and Ecommerce Products aren’t supported currently.
Blog Post Loaded As Normal
Blog Post Loaded with Accelerated Mobile Pages APM Technology
Awesome! Try loading this page you are viewing now as an Accelerated Mobile Page here to see how this looks: https://www.contradodigital.com/2016/05/09/wordpress-accelerated-mobile-pages-amp-speed-test/amp/
As a footnote. Yes, 30 seconds is darn slow for a website to fully load. Yes, tools like Pingdom are not perfect as many users are more interested in the time when the website appears to have completed to load opposed to when the last byte has arrived. And yes, this is only a single site as a comparison. Get involved and give this a go on your own website to see how this performs for you. Every website is significantly different and ever web server has been configured differently based on your individual needs.
by Michael Cropper | May 9, 2016 | Developer, Security, Technical |
Just like the computer that you are reading this blog post on, your web server has a lot of software installed to keep it running. And like all software, it needs to be kept up to date to avoid security issues. Web server security is an enormous topic with many moving parts, many which are often uncomprehendable to the non-teckie.
Seriously though who is updating your web server software?
Your web developer? Unlikely, often web developers have very limited knowledge of the underlying technologies of web servers.
Your web hosting company? Possibly, but unlikely unless you’re paying them to do so.
Your IT team? Unlikely, your IT team is often focused around the computers, laptops and devices around the office and often believe that it is the web developer’s or web hosting company’s job to do this.
As a business owner it is your responsibility to be asking these questions and making sure that you have this part of your cybersecurity looked after. If you don’t know who is looking after this for you, you need to find out. Get in touch if you find out that this is not being looked after, as I suspect is the case for most people reading this blog post. As with all software, it is essential that your web server software is kept up to date to avoid potential cyber attacks.
This is what one small part of updating server software actually looks like to the teckies managing this for you, no pretty user interface, it’s primarily command line management;
by Michael Cropper | May 9, 2016 | Security, WordPress |
Another day, another hacked website. This time, thankfully due to the security and backup solutions in place, restoring this backup was relatively straight forward and much cheaper than if these security solutions and automated backups weren’t in place.
Usually we get businesses running to us when things have gone wrong and they have failed to plan accordingly for when this happens. Thankfully, whichever way you look at this, this business had been hacked before and since then had additional levels of security and automatic backup solutions in place to reduce the risk and help to resolve issues when they do go wrong. This being said, on this occasion, no amount of technology can protect against basic human error and weak passwords.
Below is a brief outline about what happened. It’s only a rough guide but does go to show how something as simple as a weak password can even bypass the additional security. And most importantly, this highlights the reason why you should always be planning for the worst and expecting the best. Naturally, all of the sensitive information is blocked out of the screenshots, as is information about the hackers as they deserve no additional praise for the work they have done.
Website takeover
Server Logs
The website isn’t hosted with ourselves, so by the time we investigated this we couldn’t access the server logs as web server had already been rebooted, so the error logs lost.
WordPress Access Logs
Show that the user ‘Admin’ logged in on 08-05-2016 at 8:55 am;
From the IP address 41.111.14.83 which is based in Algeria, Africa;
This username belonged to a member of staff at the web hosting company whose account was compromised and used to initiate the attack;
Which when looking at the access logs on the date, there weren’t many failed login attempts which suggests that their password was rather weak;
Plugin Uploaded
Now the hackers had Administrator access to the site, they uploaded a plugin which allowed them to upload a script to the site and run this script via the command line;
This plugin & script allowed the hackers to take over control of the whole site and display the messages.
Summary
While this does seem like an extremely simple thing, essentially guessing someone’s password. This does highlight the importance of being able to trace exactly what happened when a hack occurs. Being able to track the path the hackers took to exploit the website ultimately led to the root cause of the problem being able to be fixed to prevent this from happening again.
If you don’t have website security and automated backups in place, then get in touch. Seriously, get in place what you need to before you get hacked. Do not think that this will not happen to you. As we have just shown, even with added levels of security in place, a hack can be as simple as a weak password. Make sure you have the right technology in place to be able to identify the root causes of problems when they do occur to save yourself significant time and money dealing with things when the inevitable does happen. No-one wants to get hacked and everyone believes they will never be a target. The reality is that most businesses will get hacked at least once in their lifetime. It is only the large businesses that we often hear about in the news. In this instance, this is a story of a small business, less than 10 staff, for whom this kind of issue is something that they would prefer not to have to deal with.
In this example, we were able to respond to the issue, identify the affected areas of the website, identify the root cause & patch all within a morning. All for the cost of the price of a coffee per day. You have to ask yourself, if you don’t have the right technologies in place to deal with situations like this, how long would your website have been offline for? A day or two? A week? More? I can’t stress enough that preparation is key when dealing with cybercrime. Your digital back door is probably wide open, waiting for someone to walk by and exploit it, you just probably aren’t aware as it’s not something you can physically touch and feel.
by Michael Cropper | Apr 24, 2016 | Data and Analytics, Digital Marketing |
Sorry, I’m calling it. Businesses purchasing subscriptions to companies that are offering free data at a hugely premium price need to stop paying for this now. Stop wasting your money. Hell, give it all to a charity of your choice, it will be better money spent.
The company I’m talking about, for anyone working in digital will be well aware who this is, but for legal purposes let’s just call them a fictitious name which bears no resemblance to any company or person which happens to have a similar name, EnquiryForensics.
The offer for these types of companies is that you can “See who is looking at your website”. In summary, no you can’t. The only information that these types of companies generally provide is information that someone in Company X was looking at your website, then joining this data together with generic contact details for this company or publically accessible contact information for various staff within this organisation. This does not tell you who specifically is looking at your website. There are a whole host of data protection issues with tracking individual user website usage which is why services like Google Analytics are obliged to anonymise certain aspects of the data.
So beware, any company trying to sell you access to a tool or service which can show you who is looking at your website in a company, it is likely a snake oil salesman or company. You can generally get whatever data that have completely for free, and here’s how.
Google Analytics is Awesome
I’m not going to go into depth about Google Analytics. In summary though, Google Analytics tracks over 560 points of data for every website visitor and when you know where to look, you too can start to see how valuable this data is.
Identify the Companies Looking at your Website
Simple. Within Google Analytics, go to the following report: Audience > Technology > Network. This report will show you the companies who have been looking at your website, which can be extremely valuable for any type of B2B business as you can then feed this into your outreach pipeline and support the work your business development manager is doing on a daily basis. Look;
It’s all there. The companies who have been looking at your website which you can filter through to your hearts contempt. For any companies listed within here which are SMEs, a quick Google search for the company name and check on LinkedIn will soon allow you to identify the relevant person to speak with in the company who could be interested in your products and services.
Even better, you can then begin to filter down by looking at these individual companies and see what pages they viewed on your website too, so that you get a better understanding of what they are looking for.
The key here is not to be too creepy though when you reach out to these people.
All in all, you rarely ever need to pay for premium services and tools which allow you to identify who is looking at your website. With a bit of know-how, you can have access to all of this type of information for free.
If you want to find out how Google Analytics can help your business, get in touch.
by Michael Cropper | Apr 15, 2016 | Ecommerce, WooCommerce, WordPress |
WooCommerce is the leading ecommerce technology in the WordPress ecosystem, yet many people, both beginners and experienced users, struggle with adding products to WooCommerce in the right way. I say the right way, as getting this part wrong can lead do long term issues and time consuming mistakes to fix when data isn’t entered in the right way. WordPress and WooCommerce are database driven technologies, meaning that to be used effectively data must be stored in the correct place for it to be used effectively.
To put this into something that is simple to understand. Imagine you’re a chef, a really good one too. You don’t just see a carrot as a carrot which can be eaten. You see the carrot as an ingredient which can make many things. Now imagine a whole host of ingredients that you have in your fridge ready to cook into anything you fancy. The ingredients in this instance it essentially the data behind a product, the pieces of information. If you take all of your ingredients and keep them in separate containers, you can continue having the flexibility to choose what you cook when you like, i.e. putting your product data in the right place within WooCommerce. Compare this with if you took all of your ingredients and mashed them up into a single container. You simply wouldn’t be able to work with that data in any effective way at all. Whatever you made, you’re stuck with essentially. This is why it is important to create your products in the right way when adding to WooCommerce, so you can work with them effectively longer term.
So let’s look at how to add a product to WooCommerce in WordPress. We’re not going to be talking about how to SEO the product you’re adding. For information around SEO, have a read through a few guides we already have on the topic including the Really Simply Guide to Search Engine Optimisation or our guide on How to Get the Most from Yoast SEO in WordPress.
Basic Product Data
The basic product data you’re working with within WordPress includes the following details which all need to be entered with specific information as this determines how they display to the customer on the website.
- Product Name: This is used to describe your product in a short succinct way. This data is used to generate your Meta Title (see SEO guide above), your Permalink (see below) along with your main Heading 1 tag on the page (see SEO guide above).
- Permalink: This is the full website URL which is generated for the specific product such as, www.example.com/product/product-name-here/
- Main Product Description: This is the main content which describes the product in as much detail as possible. The more detailed you can be with your product, the more visible your products will be on Google, the more information is provided to potential customers which can boost conversion rates and generate more sales.
- Product Short Description: This is a section which often sits next to your main product images, it is the piece of content which is designed to provide key unique selling points about the product to entice customers to read further and ultimately purchase the product.
- Product Image: This is the main product image which will be visible on the page when it is first loaded. This is also the image which will be used throughout the website when your product is linked to for example from the Category pages. Make sure it’s a good quality image and it is clear what the image is.
- Product Gallery: The more high quality images detailing every aspect of your product you can add the better. Product with more high quality images often significantly out perform in terms of sales when compared with products with low quality images or lower number of images.
- Product Categories: This is how your product is going to be categorised on your website. Think of a logical way to categorise your products, just in the same way supermarkets do. You wouldn’t find a fresh apple in the clothing section. Categorise your products by core areas to help your users navigate around the website with ease. If you want your product to appear in multiple categories, select multiple categories when adding your product to the website.
- Regular Price: This is what you are selling your product for. Depending on how you have configured your WooCommerce settings, this price may include or exclude VAT so make sure you get this one correct.
At the basic level, this is the only information you need to add a product to WooCommerce in WordPress. Make sure you are as detailed as possible when adding products to help users in the best way possible.
Advanced Product Data
Now we’ll look at some of the finer details as WooCommerce is an extremely powerful platform for powering ecommerce shops. To try and keep this as simple to understand as possible, let’s look at the different product options based on the type of product that is being sold, whether that is a single item, a grouped item, an external or affiliate item, or a variable product. Each of these allows you to customise many options for your products to offer your customers the best products possible. When adding a product to WooCommerce, all of the following data is within the following section;
For simplicity, if a piece of information has already been discussed previously, I’ve omitted this from the data below to avoid repeating myself.
Simple Product
General Tab
- SKU: This is your Stock Keeping Unit, the unique identifier which you can link to your own internal systems about the products you keep. This is a number that you can identify yourself. For example you may use something as simple as 001, 002, 003 or something much more complex. This piece of information is useful as your ecommerce store grows and you begin to add more products to the website. Customers may enquire about a specific product and being able to refer to the SKU can help you quickly identify what product they are talking about.
- Sale Price: Everyone loves a discount, right! If you want to run a special promotion, here you can select what discounts you want to apply to the individual product. You can even schedule the discount to be only applicable between certain dates so you don’t have to remember to turn the discount on and off.
- Tax Status: Tax is a complicated area which you need to discuss with your accountant and website developer to understand how this works for your business. With so many tax rules for businesses depending on who they are selling to and what they are selling, this is too big of a topic to cover here. The important point to note though is that you can edit the tax settings at the individual product level should you need to do this.
- Tax Class: As above.
Inventory Tab;
- Manage Stock: Select this option if you are using WordPress to manage stock levels. This can be useful if your WordPress website is your only point of sale for your products so you can be sure this information is accurate. The challenge over time comes when you are selling on multiple platforms both online and offline, stock management becomes a bit of a nightmare to get perfect which is a topic we’ll be covering in the future. If you do wish to manage stock levels for the product, simply tick the box.
- Stock Quantity: Self-explanatory, i.e. how many of this product do you have in stock to sell
- Allow Backorders: For many businesses, items which are out of stock are often only out of stock for a day or two. In which case you may prefer to allow backorders for products to generate the sale and post out as soon as the product is back in stock. This can often be better than losing the sale completely.
- Stock Status: If you prefer to manage this manually, you can simply change the stock status from either ‘In Stock’ or ‘Out of Stock’ accordingly which can be useful should an item suddenly become unavailable or otherwise.
- Sold Individually: This setting allows you to restrict the number of this specific item a customer can order on a single order. This can be useful for products you wish to offer to customers for free with their orders. For more advanced customisations related to order freebies there are other plugins and technologies to look at which significantly boost this functionality.
Shipping Tab;
- Weight: Self-explanatory
- Dimensions: Self-explanatory
- Shipping Class: Within the main WooCommerce settings, you can customise shipping classes to make things easier to calculate. This can be very useful if groups of your products all fall under the same shipping category. In which case you can then simply select which shipping class they fit under and the price will be automatically reflected.
Linked Products Tab;
- Up-Sells: Up-sells are products which will be recommended to customers who are interested in this product. This could be another product which is either more profitable which could be a very similar product with more features and functionality.
- Cross-Sells: Cross-sells are products which are recommended to customers who are interested in this product. This could be a product which is often used with this product. For example if someone is looking to purchase a winter hat, then they may also be wanting to purchase a winter scarf which matches this one.
- Grouping: This allows you to select if this product is part of a group of products which can be sold as a bundle. There are additional WordPress plugins which achieve this functionality which give you much more flexibility to customise grouped products.
Attributes Tab;
- Name / Value: This data is used to create additional attributes for your products which could include items such as colour and size. For example, you would add a Colour attribute which included items such as Blue | Green | Orange etc. And then add another attribute which is for Size which included items such as Small | Medium | Large etc. This data is really important to structure correctly as this data can be used in the future to filter search results on pages on your website based on these attributes. Make sure you don’t include all of this structured data about your products within the main description only as this data isn’t filterable in the same way data within the Attributes area is.
Advanced Tab;
- Purchase Note: This allows you to send customer specific information about this product once they have purchased which can be extremely useful if someone has purchased a product which may need additional information about how to set up for example.
- Menu Order: Ignore this, there are better ways to arrange menus
- Enable Reviews: Products with customer reviews significantly out-perform products with no reviews. Encourage customers to leave reviews of the products they have purchased which will then be displayed on the main product page.
Grouped Product
This product type allows you to create a product bundle which includes many products which can be easily purchased. The main problem is that this at the same price as if customers were to add them to the shopping cart individually. While this is certainly a nice feature, there are much more advanced plugins available for WooCommerce which allow you to create custom product bundles with associated discounts and special offers.
All of the settings within the Grouped Product type have already been discussed above.
External / Affiliate Product
This product type allows you to list products from external websites on your own website. This can be great if you are looking to run an affiliate based website which many small business owners do. When adding an External / Affiliate Product to your website, the purchase doesn’t actually happen on your website, the button on the product page simply takes the potential customer to the external website you specify which allows them to make the purchase directly with the product owner.
The couple of additional settings included with this product type include;
- Product URL: This is where you want to send customers to when they want to buy the product. You may want to check that if you are an affiliate that you have the relevant affiliate ID included within the link you add in here should the technology require this.
- Button Text: This allows you to customise the button text as people aren’t adding the item to the cart, they are essentially viewing more information about the product.
Variable Product
Variable products are where the more advanced features and functionality live which can help you truly customise your product page to do many of the things you have seen available on big brand websites. Again, many of the settings within here are identical to what has already been discussed so these have been omitted. The settings which are additional within here include;
- Attributes: As previously mentioned, this could include Colour or Size for the product along with many other options which are relevant to the product you are selling. When using this, make sure you select the checkbox stating that the product attribute is ‘used for variations’, this is really important and is a prerequisite to the next step, adding Variations. You need to save the product attribute that you have created.
- Variations: Think of product variations as a way to sell the same product with many different customisable options. Imagine ordering a pizza at a restaurant, £9.99 for a basic pizza. Add 50p for more chicken, add 50p for chilli sauce etc. It’s the same concept for variable products within WooCommerce. With variable products you can charge different amounts based on these different variations. For example, you may have a basic product with a variation of ‘{Product} with {Additional Feature X}’ which you can charge a premium for. You may not want to add this as a product in its own right as it is essentially the same product, so by using product variations it is possible to simplify this process to allow customers to select the features they wish to purchase for this product in a simple way. Make sure you add a Variation Description within here to explain to your customers how this variation is different than the main product or they will not be able to understand how this more expensive product is different.
Overall
In summary, adding a product to WooCommerce in WordPress is actually quite involved and is extremely flexible based on your individual requirements. Make sure you are adding in all of the relevant information for your products to make your life much easier in the long run as you build on the features and functionality on the website.
If you are having challenges with your WooCommerce WordPress website, get in touch and we’ll talk you through the options available to get you going in the right direction. WooCommerce and WordPress are complex systems which take a while to get used to.
by Michael Cropper | Apr 13, 2016 | Data and Analytics, Digital Marketing |
Blogging is a core part of any digital marketing strategy which is why we talk about this on a regular basis to businesses. We’ve already produced a Really Simple Guide to Blogging, so have a read of that resource once you have finished reading this blog post.
We get asked a lot about how important is blogging, really? Well I thought it was worth putting a few numbers behind this to put this into perspective about why we talk about this so much;
As you can see from the data above from our own website, when looking at the Blog and Resource pages which drive traffic to the website, this accounts for almost 40% of website traffic. Now to put this into perspective, imagine if you currently aren’t blogging on a regular basis. You are missing out on an enormous amount of traffic to your website. Assuming you aren’t blogging, by starting blogging based on these figures you could grow traffic to your website by almost 70%, imagine a 70% growth in traffic to your website. Pretty awesome, right?
Sure, things are a little more complex than this in the real world and for ecommerce websites this split is likely to be significantly different, but hopefully this gives you a few figures to highlight the importance of blogging. Get in touch if you would like to talk to us about how blogging can help boost your digital strategy.
by Michael Cropper | Apr 9, 2016 | Developer |
When developing basic websites, never need to be able to access your local machine form the internet. Whatever language or platform you are using, you’ll likely have your local development environment at something like http://localhost:8080 or similar. This allows you to test the code you are writing with ease without keep uploading it to your web server which is a more time consuming process.
When you start getting involved with building more advanced web applications, opening up your localhost environment to the internet is such a valuable tool, I can’t stress this enough. Particularly when working with technologies like webhooks, payment gateways and other integrated technologies. This is required because these websites need to send data back to your application outside of the initial request, and they cannot send this data to your local machine without being able to access it. You see, your local machine is protected by multiple firewalls, within your router as a hardware firewall and also with the firewall software running on your machine. So even if the outside website did know your IP address to send the data to, they wouldn’t be able to get through your firewalls anyway.
This is why you need to be able to access localhost from the internet which is only possible with specialist software designed to do exactly this. There are several different tools available to allow you to do this, personally I find the ease of use of Ngrok making it the only choice for me. To understand how Ngrok works, a couple of images shown on their website helps explain this in more detail as this is quite a technical topic to get your head around if you haven’t used before;
How to Access Localhost form the Internet with Ngrok
Image source
How to Send Data from Webhooks to Localhost with Ngrok
Image source
Ngrok have a list of platforms that it works on and this can be set up in as little as 30 seconds, it is seriously awesome. So anyone wanting to know how to access localhost from the internet or send webhooks data to your local machine, then take a look at Ngrok. They have free versions of the software and paid subscriptions which are dirt cheap.
by Michael Cropper | Apr 6, 2016 | Developer, WordPress |
The WordPress REST API Version 2 is brand new in the developer world which means that the documentation is extremely limited. Hopefully this can help a little for others trying to debug problems like this.
Posting comments to WordPress via the REST API Version 2 is actually relatively straight forward once you figure out how to do this. For those looking for a quick solution, using POST on the following URL will do just that, post a comment on this blog post you are reading right now.
https://www.contradodigital.com/wp-json/wp/v2/comments?author_name=Your%20Name%20Here&author_email=your-email-address@website-address-here.com&author_name=Your%20Name%20Here&content=Your%20Comment%20Here&post=1604252
Now for a few comments on the technical aspects to understand how this works.
Read the WordPress API documentation under the Create a Comment heading. As you will see, the documentation is minimal to say the least. It’s something the WordPress Core team are working on, so stick with it.
Essentially though, there are various query string parameters you can append to the request to send data into WordPress as a comment. It’s important to note that this is a POST request not a GET request. GET requests on this URL will not work. You need to use a tool such as Advanced REST client which allows you to POST data to API URLs which is extremely handy.
You will no doubt have debugging to do when you are first testing this as nothing ever goes to plan. Make sure you have comments turned on at the global WordPress level under the Settings > Discussions tab and also make sure that you have comments turned on for individual posts as sometimes these have been disabled. It’s always best to show these on your website too.
As with anything comment related with WordPress, make sure you are using the Akismet plugin to block any spam as this is a real nightmare on WordPress without Akismet.
There are lots of extremely useful uses for using this, we’ve been recently using this to post comments from a mobile app into WordPress which is used as a comment moderation system which means we don’t have to go and build that side of the functionality.
Make sure you are escaping the content which is included in the query string too and keep an eye out for any rogue spaces or special characters which may be breaking your POST request if this hasn’t worked. The usual things to check which you often miss J
When you successfully post a comment, you will see this waiting in your comment queue;
Then when you approve the comment you will see this displays correctly on your website as you would expect;
Have a go yourself. Customise the above URL with your own details and comments within the parameters and I’ll publish any successful comments. This handy URI Encoder / Decoder tool may come in useful when writing a comment or your name if it includes spaces or special characters, something I haven’t got around to migrating over to this site just yet.
by Michael Cropper | Apr 4, 2016 | Developer |
If you have ever worked with websites, web servers, cPanel, WHM and Exchange, you will know that there is never a situation that is ever the same. With default configurations for installations, changing settings and automatic updates, it’s easy for systems to sometimes get tangled in a few knots. Thankfully, there are tools available to debug things to identify specifically what is going on, below is a process which will hopefully save you hours of time investigating issues like this to quickly identify what is happening when your emails are going astray.
Sending & Receiving Emails
Ok, so if we look at the underlying technologies about sending & receiving emails first, this helps to understand where to first look for problems when they do occur. Let’s break this down into the two main categories of IMAP and Exchange as they both behave significantly differently. For Exchange, whether you are running a private Exchange server or whether you are on the Microsoft Hosted Exchange platform, this doesn’t make a huge deal of difference, so this information is going to be generic to the two with the understanding that there are always the odd discrepancies depending on configurations etc.
I’m going to assume that you have already set your MX Records correctly at the DNS. Make sure you have an SPF Record / TXT record with SPF details in if you are sending from IMAP or you will get blacklisted by Microsoft Exchange and will need to un-blacklist yourself.
Microsoft Exchange – Sending & Receiving Emails
Whenever you send emails from name@website.com from Microsoft Exchange, these emails are automatically handled by Exchange itself. Meaning that Microsoft Exchange respects the rules of the internet in relation to the DNS MX Records. So when you send emails to hello@another-website.com, Microsoft Exchange looks up the MX records for another-website.com and simply sends the message on its merry way to be picked up by the mail exchanger. Simple.
Whenever you receive emails into name@website.com, likewise, Microsoft simply looks within the Exchange settings for the website.com domain and decides which account this belongs in, if this is an invalid account or if this needs to be forwarded on to a distribution group or similar. Fairly straight forward.
Overall, Microsoft Exchange does just work and rarely is the problem at Microsoft’s end.
IMAP, cPanel, WHM and Web Servers – Sending & Receiving Emails
In comparison to Microsoft Exchange above which is essentially a single piece of software, sending and receiving emails with IMAP is a whole different ball game. It is one of the reasons why we always recommend companies use Microsoft Exchange, not only because it is a much more powerful system, but because it also costs you more for us to support you with IMAP related issues as they take 10x longer to get to the bottom of. Just use Exchange where possible.
Never the less, many businesses use a mixture of technologies for a variety of reasons, so it’s important to be able to understand how things fit together and most importantly how things can be quickly debugged and tweaked. So let’s look at how these different technologies fit together.
WHM is an administrator piece of software for managing web servers, much like an administrator account on your personal computer which allows you to install software, configure settings, create additional users etc. The difference being is that it allows you ultimate control of functionality through an easy to use interface and most importantly debugging tools for email delivery from and to your web server.
cPanel is designed for individual users and accounts. It is the equivalent of having an administrator account on your home computer (i.e. WHM) and then an account each for Mum, Dad, Son, Daughter (i.e. cPanel accounts). It is designed to allow each user / account to manage their own items on the web server in a safe and secure way without being able to interfere with other people’s data.
IMAP is a technology which transcends both WHM, cPanel and often desktop applications which makes things a little more complex when debugging issues. Domains are assigned to individual cPanel accounts, such as website.com and my-other-website.com. And individual emails for name@website.com are configured within the individual cPanel accounts, even though the IMAP technology means that it is the actual server sending and receiving emails and deciding how to deal with email routing.
Websites also send emails from scripts and code which has been developed which introduces another layer of complexity as from an underlying technology point of view, the settings which are configured mean that emails from websites may or may not be delivered, even if they appear to be sent when you send a message on the contact form on the website for example. Always test your contact forms on a regular basis.
Whenever you receive email to your IMAP email address for name@website.com, ‘the internet’ routes this email from the sender, via the DNS MX record settings, to your web server where your emails live. The software running on your web server then ultimately decides which account this should end up in. Receiving emails is generally ok as long as the accounts are set up.
Whenever you send email from your IMAP email address for name@website.com, this is when things get a little different. This is because there are many technologies interacting together, particularly when you are using a desktop email application to send emails or even a website to send emails. There are other things to consider particularly when web servers, websites and email applications may have been configured a specific way which may in fact mean that emails are actually sent through SMTP opposed to IMAP.
Debugging Email Deliverability Problems
Ok, so now we’ve had a look at the many moving parts behind the scenes when it comes to emails, let’s look at how you go about debugging email deliverability problems and where to start. I’m going to stick with IMAP here as Exchange is an enormous topic on its own and takes quite a bit of learning. Once Exchange is configured, it just works. Get in touch if you are having issues related to Microsoft Exchange configuration and I’m sure we can help walk you through what needs to be done.
DNS Configuration
This has been mentioned before, but it’s worth reiterating. Make sure your DNS is configured correctly and is pointing to the right place it should be. This relates specifically to your MX records, and TXT, A, SPF, or SRV records which may need to be created depending on your email technology in the background. Speak with your web hosting company for the actual settings here.
Review What is Actually Happening
Next, review what is actually happening to try and determine where the problem may lie. Review;
- Can you send emails via webmail?
- Can you receive emails via webmail?
- Can you send emails via your desktop application?
- Can you receive emails via your desktop application?
- Is your website sending emails?
- Are you receiving emails sent by your website?
- Can you send emails to specific domains?
- Can you receive emails from specific domains?
- Can you send emails to specific domains when sending from another specific domain?
- Can you receive emails to a specific domain when sending from another specific domain?
- Can you successfully authenticate with the server when trying to send and receive emails?
- Have you correctly configured your email settings and port settings?
- Have you waited 24 hours for DNS propagation if you have just changed your DNS settings?
- Are there any service interruptions with your web server?
The more you can narrow down exactly what is happening here, the greater the clues are to what is actually going on. Yes, your emails don’t work, but why, what is the specific issue we are facing. Ultimately, when emails are sent, they should be received at their intended destination. When things don’t quite go to plan here, it means that something in the process is doing something different which is generally down to some settings somewhere which need to be tweaked.
Tracking Received Email Deliverability
Receiving emails into IMAP is much easier to debug than sending emails, so it’s always best to start here. If your DNS records are set correctly, then you should be receiving emails by default. As long as you have an email account set up within cPanel, your emails should be arriving successfully. Always check your junk folders and make sure that you web server isn’t automatically junking / deleting emails that it believes are junk. I have seen this happen in the past with over active junk email settings. Likewise, keep an eye out for any email forwarders that may have been set up and are taking priority when the email arrives and sending it off somewhere else before it lands in your inbox.
Generally speaking though, if you aren’t receiving emails it is due to your DNS records being incorrect or the email address you are sending to blocking emails from your server as they look like spam. Make sure you are sending these test emails from an email account that you know already works. It helps to have a few email accounts such as Hotmail / Outlook / Gmail etc. so that you can test this from various sources. If you aren’t receiving emails from your other email accounts that you know are working correctly, then re-check your DNS settings.
Tracking Sent Email Deliverability
Now onto tracking your sent email deliverability issues. This depends on where you are sending emails from in the first place. If you are sending emails from your website and they aren’t being received, this again can be for many reasons. It could be related to how your web server is configured for when it is sending emails to @website.com, it could be trying to deliver your emails locally to an email account hosted on the web server and in which case if you are running your emails remotely through Microsoft Exchange, then you may need to update these settings. It could also be the way in which you are sending emails as many web hosting companies actively turn off specific functionality on web servers which allow code to send emails as a security measure. Speak with your web hosting company about this, it is the only way to know for sure.
If you are having issues sending emails via webmail or via a desktop application, first try to verify that this isn’t an issue which relates to your desktop application communicating with your webserver. Try and aim for a single point of failure to test if this point is failing or not and work your way through each one by one, I’m talking about testing specifically, you should never have a single point of failure for anything in business – if you do this is a whole other discussion to be had. Assuming you have tested this, next step is to look within WHM to see where your emails are actually going when they are being sent from one of the accounts on IMAP. There is debugging tool within WHM under Email > Mail Delivery Reports which allows you to see the email you just sent, where did it end up. Often this relates to an issue with local / remote / automatic mail delivery. Check what is happening here, if the recipient isn’t who you intended, then you need to configure your settings within cPanel accordingly so that cPanel isn’t overriding any settings when it shouldn’t be doing.
Outliers and Things to Remember
The above is a rough guide and doesn’t cover everything you could ever check for. For example, one project we recently worked on had a unique issue which doesn’t crop up very often whereby sent emails were getting lost in the ether due to configuration settings within cPanel which means that only a tiny amount of emails were getting lost, everything else was absolutely fine. The situation was that within cPanel, multiple domains were configured including website1.com and website2.com. One website was running Microsoft Exchange, the other IMAP. Exchange was working perfect. IMAP was working perfect….almost. The issue cropped up when website2.com on IMAP wanted to send emails to website1.com on Exchange. Because both domains were configured under the same cPanel account, MX Entry settings within cPanel meant that emails sent from website2.com on IMAP were attempted to being delivered locally due the website1.com domain having an automatic configuration in place which decided where emails should be delivered. Changing this to a Remote location resolved this issue, meaning that cPanel no longer overwrote the default behaviour of the DNS and things continued to function normally.
There are always things to bear in mind and check.
Summary
When using older technology such as IMAP instead of the far superior Microsoft Exchange, debugging email deliverability issues can be tough if you don’t know your way around your web server, the software that is running and particularly on cheap web hosting packages where you may not even have access to many of the tools outlined above. It is so important to make sure you have good web hosting in place for your business to allow you to debug problems like this quickly and continue working away. The last thing you want is weeks of delays trying to get to the bottom of these types of issues as you could be missing vital orders and customer queries.
Should you need any support managing your email technologies, get in touch and we can happily support you in this area. The reality is that all web servers are configured differently, have many technologies running and various settings based on your individual requirements. Cheaper web hosting is more of a challenge to debug as you simply often don’t have access to the tools you need and can only deal with the issues by speaking with the web hosting company yourself or letting someone like ourselves do this on your behalf so they don’t try and bamboozle you with technical jargon with the aim of fobbing you off. It’s always good to work with companies who understand these technologies inside and out. Get in touch for more information.
by Michael Cropper | Apr 3, 2016 | Developer, Retrospective |
Rarely a day goes by without a new project with a new set of requirements, legacy systems and challenges which need to be overcome. This was one of these projects, where one of the many popular solutions we work through simply wasn’t going to meet the requirements on this occasion. I’ll leave the full reasons behind this story for a case study once we write this up. For now, to keep things simple we had to migrate around 250,000 emails from an old web server running IMAP technology and split these between two new places, the most used email accounts to run on the recommended Microsoft Exchange platform and the other email addresses to run on the older IMAP technology. There are many reasons why this had to be done, but I’ll skip over these for now.
So how exactly do you go about migrating 250,000 emails without losing access to anyones account, without any emails going astray during the migration and without losing any of the emails from the old system? Well, it requires quite a lot of planning, clear communication and most importantly a strategy that is going to work within the situation. This guide is for anyone else experiencing a large email migration project where it is essential everything continues running at all times. Did we get is 100% perfect, not quite. I’ve yet to experience any IT or software project be 100% perfect, but I’d like to think we did this with 99.5% perfection given the restrictions we were working with. This guide is going to cut straight to the chase and explain what the key stages of this process you need to go through to get ever closer to the perfect migration system.
Technologies
If you aren’t familiar with the differences between IMAP and Microsoft Exchange, then have a good read through our guide on the topic to get familiar. In short, when needing full traceability and efficient working with email technologies from anywhere in the world and on any device, quite simply you need to be using Microsoft Exchange. IMAP is not suitable for running a large organisation for a multitude of reasons.
With IMAP being the less advanced technology, this ultimately means that there are things that it doesn’t do as efficiently as Microsoft Exchange and most importantly, IMAP isn’t actually a piece of software per say. It is actually a protocol, a set of rules for how things are handled. What this means is that while there are a set of rules for how things are handled, as the protocol is implemented by software running on a web server, this in itself can lead to a whole host of problems depending on the how well the web server has been set up and configured, or how badly. As such, it is difficult to predict with 100% accuracy exactly what is going to be needed without actually jumping in and getting on with it. This being said, you can certainly improve your odds by working through a structured process to maximise the efficiencies within the process.
Requirements Gathering
This has to be one of the most important aspects of any project. Before you jump in and start recommending any solutions, you need to fully understand the situation that is in place at the minute, the issues with this set up and the ideal functionality that is required within the organisation. Spend time talking to as many people as possible to understand the challenges they face and most importantly explaining the variety of options and their benefits. Once you have a full list of requirements you can then begin to plan the best solution.
Planning
With your set of requirements in place, you can review the options that are available to you and decide upon the best solution. There is no one-size fits all when it comes to technology, so make sure you are taking advice from a reputable company, it will save you a fortune in the long term. Or if you are the company implementing this process for someone else, make sure you don’t bite off more than you can chew, project like this can go really bad really quickly if you aren’t 100% sure what you’re doing.
In this specific example, our planning stage was a little more complex due to splitting all of the old IMAP emails between a new IMAP destination and Microsoft Exchange. This in itself required a change in email addresses for some of the users as you cannot use multiple technologies for emails on a single domain such as @example.com. In this case we had to set up a sub-domain for something.example.com which meant that the @example.com domain names could be powered by the superior Microsoft Exchange technology while the @something.example.com email addresses could be powered by the older IMAP technology.
To achieve this requires good web hosting and DNS management in the background. When you are on cheaper budget systems, you often cannot set up sub-domain delegation for MX records which is required to do this. If you aren’t sure what an MX record is, essentially this is a single piece of information which is used by ‘the internet’ which means that when an email is sent to hello@example.com, ‘the internet’ checks the MX record associated with example.com to see where it should forward the email onto. This is generally either Microsoft Exchange or an IMAP server somewhere, generally the one running on your web server at a specific port. As we were using two different technologies in the background, this meant that we needed to have two MX records which is not possible on a single domain, hence needing to set up an MX record at the sub-domain level too.
A key part of the planning process is communication which brings me on to the next section, before we look at the nitty gritty of how to actually migrate 250,000 emails with zero downtime.
Communication
Throughout any project, success or failure is generally down to lack of communication in one way or another. Migrating 250,000 emails is no different. Particularly when email is generally the most efficient way of speaking to people, you cannot rely on this should anything go wrong. In other words, you need another form of communication as a backup should things go pear shaped for whatever reason. Whether another email account on a 3rd party platform or the good old telephone.
As the technical person implementing this, the process behind migrating the large number of emails requires this information to be communicated to the key stake holders in a non-technical and easily understandable way – which can always be a challenge when working with naturally very technical aspects. So whatever you do, make sure you are over communicating with everything you are doing.
Implementation
So, let’s get down to the nitty gritty for how to migrate 250,000 emails with zero downtime.
Old Web Server Review
You need to know what you are starting with. Generally speaking, when you are migrating web servers for whatever reason, it is usually due to the old web server being a bit rubbish in whatever form. This often is coupled with a lack of flexibility that you are used to when working with industry leading technology. You must fully review the old web server, every aspect and speak to the key stake holders who have had anything to do with managing this in the past to fully understand any interesting configurations that may have been set up in the past which can be easily missed.
We’ve already covered about how to migrate a web server seamlessly with zero downtime, so have a read of that if you need more information on the topic. For now we’re going to look specifically at the email side of things. Remember, we wanted to migrate 250,000 emails from IMAP, half to Microsoft Exchange and half to the new web server which was also running IMAP. This required two significantly different approaches to achieving this task.
Prepare the DNS
First things first, we would always recommend migrating things in stages. So let’s assume that you are first going to migrate the contents of the web server, namely your websites, then secondly migrate your emails. Or vice versa. To do this requires you to prepare your DNS settings accordingly.
As DNS propagation can take up to 24 hrs to complete fully, you need to plan this step in advance. With good DNS providers, this is generally extremely quick, although do not count on this, plan in advance.
In our example, we were also migrating the DNS settings which were previous set at the old web hosting company. Instead we moved these to an independent setup, meaning that we could prepare these in advance so propagation could take place throughout the system prior to us changing nameservers at the registrar.
Registrar – Changing Nameservers
Once we had successfully added the DNS settings at the new DNS provider so that the website would run off the new web server and the emails would continue running off the old web server, we were set to go with changing the nameservers. Once we made this change, this meant that we could be confident that and downtime was kept to a minimum.
Ok, so now the first process had been completed, this meant that we could move onto the emails. To do so requires more planning and careful implementation.
Preparing Microsoft Exchange
If you have ever set up Microsoft Exchange before you will know that there are a few settings to configure at the DNS level. To configure Microsoft Exchange in preparation for when we needed it, we actually had to add some of the DNS settings on at the old web hosting company prior to switching the nameservers. This is required to give you the time to authorise your account along with configuring all of the relevant mailboxes and communicating with the individual users for the project.
This guide is not designed to look at how to configure Microsoft Exchange, so if you don’t know how to do that, get in touch and I’m sure we can help you in some way with what you’re working on. For now, assume that a significant amount of time was taken to get everything configured correctly. Interestingly, as part of this specific project, we had to hunt down a rogue Microsoft Exchange account that had been created many moons ago by someone within the organisation with zero information about what had been set up and why and most importantly who owned this. As the Microsoft Exchange system is extremely secure, there are safeguards in place to prevent someone else taking control of an Exchange domain if one has already been set up, meaning that we first had to hunt this down and regain control of this. I certainly hope you never have to do this as I can’t tell you how time consuming this is to do.
Ok, so now you have Exchange all prepared for your email migration project. Next is preparing the IMAP email accounts on your new web server.
Preparing the new IMAP email addresses
First things first, you need to create all of the accounts on the new web server for all of the email addresses which are going to be using IMAP. Make sure you do an audit up front on the old web server to understand how large the individual email accounts are. Often when you create an email account with a default size of say 1GB on IMAP, then for certain users this may not be enough space. You want to plan for this in advance as you do not want to find this out when you are half way through transferring emails only to have to start again.
Thankfully on good web server control panels such as cPanel, doing this initial research is simple and relatively quick. Unfortunately on poor control panels which often come with budget web hosting packages, finding this out before hand can be a little challenging and in some cases simply just not possible which is a little annoying.
Switching the New Email Addresses On
Now everything has been prepared with regards to your new email technologies, it’s time to switch the DNS settings to point all the email related activities to the correct sources. Make sure you don’t make any mistakes when doing this as everything will start to go wrong and emails could go astray.
Migrating 250,000 Emails
Ok, so now we have everything prepared, we’ve switched the emails on but there is nothing there other than emails which are now sent from this point forward. Jumping back a little, it is actually possible to migrate the emails to Microsoft Exchange when configuring all of the accounts. There is a handy tool within Microsoft Exchange which isn’t 100% but can often connect to your old mail server and import the IMAP emails into Exchange which is a different protocol. When this works, this is perfect as you now have a system whereby all of the sent and received emails along with relevant folder structures that people use within their emails has been successfully ported over to Exchange. I’d recommend doing this section prior to switching your new emails on because it can take a while, as in days. So set this running and be patient. The larger number of mail boxes you are dealing with, the longer this will take. When migrating in batches, migrating around 20 mailboxes from IMAP to Exchange took around 8 hours to put this into perspective.
Now, as mentioned before, IMAP isn’t as good as Exchange when it comes to the underlying technology. Meaning that while there is a handy little tool that comes with Microsoft Exchange to do this task, the same is not true for IMAP. As IMAP runs of your web server, there are endless different configurations, setups and options which come into play. Meaning that you’re going to have to figure this bit out for yourself to a certain extent with a lot of testing various options.
Thankfully there is a seriously awesome tool called IMAPSync, which is designed to help with a lot of this process. At only £100 it is worth its weight in gold and anyone attempting to migrate any amount of emails between web servers should purchase this tool. The developer behind the scenes who built the tool is also extremely useful when helping with the usual configuration niggles that you face with things like this.
I’m not going to cover how to use this tool in detail as this is going to differ depending on your individual requirements. Suffice to say, that with a bit of testing for various configurations etc. you will get there in the end. Make sure you test things first on a test account on your new IMAP web server though so you can be confident that this is working as expected. The last thing you want is to be all gung-ho and start everything running only to find you did something wrong.
As a quick overview of IMAPSync, this tool requires you to install the software from the Yum repositories, upload a tarball to the web server under the Root account via SSH on the new web server, unzip this zipped folder, test a few configurations to make sure the scripts can actually access the internet and then you’re off to go. Something to bear in mind is that if you are having trouble getting the script to connect to the internet, it’s likely due to the firewall running on your web server, either physical or software based, which is blocking specific ports on the outbound side, generally port 143 for IMAP.
So there you go, once you’ve managed to run through this entire process, you’ll be in a situation where all of your IMAP emails are now also migrated to the new web server.
Sender Policy Framework
Something to bear in mind is that when setting up a IMAP emails, you will likely be blocked as spam if you don’t have SPF records set up at the DNS level. Leading technologies such as Microsoft Exchange will block emails from domains who don’t authenticate their self through SPF records by default as many of these emails are automatically generated spam. Something to bear in mind if you start getting issues with deliverability to certain email accounts or domains
Summary
We deal with email migration on a regular basis and you often can’t quantify the project before you get involved, it is extremely difficult to predict. Overall though, what I can say is that if you are going through a similar project, you need to be planning things effectively and implementing the right technologies to make sure everything works. For larger organisations, losing access to their emails for even an hour is a lifetime. With a well thought out strategy, detailed requirements gathering and planning your implementation will be much more straight forward. Follow a similar process when you are migrating emails and you too will be able to migrate a significant amount of emails with virtually zero downtime.
The only other thing I would finish with is that it is best if you can implement some of the critical items such as DNS changes at times which are less likely to have an impact. For example late in the evenings or at weekends. It’s not ideal for the person implementing this, although the benefits to the organisation you are working with is enormous, it will result in a situation where emails don’t end up getting lost in the ether while DNS propagation is taking place. It is also recommended to keep your users informed at all stages and most importantly tell them when not to send emails when you are making DNS changes, users often work at all hours of the day when you may be expecting them not to be, so keep in touch with people throughout.
